At the end of last month, the Ohio Board of Regents announced the University System of Ohio Textbook Portal. The service has been talked about in the media, in trade publications, and in numerous blog postings. Enough time has passed now that word has gotten out, and I won’t be taking any of the chancellor’s thunder about the project. I did the back-end development work for the portal and wrote this document as an introduction to the project for our development team and anyone else interested about the project.

The textbook portal is based on the Drupal (version 6) content management system. In particular, the portal makes heavy use of the search module to execute and format search results. If you are familiar with Drupal, it is going to be different enough, however, that you’re going to want to read this to see why some decisions were made. If you are not familiar with Drupal, this document will give you a head start into understanding the Drupal way of the world.

A couple of points before we start. First, before starting this project I had only a passing familiarity with PHP as a programming language and no experience with code development for Drupal. ¹ Read the code with that frame of mind; if you have more experience in either of these areas and know of a better way to do something, please let me know and I will gratefully incorporate your suggestions into the code. Second, you can find the code in OhioLINK’s public Subversion repository and reference to it in OhioLINK’s public Trac project server, should you want to take a look at it yourself.

This code in the Subversion repository corresponds to everything under the /sites directory of a Drupal installation. In the basic Drupal installation, there are two subdirectories in this directory: all and default. In a multi-site Drupal installation, the “all” directory is supposed to correspond to modules/themes that are made available to all sites within an installation while the “default” directory is intended for modules/themes for the “default site“. I’m using the distinction somewhat differently. Everything in the “all” directory is third-party modules and everything in the “default” directory is stuff I’ve created. It is an arbitrary, unnecessary distinction, but I think it will help with maintenance.

At a very high level, you can look at the Installation Documentation for the ETextbook Portal. This document was written from the perspective of a bare metal restore of the service. (Well, not quite — it assumes Ubuntu is installed on the server.) It has the various applications and modules that need to be installed to get the site up and running. This should make a good checklist should you wish to reproduce the portal. Knowing how to install Drupal comes in handy, but the installation process itself it pretty easy.

If you follow the documentation up to the point of restoring the database, you’ll have a good foundation. But doing so will mean that there are several configuration options you’ll need to set that would otherwise be in the database backup. You’ll need to activate these modules:

eTextbook Metasearch

Integrates the results from the various textbook search modules. This corresponds to the Drupal node type “all”.

CourseSmart Search

Searches the CourseSmart eTextbook Database. This corresponds to the Drupal node type “csmart”.

OhioLINK E-Books Search

Searches the OhioLINK E-book Center. This corresponds to the Drupal node type “ebooks”.

OhioLINK Library Catalog Search

Searches the OhioLINK Central Catalog. This corresponds to the Drupal node type “libcat”.

Safari Search

Searches Safari Books Online. This corresponds to the Drupal node type “safari”.

Each of them has minor, but important, configuration parameters that you’ll need to set up in the Drupal installation’s /admin/settings directory. In particular, the CourseSmart Search module will have parameters for the discount coupon code plus the username/password for the private API (the private API is discussed in the module-specific section below).

Structure of the Search Modules

function module_menu() {
  ...
}
 
 
function module_perm() {
  ...
}
 
 
function module_search($op = 'search', $keys = NULL) {
  ...
}
 
 
function module_form_alter(&$form, $form_state, $form_id) {
  ...
}
 
 
function module_search_process($keys) {
  ...
}
 
 
function module_format_result($item) {
  ...
}
 
 
function module_search_box_form_submit($form, &$form_state) {
  ...
}
 
 
function module_search_query($keys = '', $query = array(), $search = 'web', $version = 'v1') {
  ...
}

Some explanation for each of these:

• module_menu() is a Drupal hook that defines the menu options for the setting screen. The code to generate the menus themselves will be in a file in the module called “module.admin.inc”.
• module_perm() is a Drupal hook for defining the user permissions appropriate for this module. It isn’t really used in the portal. (The settings screens look for the “administer site configuration” user permission value.)
• module_search() is a Drupal hook that defines a custom search routine for nodes of this type. The code pattern in other Drupal modules seems to be to use this as a level of indirection to a non-hook function, such as module_search_process().
• module_form_alter() is a Drupal hook for changing the behavior of a form before it is rendered in the HTML back to the user. In conjunction with module_search_box_form_submit(), the code in this hook will turn FORM POST requests into pretty URLs.
• module_search_process() is the function called by the module_search() function. This function prepares the query, including the pagination-of-results calculation, and calls another function — module_search_query() — to do the actual searching. We’re adding this level of indirection because the “metasearch” module will also call module_search_query() to get results, but the code in that module does do all of the things module_search_process() does.
• module_format_result() is called with information about the search hit, and formats it in a way that can be fed back into the Drupal search.module output engine. The issue here is that we’ve got fielded data (author, copyright year, publisher, and ISBN) that we want to display as fielded, but Drupal doesn’t give us a way to do that. Rather, Drupal’s standard search module is looking for an array with keys for ‘title’ of the hit, ‘link’ of the hit, and a ‘snippet’ to display to give the user context for the result. (See the “Return Value” heading of the hook_search API documentation.) So this module will create a snippet of HTML that builds a nice display of the fielded data.
• module_search_box_form_submit(), in conjunction with module_form_alter(), forms the callback to turn FORM POST requests into pretty URLs.
• module_search_query() performs whatever functions are required to get hits from the remote service. This, of course, is the real heart of what we’re doing. Rather than searching text of nodes internal to Drupal, this function will return an array of results that comes from a query of a remote service. The array returned has two elements: ‘total’ — an integer representing the total number of hits for the query, and ‘items’ — an array of individual hits from this search.

Module-specific details

CourseSmart is probably the simplest module of the bunch and a good place to start when looking at the code. Note that we are using the private API (appending md=1 to the end of the URL) in order to get the ISBNs as listed on the CourseSmart website. Calls to the private API is restricted to particular IP addresses, so in order to use it you’ll need to contact CourseSmart. CourseSmart is also a little funky in that they will return items in their inventory that they won’t sell. This is designated with an esubscription price of $0, and are filtered out in the module_search_process() function.

OhioLINK EBook Center uses the SRU interface to the underlying XTF installation in order to get search results out. The search results come back in an XML document returned with multiple namespaces, which complicates somewhat the DOM parsing of that document. Basically, it means one has to register the namespaces with the XPath processor and take them into account when using XPath to pull out elements for formatting the result record.

OhioLINK Library Catalog uses the Shrew PHP class created by David Walker at California State University. Shrew hacks through the MARC display of records for an Innovative Interfaces WebPAC and returns a MARCXML document. Without this, I’d really be stuck as to how to efficiently get the library catalog search results into the portal. I’m grateful to him for releasing the code at exactly the right time and to Rob Casson at Miami who pointed me in David’s direction when I was considering having to write the Shrew-equivalent myself.

Safari Books Online is using the same underlying engine as CourseSmart to deliver materials, so the search module is very similar.

Structure of the Metasearch Module

• When the Drupal all_search hook is called with the ‘search’ operation parameter, a results array with explicitly 1 “result” and the search keys as the item returned. What we’re really doing is faking out the Drupal Search module into thinking that there are actually results so we can get to the all_search_page() hook. If we didn’t set the number of results to a value greater than zero, Drupal would display the “no hits found” message for us (which we don’t want it to do).
• The undocumented hook_search_page(), when defined for a module, is called by Drupal rather than using the built-in internal search results page. (The other modules use the built-in results page rendering.) We override the hook using all_search_page(), and that function calls each of the module_search_query() functions for the four remote sources in sequence. The results are then put into output block and the block is returned to the calling core code.
• “all” also contains several utility functions used by the other modules. all_parse_keys() will look at the user’s search string for ISBN values and return the user’s search string as an array of an ISBN and everything else. all_proxyify_url() will determine whether a user is outside of a campus network and prepend the OhioLINK proxy server string to the URL.

Plans for Enhancements

• We want to include bookstores in the search results. In particular, where possible, we’d like to search the bookstore’s inventory control system and display results right in the metasearch results.
• For the metasearch results, each of the target remote services are called in sequence. Ideally, the four services would be called in parallel. Even better, perhaps, would be to render the base page, then inject search results from the remote services via AJAX as they become available.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/etextbook-installation.html?rev=1042 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/ on January 13th, 2011.

The text was modified to remove a link to https://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/all/all.module?rev=1042#L173 on January 13th, 2011.

The text was modified to remove a link to https://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/all/all.module?rev=1042#L61 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/ebooks/ebooks.module?rev=1042#L137 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/ebooks/ebooks.module?rev=1042#L130 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/csmart/csmart.module?rev=1042#L101 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/libcat?rev=1042 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/ebooks?rev=1042 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/csmart?rev=1042 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/all?rev=1042 on January 13th, 2011.

The text was modified to remove a link to https://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/all/all.module?rev=1042#L206 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/eTextbookPortal/default/modules/safari?rev=1042 on January 13th, 2011.

The text was modified to update a link from http://drupal.org/getting-started/6 to http://drupal.org/getting-started on January 28th, 2011.

Footnotes

1. These made seem like odd choices to make for a project that had a short conception-to-production timeline, but a) there were already some helpful pieces written in PHP that sped development of some aspects of the portal, and b) I thought drinking the cool-aid of Drupal would be a good way to see what it was all about.

Schemes to add functionality to the web OPAC fall into four categories: web OPAC enhancements, web OPAC wrappers, web OPAC replacements, and integrated library system replacements. I’m outlining these four techniques in a report I’m editing for an OhioLINK strategic task force and a bit of a reality check on this categorization is desired, so if I’m missing anything big (conceptually or announcements of projects/products that fall into these categories), please let me know in the comments. Generally speaking, this list is ordered by cost/complexity to implement — from lowest to highest — as well as the ability to offer the described enhanced services from least likely to most likely.

Web OPAC enhancements are functions that are added to the existing web OPAC system. This most often entails additional product purchases from the automation vendor, such as the optional enhancements in WebPAC Pro for Millennium OPACs or content solutions in SirsiDynix. Enhancement can also be added through creative use of an existing web OPAC’s template functions, such as the method by which LibraryThing for Libraries can be added to OPAC displays.

Web OPAC wrappers use the existing web OPAC provided by the integrated library system as a source of information, but hide that information behind a completely new interface. The intervening system get that information from the integrated library system through a variety of mechanism. In some cases, it may be possible to use established protocols (such as Z39.50) or programming interfaces (such as an XML content server). In cases where such functionality is not available from the underlying integrated library system, a “screen-scraping HTML” technique may be required. ¹

One example of such a wrapper is the work at Ann Arbor Public Library on SOPAC. Short for “Social OPAC,” SOPAC is “a set of social networking tools integrated into the AADL catalog [that] gives users the ability to rate, review, comment-on, and tag items.”² It uses an open source content management system called Drupal as a structure through which the added functionality is provided. For example, when a user seeks the bibliographic information page for a catalog record, that request is made from the user’s browser to the Drupal software. The Drupal software in turn makes a request to the integrated library system for the bibliographic information it holds. The response from the ILS is parsed by the Drupal software for key information such as title, author, subjects, holdings, etc. This information is mixed with information stored in the Drupal database (ratings, tags, reviews, cover images, etc.) and a new web page is created and returned to the user’s browser.

Another example of a web OPAC wrapper is Scriblio (formerly called WPopac). Using the underlying framework of WordPress, Scriblio offers faceted browsing, tagging, and syndication feeds for the underlying Millennium WebOPAC. Scriblio is a project of Plymouth State University, supported in part by the Andrew W. Mellon Foundation. Both SOPAC and Scriblio are available under open source licenses.

Web OPAC replacements are new systems that completely replace the existing web OPAC. Unlike wrappers (which get their bibliographic data in real-time from the underlying web OPAC), these replacements operate on sets of records that are extracted from the ILS or come from another source. (In some cases, these replacements still rely on the underlying web OPAC as a source of item status information such as checked out status and due date.) The first notable OPAC replacement was at North Carolina State University when its library installed and configured the Endeca software to provide a faceted browse to the library catalog. By itself, an Endeca OPAC display does not enable tagging, annotation, or user aggregation services such as recommendation engines.  Other similar web OPAC replacements are Encore from Innovative Interfaces³, Primo from Ex Libris and Aquabrowser from Medialab Solutions. Miami University’s experiments with the open source Apache SOLR and the exported records from their Millennium system also fall into this category. Worldcat Local is also a form of web OPAC replacement noting that the source of bibliographic records is the OCLC Worldcat database rather than the local ILS.

ILS replacements offer the biggest opportunity for enhanced user services, particularly by adopting one of the open source solutions now available. At this time, neither of the open source solutions (Evergreen and Koha) offers more than faceted search and browsing. Unlike the commercial systems, however, the source code of the system can be modified to add these functions, and the modifications shared with other users of the same system.

[Update 20071015T1624 : Corrections made -- and the text improved! -- based on Betsy Graham's comment. Thanks, Betsy!]

The text was modified to update a link from http://www.iii.com/mill/webopac.shtml to http://web.archive.org/web/20071015145500/http://iii.com/mill/webopac.shtml on January 20th, 2011.

The text was modified to update a link from http://www.iii.com/encore/main_index2.html to http://web.archive.org/web/20080328163000/http://www.iii.com/encore/main_index2.html on January 20th, 2011.

The text was modified to update a link from http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content to http://web.archive.org/web/20071109170053/http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content on January 28th, 2011.

Footnotes

1. Such a technique gets the information from the ILS using the existing web OPAC. Such schemes are generally fragile because changes to the underlying web OPAC can have detrimental affects on the content scraping process.
2. Blyberg, J. (2007). AADL.org Goes Social. blyberg.net. Retrieved October 12, 2007, from http://www.blyberg.net/2007/01/21/aadlorg-goes-social/
3. As Betsy Graham, Vice President of Product Management at Innovative Interfaces, notes in the comments, the Encore will perform real-time queries to a Millennium ILS for bibliographic data, and in such cases the data extract is not needed.

A while back we created an LDAP directory to consolidate account information for various back-room services, and when we created it we decided to use the individual’s e-mail address as the account identifier (uid in LDAP-speak). It seemed like the logical thing to do — it is something that the user knows and it is a cheap and easy way to assume that the account identifiers will be unique. This is not uncommon for many internet services, of course.

Now we’re bring up a Drupal content management system and of course want to tie the authentication into the existing LDAP directory. The initial configuration appeared to work, but there were odd, unexplained failures — most notably, Drupal would not consider it a ‘real’ account because it didn’t have an e-mail field. Even weirder was the fact that we configured Drupal to know exactly which LDAP attribute to use as the e-mail address (mail, in LDAP-speak). It wasn’t until one of our system engineers wondered out loud if the at-sign (‘@’) in the user id wasn’t causing problems that we started making progress towards a solution.

As it turns out, he was right. Without spending so much time in the guts of the Drupal code to know exactly if this is true, it seems like Drupal wants to reserve the ‘@something‘ construct for inter-Drupal authentication. In other words, if you have an account on one Drupal server (let’s call it DrupalA) and want to access a second (let’s call it DrupalB) — and if the two servers agree to share user accounts — the account from DrupalA would be recorded in the database of DrupalB as “UserId@DrupalA“.

The ‘at’ symbol for us, though, is just a normal part of an e-mail address. We really didn’t want to reconstruct our LDAP account scheme, so the best choice seemed to be to find a way to trick Drupal into accepting these account identifiers. This, unfortunately, was no easy task. I couldn’t find the root cause of the problem, but did diagnose enough of the symptoms to force a patch into the system. The patch, in the form of a new module (code included below) forces the account to have two necessary attributes that seem to go missing whenever a ‘@’ character appears in the user id. If you have similar problems, I can’t claim that this will work for you, nor can I guarantee this approach will be supportable in the future. All’s I know is that it seems to work for us in our situation right now.

 <?php
 
function olinkldap_help($section) {
  $output = '';
  switch ($section) {
    case 'admin/modules#olinkldap':
      $output = 'olinkldap;
      break;
    case 'admin/modules#description':
    case 'admin/help#olinkldap':
      $output = t('Sets up OhioLINK-specific LDAP parameters.');
      break;
  }
  return $output;
}
 
function olinkldap_settings() { }
 
function olinkldap_user($op, &$edit, &$user, $category = NULL) {
  switch($op) {
    case 'load':
      olinkldap_user_load($user);
      break;
  }
}
 
function olinkldap_user_load(&$user) {
  // Calculate the DN for the user -- you'll need to adjust this to match your LDAP base DN
  $ldap_dn=sprintf("uid=%s,ou=People,dc=somewhere,dc=outthere", $user->name);
 
  // Create a new array with the two LDAP-specific values that seem to be missing.
  $forced_data=array('ldap_authentified' => 1, 'ldap_dn' => $ldap_dn);
 
    // It seems like this should work, but it doesn't (it throws a segmentation fault)
    //  user_save($user_edit,array($forced_data);
    // so we're going to interact directly with the database
  if ($user->uid) {
    // Get the 'data' field for the user and put it in the $data array
    $data = unserialize(db_result(db_query('SELECT data FROM {users} WHERE uid = %d', $user->uid)));
    // Put all of the attributes from $forced_data into $data
    foreach ($forced_data as $key => $value) {
      $data[$key] = $value;
    }
    // Reserialize the $data array and update it in the database
    $v[] = serialize($data);
    db_query("UPDATE {users} SET data='%s' WHERE uid=%d",array_merge($v,array($user->uid)));
  }
}
?>

Save this as ‘olinkldap.module’, update the DN to reflect your LDAP server’s base DN (see comment in code), copy it into your Drupal modules directory, and activate it. Your ‘@’-impaired userids should start working again. If you are using the inter-Drupal account sharing (we’re not) this might break something for you. That’s not interesting for us, so I’m not testing it against that condition. If you use this and find that it works or doesn’t work, or you have a better way of solving the problem, please leave a comment or traceback…