Earlier this year, Index Data started a new blog. I wanted to call out a series of posts (parts one and two) their newly started blog called Z39.50 for Dummies. Wolfram Schneider on the Index Data staff wrote these two posts.

In the introductory posting, Wolfram says:

Z39.50 is a client-server protocol for searching and retrieving information from remote computer databases. It is a mature low level protocol like HTTP and FTP. You don’t implement Z39.50 yourself, you use the YAZ utilities and the libraries and frameworks for in other languages (C++, PHP, Perl, etc.).

There are many people who thinks that Z39.50 is a dead standard, and hard to understand. That is not true. Z39.50 is still growing in use, stable and very fast. It is the only widely available protocol for metasearch.

Using Z39.50 is not harder than using FTP. I think that the overhead for learning Z39.50 is less than a half day for an experienced programmer. Every problem which you have later is not related to the Z39.50 protocol itself, it is related to underlying system behind the Z39.50 server. Keep in mind that Z39.50 is an API to access (bibliographic) databases. It does not define how the data is structured and indexed in the database.

I haven’t had to create an FTP client from scratch (which, looking at its standard document, I would propose would be a complicated task), but I have done HTTP before. Z39.50 is cumbersome enough to not want to do it from scratch, but the YAZ toolkit is a great abstraction from the underlying protocol complications. I would second Wolfram’s observation that difficulties with Z39.50 are most likely problems with the configuration of the server as well as underlying data structures and data encoding.

Thanks for taking on the “for Dummies” task, Wolfram, and I’m hoping to see more in the series — how to troubleshoot connections, what all of the codes in the standard and the community profiles mean, perhaps some insight into how different flavors of Z39.50 servers operate, etc.

Boundaries are being blurred between the academic and commercial Web, between library resources, between the citation and the item itself. Students have no patience with these arbitrary boundaries; they want information, and they want it now, wherever it may be located.¹

Earlier this year, the New England Law Library Consortium (NELLCO) announced that they had received a grant from the Institute of Museum and Library Services to build a “Universal Search Solution” — a ‘one-box’ search into a unified index of a range of electronic resources. Indexed databases include OPACs, subscription-based resources, and selected free web resources. It is a two year grant to build and implement the tool for NELLCO members and release the code into open source. Index Data will be contracted to build the tool.

The tool “will be based on open standards and open source software, and will result in the creation of a … master index of material, including participating library catalogs, as well as subscription-based databases and open content, special collections, and other resources that a participating library wishes to make discoverable to its patrons.” ² The project specifications include the concept of differentiating search results based on the user’s institutional affiliation — search results from the user’s OPAC and hits from commercial data sources in the unified index that are limited by license to just that user’s institution. “Technologically, the Universal Search Solution will combine multiple technologies (consolidated indexing and data storage; metadata harvesting; and metasearching) to put together a single window to disparate resources.”³

I spoke with Tracy Thompson earlier this week about the project. The concept is an out-growth of an earlier pilot to use a Google Search Appliance (GSA) to create the unified index. The pilot was technologically successful, but ran into problems in the business model for the GSA because the pricing was based on the number of “documents” and from the GSA perspective each record indexed in a database was a document. (To give a perspective, the GSA page says the base model “starts at $30,000 to search up to 500,000 documents”⁴ — and how many bibliographic records are in our OPACs, just as a baseline for costing out a GSA?)

Current Status

In broad strokes, the first year is geared mainly towards building the tool. In the second year, the IMLS grant will fund the implementation of the tool for interested NELLCO members. After the grant period, NELLCO expects they will be able to sustain the implementation by subscription fee from the members. As described earlier, this is a consortial product/service, so there are already some economies of scale to be gained by not implementing redundant hardware/support at many local institutions.

A Different Model for Open Source Development

The grant also offers a new model for libraries to obtain affordable software services that are under their control. It combines the powerful financial leverage of IMLS and the organizational capabilities of NELLCO and its membership with the software development expertise of Index Data to bring cutting edge open source software and services to all types and sizes of libraries — affordable and with commercial support — but without vendor lock-in.

The tension between the inherent values of commercial, closed-source software vendors and higher education institutions (with an acknowledging nod towards commercial support for open source solutions) has been discussed on DLTJ before. The IMLS/NELLCO/Index Data collaboration suggests a new way for software tools to be built: a granting agency funds the initial hurdle of software development, and then the project transitions to self-supporting either through subscription charges that are in the self-interest of participants or through the sweat equity of participant-developers.

The text was modified to update a link from http://www.nellco.org/IMLS/Press%20Release.pdf to http://www.nellco.org/index.cfm?fuseaction=Feature.showFeature&featureid=21&pageid=4 on January 20th, 2011.

The text was modified to update a link from http://www.nellco.org/IMLS/Press%20Release.pdf to http://www.nellco.org/index.cfm?fuseaction=Feature.showFeature&featureid=21&pageid=4 on January 20th, 2011.

The text was modified to remove a link to http://www.uwec.edu/library/aboutUs/who/cox.htm on January 20th, 2011.

Footnotes

1. Cox, Christopher. An Analysis of the Impact of Federated Search Products on Library Instruction Using the ACRL Standards. portal: Libraries and the Academy. 6(3), July 2006, pp. 253-267.
2. From the NELLCO press release.
3. From the Index Data press release.
4. This figure is on the GSA page at the time this posting is being published, but it is also on the 8-Aug-2007 copy of the page in the Internet Archive.