When I say “<blank> is a question answering system. A question can be posed in natural language and … <blank> can come up with a very precise answer to that question” — what comes to mind to fill in the <blank>? If you guessed a system developed by IBM to appear alongside human contestants on Jeopardy, you’d be right. That quote comes from video posted by IBM earlier this year that is the topic of the first DLTJ Thursday Threads entry. This weeks other entries look at possible erosions of copyright first sale doctrine, the state of open access publishing, and a proposition for new definitions to terms of art in data modeling.

If you find these threads interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the right. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments and tips, as always, are welcome.

Reference Librarian of the Future? IBM Supercomputer ‘Watson’ to Challenge ‘Jeopardy’ Stars

An I.B.M. supercomputer system named after the company’s founder, Thomas J. Watson Sr., is almost ready for a televised test: a bout of questioning on the quiz show “Jeopardy.”

I.B.M. and the producers of “Jeopardy” will announce on Tuesday [December 14, 2010] that the computer, “Watson,” will face the two most successful players in “Jeopardy” history, Ken Jennings and Brad Rutter, in three episodes that will be broadcast Feb. 14-16, 2011.

For I.B.M., “Watson” is an important test of artificial intelligence. Scientists there have been talking to “Jeopardy” about a man vs. machine match-up for the better part of two years. “If the program beats the humans, the field of artificial intelligence will have made a leap forward,” John Markoff of The New York Times wrote in April 2009.

“Reference Librarian of the Future?” is the question Bernie Sloan asks as he forwards news of this competition to the Next Generation Catalog for Libraries mailing list. The quote above comes from the New York Times Media Decoder blog post that describes the competition, and the four minute YouTube video posted by IBM in June 2010 describes and demonstrates “Watson” in action in a mock competition. Is “Watson” good enough to replace a reference librarian backed by a slew of resources? Maybe — particularly if speed of an answer isn’t valued as much as accuracy. (Some of Watson’s “goofs” in the video are pretty funny, but may come as a result of trying to get to an answer too quickly.) Whether such technology is now affordable is an entirely different question.

Is the sky falling on library lending?

First sale is the rule that once a lawful copy of a work is sold, the exclusive right to control distribution of that copy is “exhausted.”  Therefore libraries can lend books, consumers can resell CDs and NetFlix can rent DVDs through the mail.  First sale is not a necessary or automatic part of a copyright law; many countries have different provisions on what is sometimes called “exhaustion,” such as a statutory fee for each library loan of an item.  Most importantly, first sale does not apply when a work is licensed rather than sold.  Many of the current threats to first sale are controversies over where the boundary between a sale and a licensing transaction really is.

Here is a catalog, quite long I’m afraid, of some of these controversies and, I think, threats to library lending.

Without strong copyright “first sale” doctrine, the foundation for much of what we do as libraries could be erroded faster than patrons are taking to internet search engines over our library services. Okay…that statement is hyperbolic, but Kevin Smith’s post at Duke University’s Scholarly Communications blog can make one wonder how fast that end is coming. First on the list is the UCLA video streaming case mentioned in last week’s DLTJ Thursday Threads. (By the way, no new information from the court on that case this week.) Keven goes on to mention several other cases that may impact how first sale doctrine is applied to libraries. [Via Barbara Fister at Inside Higher Ed]

On the State of Open Access Publishing

OA self-archiving has come to be called the “green” road to OA (or “Green OA”), to distinguish it from OA journal publishing, which is called the “gold” road to OA (“Gold OA”). The most frequent misconception about OA is that OA only means Gold OA (publishing).

Stevan Harnad posted an announcement of this pre-press version of an article of his to appear in a future issue of Logos Journal. Although some of my colleagues on FriendFeed take issue with the premise that is the title of the article — “Gold Open Access Publishing Must Not Be Allowed to Retard the Progress of Green Open Access Self-Archiving” — I think first half of the article gives a concise overview of the history of open access publishing and sound definitions of “Gold” open access publishing (where the publisher makes the articles freely available) and “Green” open access publishing (which is author self-submission in public archives). [Via Celeste Feather]

Different Kinds of Data Models: History and a Suggestion

In this article, David C. Hay tries to redeem himself for some of his contributions to controversy in the data modeling world.

So, where does this leave our original problem with conceptual and logical models? I hereby modify my original organization described above. Harking back to the original ANSI ideas about the “External,” “Conceptual” and “Internal” schema, as updated by the upgraded Zachman Framework, I propose the following definitions:

• Conceptual Model – Any model that describes the business. It may be one of the following:
  • Strategic Model – This may be a model of basic terms, linked with many-to-many relationships, if desired, but focusing on establishing basic categories.
  • Business Owner’s Model – This is about the semantics of the organization. If appropriate, entity/relationship models can be developed, but more useful is an SBVR analysis, and OWL descriptions. This is the “Semantic Model” (the “external schema” in the original ANSI view).
  • Architect’s Model – This is an entity/relationship model of fundamental entity classes, encompassing as much of the enterprise coherently as possible. This is the “Architect’s Model” (the “conceptual schema” in the original ANSI view).

• Technology Model – Any model that reflects the technological environment being addressed. It may be one of the following:
  • Designer’s Model – In the data world, this is the model that accommodates the technology being used for data management. It may be in terms of tables and columns, object-oriented classes, dimensions, XML tags, or whatever. This is the “Designer’s Model” (the “logical” part of the “internal schema” in the original ANSI view).
  • Builder’s Model – This is the configuration of physical databases, tablespaces, or even cylinders and tracks of the physical database. The builder is the one who spreads the “People” table over three continents (the “physical” part of the “internal schema” in the original ANSI view).
  • The working system

For all of the developer geeks in the DLTJ community is this article by data modeling luminary David Hay. What is the physical model versus the logical model, and how are they related to the conceptual model? The answer varies depending on who you ask (and who’s writings you might be a disciple of). In this brief note, David proposes new definitions for these concepts in the hope of moving the conversation forward. [Via Ron Murray]

Are we building the “next generation” catalog for us (librarians) or our users? As a read a report from the Next Generation Summit Search Interface Working Group of the Orbis/Cascade Alliance, I have to wonder. Portions of this report are dated¹ other portions are timeless. In particular, this section from page 2 (emphasis added):

How do we define “next generation”?

The working group has considered what it means to create a “next generation catalog” within the context of the current Summit interface and the current definition of “next generation” as understood within the library community. However, maybe this isn’t the right question. In part, library systems have failed to even keep up with our current generation of users, with neither the library community or vendor community really understanding how a current generation catalog might function. We have ideas from looking at vendor sites and social software tools that provide tagging, faceted browsing, and user reviews, but are these really “next generation”? No, they represent current generation functionality that library systems simply have yet to assimilate into their current service offerings. It’s a dangerous confusion of vocabulary. While these services represent “next generation” services for the library community, they don’t for our users. If a simple makeover of the ILS is to be our aim, then we will continue to fail to provide services for our current generation of users. Our current library information systems are failing our users and inhibiting our users’ attempts to build communities around our services and systems.

Libraries should rectify this problem by seeking to build systems that meet the needs of this current generation, while allowing the library community to plan for and implement functionality that will be necessary within the “next generation”. In part, this is what some libraries are doing — some examples are discussed in this report. North Carolina State University’s utilization of Endeca has been lauded far and wide, but in essence, they’ve simply started to catch up with today’s current generation of users. Yet in just catching up to the current generation, they have distinguished themselves from the rest of the library community. They have placed themselves in a position to look beyond the needs of the current generation of users and focus on the services and needs of the next. At this point, few organizations, including the Alliance, can make such a claim. That, in part, is the challenge facing the Alliance and this working group as it made its assessments.

It would seem helpful to shift terminology, because in doing so we can focus more readily on the needs of our users. They aren’t looking for a “Next Generation” interface. “Next Generation” to us means “Current Generation” to our users. “Next Generation” for our users is “what have you done for me lately?” In order to meet that need, we need a platform that is “developer-friendly.” Again, from the report:

a platform that supports and encourages interaction with the system. This can take many shapes, including OAI harvesting, SRU, OpenSearch or a simple web-services-based API to allow the Alliance to take a more proactive role in developing services.

And, finally, in the report’s recommendations:

Additionally, in researching current and in-development solutions, it became clear that if the Alliance is to continue to meet the needs of its users, it will have to demand greater access to the metadata (holdings, items, bibliographic) found within the catalog. Regardless of who provides the Alliance’s next generation OPAC product, one of the deliverables that must be available as part of any solution is API or web services access to the catalog. Access at this level is important for two reasons:

1. It allows libraries to integrate and share development resources:
   The Alliance members are currently hamstrung by the closed nature of the Summit catalog. The Summit INN- Reach catalog currently only provides two methods of interaction – Z39.50 and HTML access. For developers looking to build services around the Summit catalog, the Z39.50 protocol, as implemented, is currently too limiting and expensive for production development services. All major ILS vendors but III provide their customers a web services or HTTP REST API access to their systems, allowing for continued development around the catalog. Lacking such access, the Summit catalog will continue to be marginalized within the consortium’s academic campuses as tools and services are developed that take advantage of web service friendly applications.
2. Allows for the development of library-created or user-created mashups:
   The Alliance should strive to create a resource that encourages users, libraries, and campuses to develop services around the Summit catalog. The library community has recognized that our patrons want social tools, which we tend to identify as tagging, commenting, etc. However, Web 2.0 applications like Flickr are popular because of the API access that they provide to their users as well. This access has enabled other web services, individuals, and organizations to develop different methods for exporting and utilizing the images placed within the Flickr photo archive. The Alliance should strive to make the Summit catalog open in this way, so that users and members alike are free to enhance Summit to meet individual, campus, or consortial needs.

If it is possible to sum up what we need in “next generation” systems, I haven’t seen more succinctly put than this report.

Footnotes

1. Although the report itself does not contain a date, mention of the report appears in the agenda of a March 2007 meeting of the consortium’s Summit Catalog Committee.

Schemes to add functionality to the web OPAC fall into four categories: web OPAC enhancements, web OPAC wrappers, web OPAC replacements, and integrated library system replacements. I’m outlining these four techniques in a report I’m editing for an OhioLINK strategic task force and a bit of a reality check on this categorization is desired, so if I’m missing anything big (conceptually or announcements of projects/products that fall into these categories), please let me know in the comments. Generally speaking, this list is ordered by cost/complexity to implement — from lowest to highest — as well as the ability to offer the described enhanced services from least likely to most likely.

Web OPAC enhancements are functions that are added to the existing web OPAC system. This most often entails additional product purchases from the automation vendor, such as the optional enhancements in WebPAC Pro for Millennium OPACs or content solutions in SirsiDynix. Enhancement can also be added through creative use of an existing web OPAC’s template functions, such as the method by which LibraryThing for Libraries can be added to OPAC displays.

Web OPAC wrappers use the existing web OPAC provided by the integrated library system as a source of information, but hide that information behind a completely new interface. The intervening system get that information from the integrated library system through a variety of mechanism. In some cases, it may be possible to use established protocols (such as Z39.50) or programming interfaces (such as an XML content server). In cases where such functionality is not available from the underlying integrated library system, a “screen-scraping HTML” technique may be required. ¹

One example of such a wrapper is the work at Ann Arbor Public Library on SOPAC. Short for “Social OPAC,” SOPAC is “a set of social networking tools integrated into the AADL catalog [that] gives users the ability to rate, review, comment-on, and tag items.”² It uses an open source content management system called Drupal as a structure through which the added functionality is provided. For example, when a user seeks the bibliographic information page for a catalog record, that request is made from the user’s browser to the Drupal software. The Drupal software in turn makes a request to the integrated library system for the bibliographic information it holds. The response from the ILS is parsed by the Drupal software for key information such as title, author, subjects, holdings, etc. This information is mixed with information stored in the Drupal database (ratings, tags, reviews, cover images, etc.) and a new web page is created and returned to the user’s browser.

Another example of a web OPAC wrapper is Scriblio (formerly called WPopac). Using the underlying framework of WordPress, Scriblio offers faceted browsing, tagging, and syndication feeds for the underlying Millennium WebOPAC. Scriblio is a project of Plymouth State University, supported in part by the Andrew W. Mellon Foundation. Both SOPAC and Scriblio are available under open source licenses.

Web OPAC replacements are new systems that completely replace the existing web OPAC. Unlike wrappers (which get their bibliographic data in real-time from the underlying web OPAC), these replacements operate on sets of records that are extracted from the ILS or come from another source. (In some cases, these replacements still rely on the underlying web OPAC as a source of item status information such as checked out status and due date.) The first notable OPAC replacement was at North Carolina State University when its library installed and configured the Endeca software to provide a faceted browse to the library catalog. By itself, an Endeca OPAC display does not enable tagging, annotation, or user aggregation services such as recommendation engines.  Other similar web OPAC replacements are Encore from Innovative Interfaces³, Primo from Ex Libris and Aquabrowser from Medialab Solutions. Miami University’s experiments with the open source Apache SOLR and the exported records from their Millennium system also fall into this category. Worldcat Local is also a form of web OPAC replacement noting that the source of bibliographic records is the OCLC Worldcat database rather than the local ILS.

ILS replacements offer the biggest opportunity for enhanced user services, particularly by adopting one of the open source solutions now available. At this time, neither of the open source solutions (Evergreen and Koha) offers more than faceted search and browsing. Unlike the commercial systems, however, the source code of the system can be modified to add these functions, and the modifications shared with other users of the same system.

[Update 20071015T1624 : Corrections made -- and the text improved! -- based on Betsy Graham's comment. Thanks, Betsy!]

The text was modified to update a link from http://www.iii.com/mill/webopac.shtml to http://web.archive.org/web/20071015145500/http://iii.com/mill/webopac.shtml on January 20th, 2011.

The text was modified to update a link from http://www.iii.com/encore/main_index2.html to http://web.archive.org/web/20080328163000/http://www.iii.com/encore/main_index2.html on January 20th, 2011.

The text was modified to update a link from http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content to http://web.archive.org/web/20071109170053/http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content on January 28th, 2011.

Footnotes

1. Such a technique gets the information from the ILS using the existing web OPAC. Such schemes are generally fragile because changes to the underlying web OPAC can have detrimental affects on the content scraping process.
2. Blyberg, J. (2007). AADL.org Goes Social. blyberg.net. Retrieved October 12, 2007, from http://www.blyberg.net/2007/01/21/aadlorg-goes-social/
3. As Betsy Graham, Vice President of Product Management at Innovative Interfaces, notes in the comments, the Encore will perform real-time queries to a Millennium ILS for bibliographic data, and in such cases the data extract is not needed.

Earlier this week, Aaron Swartz of the Internet Archive announced the demonstration website of the Open Library project, a new kind of book catalog that brings together traditional publisher and library bibliographic data in an interface with the user-contributed paradigm of Wikipedia. Okay, I’ll pause for a moment while you parse that last sentence. Think you got it? Read — and watch — further.

Open Library has been mentioned a bit in the blogs this week, but not to the extent I thought was worthy of the magnitude of the project. So I recorded a screencast introduction (in Flash Video format below followed by a rough transcript) that looks at not only the browsing side of the system but also the record editing and record creation aspects of Open Library. As I say at the end of the recording, Open Library is one of those mind-bending, assumption-shattering projects that, at least for me, is challenging my thoughts about what library service could be and should be. Congratulations to the team at the Internet Archive, and I’m looking forward to future enhancements and directions for the project.

Rough transcript of the screen cast is below.

Introduction

Hello, and welcome to this screencast overview of the Open Library project. Open Library is an effort by the Internet Archive to create a comprehensive catalog of every book. As the project’s “about” page says, “Not every book on sale, or every important book, or even every book in English, but simply every book.” The about page goes on to describe the characteristics of Open Library project — that it is a project enabled by Internet technology because no physical space could hold it and that it aims to pull together records from publishers and libraries. It is also a project in the same vein as Wikipedia, meaning that any user can create and edit the records in the system.

In this overview, I’ll lead you through searching and browsing the Open Library’s demonstration website from the perspective of any modern library catalog interface. Then I’ll show you where it deviates from traditional library catalogs by exposing the underlying wiki nature of the database; we’ll examine the changes that users have made and we’ll even make a change ourselves. And finally I’ll show the process of creating entirely new records in the system. So let’s get started.

Searching

We’re looking at the home page of the Open Library project demonstration site. In the middle is a search box with a suggested search — “tom sawyer adventure”. That is a good suggestion so we’ll click on Go. Open Library returns a classic, relevance ranked list of matching records with some book covers along the left side and a faceted list of refinements along the right. So right away you can see that there are some authority control problems here in the author names — Twain comma Mark, Mark comma Twain, and Twain comma Mark with birth and death dates — and here in the language field. But I have high hopes that the developer team will find some intriguing ways to address these problems.

Back over here in the results area we have the various editions of Samuel Clemen’s “The Adventures of Tom Sawyer” — let’s pick the 1876 edition to see the full record display — there. We have the publisher, publication date and place, language, and a summary or review of sorts at the bottom. We also see signs of the availability of full text — over here in the options box there is a download from the Internet Archive link, a “Scan Sponsor” field here and a “View this book” graphic. This is one of the items scanned by the Open Content Alliance and made available by the Internet Archive through the Open Library project. A very nice interface for paging through the book. So one could imagine that the Open Library could become the primary vehicle by which Open Content Alliance materials are made available to the public.

So let’s go back here to the metadata page. Remember in the introduction that I said that the data was malleable in a wiki-like fashion. The Open Library developers created a system that allows for user-contributed updates (a la Wikipedia) to fielded data (like your classic bibliographic record). The two hints that the record is modifiable are this big edit button in the middle of the metadata and this more subtile “[history]” link near the top of the page. Let’s start with the history link to see what has been done to this record.

This page should look familiar to those who have worked with wikis before. It shows a listing of edits that were made to this record from most recent to the very first edit, who made the change (identified by IP addresses in this case because the people making the changes were not logged to an account at the time), an editor-supplied comment about what was done, and when the change was made. We can go back in time and see the page at a particular version through the links under the “When” column, or we can use the compare function to see the difference between two version. In the case of the changes between version 2 and version 3, we see that the editor added “Canada” as the place of publication. On this page you start to see the fielded nature of this wiki structure, but the best place to see it is look at the record edit screen itself.

These are all full-text fields on this page with no controlled vocabulary. You’ll note the absence of any MARC field names here, but as you scroll through you’ll see the evidence of MARC and AACR2 in the field labels. Down at the bottom is an edit summary to describe the changes made to the record, then save, preview and delete version buttons — all classic wiki functions.

Editing

Now, I’d like to show the full record editing process, but since I don’t have this Mark Twain book in hand, I’m going to bring up another record that I created yesterday — “Eric Meyer on CSS“. Before showing the editing process, let’s linger here a moment at the “options” box along the right side. Since this is a more modern book (as opposed to the Tom Sawyer book we saw first), there are additional options here for purchasing the book through these various vendors or borrowing the book through a very nice link into Open Worldcat and two web-based book trading sites.

But back to the metadata. There is one error and one omission in this record — perhaps this is a subtile demonstration of problems that creep in with user-generated content. First, the error, is that there is an extra digit in the ISBN-10 field, which is a big problem because the links in the options box use the ISBN as a linking field and at the time of this recording they don’t work. They will work in a moment, though. The second problem is that I forgot to put in the publication date. But hey, no problem, all I need to do is “Edit” this record.

So we are back to the edit screen, and I’m going to scroll down and fix the ISBN-10 field like so, then scroll down a little further and add the publication date. Then I’ll scroll all the way to the bottom and type in an edit summary — “Fixed the ISBN and added a publication date” — and hit save. We’re now back at the metadata display screen and the link to Open Worldcat now works. So, as an aside, one wonders what the folks in Dublin, Ohio, think about this. It is competition on the one hand since Worldcat is also aiming to be the most comprehensive catalog of books in the world. On the other hand, perhaps there is room for cooperation by somewhat getting vetted changes to Open Library records into the OCLC union catalog. Who knows?

Creating a New Record

Alright, back to current reality. Let’s add a record to Open Library, and in this case I’m going to use an ARL SPEC Kit that I wrote a number of years ago called “Library Patron Privacy”. First let’s run a search in Open Library to see if it is there, and no, it isn’t. The only way I’ve figured out how to enter a new item is to go to the URL where the page would be located and get the classic wiki “This page does not exist. Create it?” message.

One of the quirks I found in the system is that I have to create author wiki pages before book wiki pages — otherwise I’ll get a Python error message on the screen. I’ve reported this to the Open Library developers, but in the meantime just know authors need to be created before their books. Which is to say that authors have wiki pages in Open Library in addition to books. The structure of URLs to Open Library author pages is the letter “a” followed by a slash followed by the author’s last, first and middle names separated by underscore characters. So I’ll go to the URL of that form, then click on the “Create it” link.

Now here is one of the tricky parts of the existing interface. The page type starts as “type/page”, and as you can see it doesn’t have any of the fielded elements that we saw in previous examples. What you have do do is change the page type to “type/author” and then you get the fielded HTML form. So I’m going to go through here and fill in some of the parts. Then go down to the edit summary field and write a summary of this change, then click save. Now that Open Library knows who I am, let’s create the record for the book.

You’ve seen the structure of the URLs to book pages before — a “b” followed by a slash followed by the book title with spaces replaced by underscore characters. I’ll put that in the URL field and get the default page type. This needs to be changed to “type/edition” in order to get the bibliographic record fields. There. Now I’ll go through here and enter the data. When we get down to the author field we enter it in the same format that we used to create it — an “a” followed by a slash followed by the name with spaces replaced by underscores.

So we’ll just finish up here and come down to the edit summary field, put something in here, and hit save. This record is now in the system, and you can see the public display here along with the links on the right because I entered an ISBN. I haven’t quite figured out how to get a cover image into the system yet — I expect there is a file upload interface somewhere, but I haven’t found it.

Conclusions

So that’s all there is, and I don’t say that in a way to denigrate the work that has been done by the development team so far. As the URL and site banner indicate, it is a demonstration system — and a compelling demonstration it is. All sorts of questions immediately come to mind, of course — will there be a controlled vocabulary or authority control built into the system, can data be exported out of records — and, for that matter, can end-users bulk import data into the system, are there Web2.0 niceties like tagging and RSS feeds in the works, and so forth.

Even with all of those questions, Open Library is one of those mind-bending, assumption-shattering projects that, at least for me, is challenging my thoughts about what library service could be and should be. Congratulations to the team at the Internet Archive, and I’m looking forward to future enhancements and directions for the project.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer?m=history on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer?b=3&a=2&m=diff on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/Eric_Meyer_on_CSS on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/a/Murray_Peter_E on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/Library_Patron_Privacy on January 19th, 2011.

No Service Oriented Architecture posting today, but here is a glimpse of the topic of the next one — the title is: “Web Services: A means to a Service Oriented Architecture end.” In the meantime I wanted to thank everyone for their public and private comments, and to ask to keep ‘em coming. The big push for writing about SOA this week was a lead up to a meeting of the OhioLINK Technical Advisory Council (TAC) today. On TAC’s agenda was a question about looking at SOA as a design strategy for new and migrated services. These blog postings served several purposes: 1) propel the topic a little further in the library community [presupposing that it was a worthy topic]; 2) serve as background information for today’s meeting; 3) flush out comments from the library community [which it did -- thanks again!]; and 4) form the basis of a whitepaper on SOA at OhioLINK. TAC agreed to keep looking at it and endorsed the writing of the whitepaper. Keep the comments and observations coming!

On a somewhat related note, I wanted to tie up a loose end from this summer:

In academic libraries, in my experience, there has been a decline in the use of library catalogs. This experience could be verified in the ARL supplementary statistics for at least that population of libraries (I think those numbers are password-protected, so it might be a challenge to try to use them). When I get back on the ground and have some time, I will either offer confirmation of that supposition or retract it.

DLTJ “Is the Writing On The Wall?” — Take 2, Wednesday, June 14th, 2006

I retract the statement. Although it has been true in my direct experience, I cannot find any statistics — for either academic libraries or the broader community — to back up that experience. Thanks, Walt, for calling me on it and keeping me on the straight-and-narrow.

This is part three of a continuing series on the application of the Service Oriented Architecture (SOA) design pattern to library systems. In the first part, the SOA concept was compared to a transportation network and the basic foundation for defining SOA was set down. The second part described what a “service” in SOA could be and proposed an example using OCLC’s WorldCat interface with item status information being pulled from a library catalog system. That part also left off with a teaser about the juxtaposition of “inventory control system” with “local catalog system” — a foreshadowing of the topic of this post: what to do about the Monolithic (er… “Integrated”) Library System.

How did we get where we are today?

Before the advent of computers, libraries frequently used a card catalog to index its holdings. Computers were used to automate the card catalog, thus the term automation system. Since the late 1980s, multi-tasking modules allowed business functions to be integrated. Instead of having to open up separate applications, library staff could now use a single application with multiple functional modules. ¹

As this article points out, libraries have a grand tradition of applying technology to its business problems. These “integrated library systems” (ILS) — so named for their unified functions of public search, circulation (check-in/check-out), acquisitions, and cataloging (description) — propelled library staff productivity to magnitudes not seen before. “Those systems,” notes Marshall Breeding, “came quite close to delivering comprehensive automation for libraries.”²

So why change? Marshall goes on to say, “the ILS remains shackled to the antiquated print-based paradigm for library automation.” Today’s integrated library system is ill-equipped to handle the desired integration of formats (audio, video, journal article) and delivery mechanisms (online versus on-site) of our users. It is also not a nimble platform; as a single, all-encompassing application, bug fixes and new functionality across the entire must be wrapped in a software distribution to be released to users.

This post is not intended to be an exhaustive examination of the problems with the software design model of current ILS applications; there is a growing body of analysis and commentary on that point. Rather, the rest of this post seeks to point a way forward — a path that involves the dis-integration (“un-integration”? or, as Marshall put it, “re-integration”?) of the integrated library system.

What is the Integrated library System?

Discovery Tool / Indexing Tool

Speaking of Roy’s mantra, I may be showing a bit of a bias here towards one particular solution by pairing the Discovery Tool with an Indexing Tool. The underlying presumption here is that there will be One Common Index for all library content. Given the issues surrounding metasearch (particularly the speed at which a metasearch query can be executed), getting a zippy response for a SOA interaction really requires that an index already exist that is used to answer the query. This bias is fueled, admittedly, by OhioLINK’s fortunate position of storing and serving much of the content under license from publishers/producers from our own servers. When you actually have access to the content and underlying metadata, you can think about creating One Common Index for all of that content. So use of the Discovery Tool is inexorably linked to the Indexing Tool.

Inventory Control Tool

A library’s inventory control challenge is modestly similar to that of a video rental store: we need to track what is within our confines and what has been “rented.” That leads us right into the next tool.

Point-of-Sale (a la video rental) Tool / User Directory Tool

If that analogy holds, then their customer database is like out patron database. In any case, we will need a User Directory Tool where demographic information is kept as well as information about users’ “rentals.” The User Directory Tool can also be a source of authentication and authorization as well as other functions such as storing recommendations and annotations. (More on this topic to come at a later time.)

Ordering and Receiving Tool

Description Tool

Unlike the integrated library systems in use today, however, our next generation Description Tool must be multi-lingual: MARC ⁶, while very expressive to the point of obsessive, cannot capture all of the information about every object. If metadata is going to truly encapsulate the descriptive fidelity of the object, we need to look at standards other than MARC. For example, OhioLINK’s current media content repository has object that are digitized bird calls; the description of those audio files includes elements such as genus and species, the type of microphone used to record the bird call, and a notation of whether the bird was seen or not seen. Odds are those elements will never find their way into MARC, so the Description Tool must be able to pull together schema such as the Nature Services Species Schema, some other schema that describes recording devices, and something that we’ll likely have to make up to represent whether the bird was observed. (See, I told you we were obsessive about description.) In addition, a really good Description Tool would pull in authority files and ontologies of various sorts, beyond the Name Authority File and Library of Congress Subject Headings that enable similar topics to be grouped together. Above all, this tool needs to be easy to use to promote a quick workflow and response SOA Service requests.

What’s Next

Thanks go out to Richard Wallis at Talis and Lorcan Dempsey at OCLC for their comments so far. I’m on the home stretch towards a plateau in this series and which point I’ll take a step back, absorb your observations, and reflect them back out in subsequent postings. Everyone else, please feel free to join in on the conversation as well, whether it be through comments here or commentary elsewhere.

[20060921T0820 Missed the "p" in Lorcan's last name -- my humble apologies, Lorcan.]

The text was modified to update a link from http://en.wikipedia.org/wiki/Integrated_library_system&oldid=76840266#History to http://en.wikipedia.org/wiki/Integrated_library_system?oldid=76840266#History on January 13th, 2011.

Footnotes

1. Adapted from “Integrated library system.” Wikipedia, The Free Encyclopedia. 20 Sep 2006, 19:29 UTC. Wikimedia Foundation, Inc. 20 Sep 2006 http://en.wikipedia.org/wiki/Integrated_library_system&oldid=76840266#History.
2. Breeding, Marshall. “Re-Integrating the integrated library system.” Computers in Libraries. 25(25): Jan 2005. 20-Sep-2006 http://www.librarytechnology.org/ltg-displaytext.pl?RC=11340.
3. A note out to those of you outside the library community who still seem to be reading this series, if the blog statistics are to be any guide: you can help us out with your ideas here — many of us have had our heads buried in ILS systems for so long (my entire career, for instance) that it is difficult to think outside that box. Let me know in the comments if I’ve missed anything.
4. “A Conversation with Werner Vogels.” ACM Queue. 4(4): May 2006. 19-Sep-2006 http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=388
5. For a definition and description of Point-of-Sale systems, I recommend this Wikipedia article.
6. Machine Readable Cataloging

This post is the second in a series about the application of the Service Oriented Architecture (SOA) system design pattern to library services. The first post in this series focused on defining “Service Oriented Architecture” using the analogy of a transportation network. This post goes into some detail about what makes a “service” in this architecture and offers an example using a hypothetical use case: a union library catalog (Open WorldCat) making a statement about the availability of a book.

Why are we even looking at SOA?

SOA is a means of organizing solutions that promotes reuse, growth and interoperability. It is not itself a solution to domain problems but rather an organizing and delivery paradigm that enables one to get more value from use both of capabilities which are locally “owned” and those under the control of others. It also enables one to express solutions in a way that makes it easier to modify or evolve the identified solution or to try alternate solutions. ¹

The first sentence is a statement that reflects SOA position as the latest step on an evolutionary path from monolithic to modular computing. In the really old days of electronic computing, every sequence of processing instructions was defined from scratch — you programmed the computer to do everything. Very quickly one realized that the programming code being written contained a lot of redundancy, so “code libraries” were formed that held all of the repeated instructions; no longer did everyone have to write and debug their own code routines to get the current date or write out a character…you used the supplied code libraries.

Gradually, as systems became more and more complex, the computing profession came up with more code libraries and developed higher levels of abstraction to deal with the complexity. “Objects” as programming entities were introduced in the 1960s via Simula 67, a programming language designed for making simulations, and Smalltalk, a programming language developed at Xerox PARC in the 1970s, introduced the term Object-oriented Programming (OOP) to represent the pervasive use of objects and messages as the basis for computation. OOP became the dominant programming methodology during the mid-1980s largely due to the influence of C++, an extension of the C programming language, but for the most part the applications developed using the OOP methodology remained as applications running on a solitary computer system. ²

OOP and SOA are similar in that each seeks to encapsulate activities to an abstract level at which point the abstractions can be reused as building blocks to larger applications. Where they differ, as explained by the OASIS Reference Model for Service-Oriented Architecture ³, is at the level of abstraction: in the OOP paradigm the focus is on packaging data and operations whereas in the SOA paradigm the focus is on “the task or business function &mdash getting something done.” Or, if you will, providing a service.

Definition of “Service”

In the SOA sense of the word, a service is a set of capabilities offered by a service provider through a service definition that makes the capabilities discoverable by a service consumer. The implementation of the service is “opaque” to the consumer, and the invocation of a service itself may seek out the SOA-defined capabilities of another service provider. The real-world consequences of a service invocation can be: 1) information transferred from a provider to a consumer (e.g. “read”); 2) a change in the state of information at the provider (e.g. “write”); or 3) a combination of both.

The amount of information received and/or changed per service invocation can be rather small, and the transaction itself can be rather light-weight. For instance, in an ACM QUEUE interview Werner Vogels, Amazon’s Chief Technical Officer says that when “you hit the Amazon.com gateway page, the application calls more than 100 services to collect data and construct the page for you.” ⁴ If it took seconds rather than miliseconds to respond to a SOA query, Amazon’s architecture would clearly be unsustainable.

Back up to the quotation near the top of this post, the second sentence ends by speaking of “capabilities which are locally ‘owned’ and those under the control of others.” SOA, by defining this abstract notion of service and focusing on the semantics surrounding a service, promotes the exchange of services across organizational boundaries. This is the subject of the following hypothetical use case.

Hypothetical Use Case: Reflection of Local Library Holdings in Open WorldCat

You are all ready to proceed with this plan, ⁵ but there is one sticking point: your local public catalog displays the status of the item (“available”, “checked out”, “lost”, etc.). And WorldCat, although it contains the combined holdings of thousands of libraries, it doesn’t know whether any particular item is available or not — the only thing stored in the WorldCat database is whether an item is owned, not whether it is checked out. The status of the item is only known to the local catalog system.

Enter a Service Oriented Architecture. The local catalog system makes known a service whereby an service consumer can ask for the status of an item. What is returned by the service provider can be as simple as a string of text that is a spelled-out version of the item status or it can be as complex as an XML schema with attributes for “checked out” or “lost” and parsable date fields and so forth. Remember, though — these SOA exchanges need to be processed quickly. As WorldCat is generating the screen of the search results for the user, it calls back to the library catalog system to find the status of the item and then displays it for the user. The result might be something like this:

Okay, but is That What the User Really Wanted?

Available Online

If we see in the description of the item that it is on the web somewhere. In this case, the word “Online” can link right to the item.

Available Onsite; Delivered in 1-2 days

This item is physically held in the library and the inventory control aspect of the library system says that it is available. So the choices to the user are “onsite”, which could link to a page that describes which library holds the item and the approximate location of the item within the library, or “delivered in 1-2 days” for a user service that will pull an item from the shelves and deliver it to an office or home.

Delivery on or about…

This item is checked out. A hold has been placed on it, and when the item is returned (hopefully no later than the original due date) it will be turned around to the next user.

Delivered in 3-4 weeks

This item was declared lost, so the 3-4 week delivery statement represents the estimated time to get the item from another library through interlibrary loan. Ultimately, does the user really care if the item was declared lost? Probably not — they just want to get to the item!

Layers of Services

Delivered in 2-3 days

The item was not available at the local site (so we don’t see an “on-site” link) but it is available from an OhioLINK member institution. The average turn-around time for an OhioLINK physical delivery request is about 2 to 3 days. But again, the user doesn’t care at this point where the item is coming from — their local library or one somewhere across the state; the only thing the user likely cares about is how long it would take to get the item if he or she asked for it to be delivered to their home or office.

But What of the “Integrated Library System”?

The text was modified to update a link from http://en.wikipedia.org/wiki/Object-oriented_programming&oldid=76627897#History to http://en.wikipedia.org/wiki/Object-oriented_programming?oldid=76627897#History on January 13th, 2011.

Footnotes

1. OASIS Reference Model for Service-Oriented Architecture, Committee Specification 1, dated 2 Aug 2006, lines 175-179. The document was retrieved from http://www.oasis-open.org/committees/download.php/19679/soa-rm-cs.pdf on 18-Sep-2006.
2. Summarized version of “Object-oriented programming.” Wikipedia, The Free Encyclopedia. 19 Sep 2006, 18:27 UTC. Wikimedia Foundation, Inc. 19 Sep 2006 http://en.wikipedia.org/wiki/Object-oriented_programming&oldid=76627897#History.
3. OASIS Reference Model for Service-Oriented Architecture, Committee Specification 1, dated 2 Aug 2006, lines 243-259.
4. “A Conversation with Werner Vogels.” ACM Queue. 4(4): May 2006. 19-Sep-2006 http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=388
5. For the record, please note the word “hypothetical” sprinkled liberally throughout this example. WorldCat cannot now do the things that are described in this post, the screenshots are unauthorized adaptations of WorldCat web pages, and, to the best of my knowledge, OCLC is not planning on doing anything like what is described here. Although it would be really neat if they could do it….
6. yeah, yeah — I can almost hear the sharp-witted speculations about our dinner-time conversations from those outside the library community.

What of a service existed where the patrons selected an item they needed out of our library catalog and that item was delivered to the patron even when the library did not yet own the item? Would that be useful? With the growth of online bookstores, our users do have the expectation of finding something they need on the web, clicking a few buttons and having it delivered. When such expectations of what is possible exist, where is the first place a patron would go to find recently published items — the online bookstore or their local library catalog? Does your gut tell you it is the online bookstore? Would it be desirable if the patron’s instinct were to be the local library catalog?

A savvy patron looking for a recently published item will likely try the local library catalog to see if the item has been selected, purchased, received, cataloged, processed, and shelved (hereafter “SPRCPS”) by the staff — in other words, gone through the traditional process libraries use for acquiring items. If not, the patron has one of three choices (that I can think of):

1. make a request for the item to be SPRCPS’d with a hold placed on the item so that the patron is notified when it is ready;
2. start an Interlibrary Loan process to try to get the item from another site that has SPRCPS’d the item faster than your library; or
3. pay a cost premium — buy the book themselves and have it delivered.

Looking at this from the perspective of elapsed time, #1 is likely many weeks, #2 is likely a few weeks, and #3 is likely a few days. Looking at this from the perspective of direct cost to the patron, #1 is the cheapest, #2 may be free or some nominal ILL transaction cost (depending on local policy), and #3 is the most expensive. All-in-all, reasonable tradeoffs.

But what if our libraries offered a service that had the speed of #3 and the cost of #1? Do you think that would be an appropriate service to our users?

Local Catalog Display

In terms of the mechanics of getting these records into our systems, it seems that we need a new form of MARC record loads into our systems. The most likely source? How about what booksellers use now — the ONIX format “that publishers can use to distribute electronic information about their books to wholesale, e-tail and retail booksellers, other publishers, and anyone else involved in the sale of books.” A feed of ONIX records from publishers, filtered through a selection criteria, converted into MARC21, and loaded into our local catalogs would do the trick.

Automated “Request This Item” Function

Speedy Copy Cataloging and Shelf Prep

• Use “on the fly” circulation procedures to lend the book out immediately. When it is returned route the item through technical services for formal copy cataloging (or decide that the Onyx data is acceptable as is for the copy cataloging).
• Use a distributor that will delivery the item shelf-ready. Just yesterday, through an LISnews posting, I learned that Amazon is now one such distributor:
  

  Amazon offers a wide array of library processing options. In addition to mylar jackets on hard-cover books, Amazon also offers MARC records, spine labels, and barcodes. By partnering with leading cataloging companies and organizations, Amazon is also able to offer you highly customized MARC records, spine labels, and barcodes that meet your specific needs.

New Roles for Staff

• Selectors/bibliographers still have front-end work to do. They are the ones to tune the profile of “items that could be purchased” records (informed by the requesting patterns from users) that are loaded into the system and to buy items not yet requested that round out a collection. This is modestly akin to the approval plan systems we use now.
• Copy cataloging staff may have a reduced workload for items that come in through the RTIB process — particularly if the distributor selected does much of the shelf prep and copy cataloging work already. This, too, is nothing new: we have been outsourcing more of our technical services work and assigning the copy cataloging and shelf prep staff to work on other areas of the collection.

Summary

Selected

An initial round of selection is done by the bibliographers and collection managers. They decide which broad categories of ONIX records from publishers/distributors will be represented as “items-to-be-acquired” in the local catalog. Patrons, then vote with their fingers and mice clicks as to which items meet their needs.

Purchased

An entirely automated business-to-business transaction. Once the user decides the item is what they need, our library computer talks directly to the publisher/distributor computer and buys the item.

Received

The publisher/distributor doesn’t dally — they ship the item to us for next-day or second day delivery. When it arrives in our mail room, we need to act fast.

Cataloged

Copy cataloging could be done by us, we could receive copy catalog records from the publisher/distributor, or we could decide that — at least for now — that the ONIX data is good enough and that like an “on-the-fly” transaction the formal copy cataloging will happen after the item is returned.

Processed

Choices here, too. Will our staff do the shelf-prep work or is that something we contract with the publisher/distributor? In any case quick processing here, too because…

Shelved

…we want to get the item in the hands of the user who requested it. “Shelved” in this case could be the hold-pickup shelf, or it could be a local physical delivery service that sends the item to the patron.

Can we do this as fast as it would take the patron to get the item directly from the online bookseller? Maybe not — we do have some necessary processing steps that a direct patron purchase process does not have. Can we make that delay short enough so that the patron considers it acceptable as compared to the direct price premium of ordering it themselves?

Do we want to?

The text was modified to update a link from http://www.bisg.org/onix/index.html to http://www.bisg.org/activities-programs/activity.php?n=d&id=15&cid=2.

The text was modified to update a link from http://lisnews.org/article.pl?sid=06/08/01/023238 to http://lisnews.org/node/19233 on January 13th, 2011.

Tom Wilson, LITA past president and all-around insightful LITA Top Technology Trendster, posted a commentary to the “Where have all the programmers gone?” post that deserves top billing ¹. Please read and digest it before coming back here. And it’s not late to the party at all, Tom — I believe it is only now just getting interesting.

As you pointed out, my observations were not intended to denigrate the choices made by our elders. They stepped up and built complex systems that needed to be specialized to library data management tasks because there was nothing else around for them to use. And the complexity of those systems meant countless hours of library staff engaged with patrons to mediate their search requests (at first) and then to teach them how to use the interfaces (as the users became more sophisticated and the interfaces simpler). Now I would argue that at this point, roughly the late 80s and early 90s, the profession as a whole got locked into this “mediated-or-instruction-needed” mindset from which we have yet to recover. I am also drawing the correlation, perhaps incorrectly so, of the technology transfer that occurred during this time of talent from large research libraries to corporate entities that supply us with the automation systems we now rely on. Or, as you put it more eloquently at the start of this quotation:

It is not productive to fault our fore fathers/mothers in libraryland for what happened in this regard. BUT it laid a foundation of thinking that remains a huge burden today. That is: that all library applications are specialized.

It is in the second and third sentence that I think we agree: the mindset at the core of the profession right now that library applications are specialized applications is a huge burden impeding our progress. And while I’ll agree that there is a mindset in the profession that needs to change, I am not convinced that a) libraries (big and small) as organizations have the creative programming talent capacity now as individual entities to capitalize on any sea-change of foundational thinking of the profession; and b) libraries (big and small) as organizations cannot look to the existing “library automation” vendors as the primary providers of solutions in a newly reconstituted vision of “what is the library.”

Addressing the first point, with rare exceptions I don’t see institutions as organizations scaling up their technical staff to handle the raw building tasks of the kinds of services we’d like to see in a reconstituted vision of the library. I do see some evidence that progress is being made here and there, but there are no large programming shops being built to create the next ILS-equivalent. (Side note: I hereby apologize for the connotations created by the phrase “next ILS-equivalent” — that phrase makes sense to me on the surface but it causes deep shudderings in my bones.) I have come to believe, though, that tools and techniques from the open source world can be used to aggregate the capabilities resident in the distributed “libraryland” to share the risk and reward of the next ILS-equivalent (damn — I used it again). I wrote about that earlier in an open letter to adherents to Christensen’s philosophies called Aggregation of Risk in Pursuit of Disruptive Technologies (comments on that post are still welcome to as we move forward in Ohio with the concepts outline there). Also, a colleague from Ohio State and I co-authored an article for the SmartClassroom newsletter of Campus Technology about “betting your career” on open source that I think will have relevance here. The article is to be published on the 19th and I’ll post a copy on DLTJ after a seven-day embargo.

Addressing the second point, it is my assertion, under Christensen’s theories, that “library automation vendors” are not going to be the source of the disruptive innovation that we need. Like the library organizations themselves, they are caught up in the sustaining technology cycle that has lead us to the position we are in today. I would also assert that it is the libraries, not the vendors, that are on firmer footing to break the cycle for the the exact “ideal driven” reasons you cite. Or, to apply Christensen’s model, the vendor’s values/resources/processes triad will drive them harder to continue the sustaining technology cycle — more so than a library organization’s values/resources/processes triad that includes “ideal driven” components.

All-in-all, though, this is going to be an exciting thing to watch over the next couple of years. Hope to continue the conversation and the mutual education….

The text was modified to remove a link to http://www.lita.org/ala/lita/litaresources/toptechtrends/toptechnology.htm on June 9th, 2011.

Footnotes

1. For those that have not yet encountered this idiom, “top billing” is a motion picture industry term that refers to actors whose names appear first in credits. They are usually in the principle performers and typically have the most screen time in the film. Read more in Wikipedia

Walt Crawford chided me — rightly so — for yesterday’s Is the Writing on the Wall for the Integrated Library System? post. My choice of language was, admittedly, sloppy. I was fired up last night…distracted, if you will, by what was happening at a really good conference. Please allow me the chance to redeem my argument.

In academic libraries, in my experience, there has been a decline in the use of library catalogs. This experience could be verified in the ARL supplementary statistics for at least that population of libraries (I think those numbers are password-protected, so it might be a challenge to try to use them). When I get back on the ground and have some time, I will either offer confirmation of that supposition or retract it.

I will dismiss the notion of asking reference librarians how they see users using the catalog because they are too close to the issue. I believe, to use a phrase from Clayton Christensen’s Innovator’s Dilemma, that the library community is facing disruptive innovation — the first it has had to deal with in quite some time. And, according to Christensen’s model, one of the traps encountered when faced with disruptive innovation is listening too closely to customers (implying here that reference librarians are one of the key customers of the catalog). Listening to customers tends to drive a product into “performance oversupply” with lots of features and tweaks that most customers don’t really want yet still pay for because there is not yet an alternative. The suppliers are acting rationally, too — after all, what supplier wouldn’t want to meet all of the desires of its squeakiest wheels?

I ask you — does that sound at all like your most favorite (or least favorite, as the case might be) OPAC supplier? Therein lies the trap — not only for libraries but also for the library software vendors.

Listening to ourselves (librarians in general) is what is getting us into this situation in the first place. We keep focusing on increasingly small improvements with a relatively low return on investment while users of a whole new modality of communication (call it “Web 2.0″, if you will) look over their shoulder (if we’re lucky) and wonder why we’re not keeping up.

You stated: “I frankly find it unbelievable that OPACs aren’t being used. Of course they are.” Agreed. They are being used. OPACs are very effective at figuring out whether your library has a known item as well as what titles your library holds by a known author.

You asked: “Are they being used as often or in the ways librarians might like them to be?” Well, I don’t know about the profession in general but I’m guessing not because instructional sessions surrounding the use of the catalog continue (again, primarily in the academic library space) and even librarians will admit to using Amazon or BN.com or the like to find an item they are looking for.

[Since you are not running an OPAC, Walt, I suspect these three questions will have to be in the abstract for you (should you choose to comment).] I’ll ask: “Are they being used as effectively as they could be?” And related: “Does their current use justify their on-going expense?” And lastly: “When is the last time someone thanked you for a new feature you worked hard to get into your system?” If it has been too long, what should we do about it?