On the second day of the OCLC Global Council meeting [agenda PDF] there was a presentation by Robin Murray (VP, OCLC Global Product Management) and Jim Michalko (VP, OCLC Research Library Partnership) called “Linked Open Data”. The title of the presentation was an understatement because the real heart of the matter was WorldCat data as linked open data. The presentation was about an hour long, and despite the technical difficulties was fascinating to listen to through the streaming video feed. OCLC says the archive of the meeting will be available at some point, and I urge you to check it out when it becomes available.

Robin Murray’s first half of the presentation walked through OCLC’s thoughts about how their implementation of linked data is evolving along side their programatic API offerings. (Almost as an aside, Robin said that use of the WorldCat API has been growing steadily and is now up to 15 million hits per month — or 347 hits/minute. Wow!) He had a really slick diagram that showed the exposure of data through linked data services and API services across a half-dozen OCLC products. He called library linked data the “Data Exposure Service” of the WorldShare platform. There was one slide that I did copy down: with a title of “Opportunity or Threat”, Robin proposed:

IF, collectively we can re-envision cataloging as ‘registering nodes in a global web of data’;

AND we can position WorldCat as the trusted, global web of library data;

THEN this dramatically increases the global value and utility of metadata management with WorldCat.

Robin went on to say that he sees publishing the OCLC cooperative’s data assets as linked data is more opportunity than threat. That doing so grows the possible base of funding support for WorldCat; or, as he put it a “significant opportunity to ‘Grow the Denominator’” (where the fixed costs of adding value to member contributed records is the numerator).

Jim Michalko’s second half of the presentation walked through the history of the cooperative’s Rights and Responsibilities document, past discussions about publishing WorldCat data, and lead up to a recommendation that OCLC leadership had for the Global Council and the Board of Trustees: publish WorldCat data as linked data with an Open Data Commons “BY” attribution license (a.k.a. ODC-BY). In fact, Jim explicitly said “no restrictions on commercial reuse” of individual records.

After the presentation and a short question/answer period, there was to be discussions at the tables where one of the questions was whether the Global Council would recommend to the Board of Trustees the adoption of ODC-BY to WorldCat data. Each table was to report the summary of their deliberations, but unfortunately that part of the meeting wasn’t webcast. We may have to wait for formal minutes of the meeting to be published to see what the conclusion of the discussion was.

This discussion at Global Council about WorldCat data follows a similar announcement from Thom Hickey about the Virtual International Authority File being published with an ODC-BY license with attribution to VIAF. Jonathan Rochkind posted his appreciation and approval of the VIAF move but predicted a couple of pain points with the ODC-BY licensing. What he says there is also true of an ODC-BY licensed WorldCat.

Both Robin and Jim made a point of modeling their practices on what is happening in the general linked data community, but I’m concerned that a Google search for “ODC-BY linked data” doesn’t show many examples beyond OCLC’s efforts. It is quite possible that my Google results are artificially skewed towards library results. On the Code4Lib IRC channel it was noted, for instance, that the Linked Geo Data (derived from Open Street Map project) is or soon will be ODC-BY. [See clarification in the comments.] If there are other examples, I’d appreciate hearing about them in the comments.

There was also a discussion about whether WorldCat data could be given by members to Europeana, which announced it is using the CC0 public domain dedication. There is an important mismatch between CC0 and ODC-BY — notably the attribution requirement in the latter that isn’t in the former. I can’t faithfully recall direct quotes from the discussion — you’ll have to watch the video archive — but the summary I remember was that doing so would be okay as long as the member was being reasonable about what was being shared with Europeana. And that the same would hold true for the Digital Public Library of America.

On the whole, the presentation and discussion were fascinating to follow, and from my point of view represents a welcome and appropriate liberalization of how WorldCat data can be reused and the intention of OCLC to public bibliographic (and other data) to become a stable, generally usable anchor of library linked data. This is an important milestone to putting libraries and library data at the core of the linked open data movement, and it will spur innovation and uses that we can only dream about now.

Update — 18-Apr-2012

Global Council passes resolution advising move to Open Data Commons Attribution License ODC-BY #oclcgc

— OCLCEMEARC (@OCLCEMEARC) April 18, 2012

As I recall the process, it now moves to approval by the OCLC Board of Trustees.

Eric Morgan posted a message to the Next Generation Catalog for Libraries mailing list this morning that points to a announcement by the University of Florida library that they are now applying a Creative Commons Public Domain Dedication statement to MARC records they create. Their announcement says:

Beginning March 2011, the University of Florida Smathers Libraries implemented a policy to include a Creative Commons license in all of its original cataloging records. The records are considered public domain with unrestricted downstream use for any purpose.

The following MARC 588 field (Source of Description Note) is added to new records contributed to WorldCat. It has not been added retrospectively to University of Florida original records in WorldCat.

588::|a This bibliographic record is available under a Creative Commons CC0 license. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.

Their announcement page also provides links to some examples from the OPAC. Scroll down to the bottom of the page to see the CC0 graphic and declaration.

• Latinoamericanismo : historia intelectual de una geografía inestable
• Mapa Everest de carreteras, España y Portugal
• The Song of Ceylon

The University of Florida joins University of Michigan in making original cataloging records available under CC0. To refresh your memory CC0 “Public Domain Dedication” statement (it isn’t a license!) says:

The person who associated a work with this deed has dedicated the work to the public domain by waiving all of his or her rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.

You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.

Now this is an interesting development because it is a kind of viral declaration of the same sort that OCLC proposed with the withdrawn draft of the record use policy. There OCLC was going to add a 996 field to all records exported from WorldCat that would say:

This would do something similar except that it would make viral the public domain declaration on records added to OCLC WorldCat. (It is probably also an oversight that the 996 field documentation is still on OCLC’s site.) Does this begin to segment WorldCat into records that can and cannot be used? Or is it redundant since some think that MARC records, as a recitation of facts, cannot be copyrighted anyway?

This week is a mostly Google edition of DLTJ Thursday Threads. Below is a high-level overview of Google’s Book Search algorithm, how Google is helping web servers improve the speed at which content loads, and how Google’s internet traffic is growing as a percentage of all internet traffic. But first, there is an uprising on the RDA test records in the WorldCat database.

If you find these interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the right. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments and tips, as always, are welcome.

Memorandum Against RDA Test

We have found ourselves in an unenviable position of opposing the work that supposedly has been authorized by agencies representing our interests. I might compare it to a military coup d’état. I mean here the RDA “test” and its implications on the cataloging world at large. After extensive discussions on the PCC, OCLC cataloging e-mail lists with opinions from the British Library, Australia and North America, we can safely conclude that there is a broad consensus against principles of RDA and the way RDA “test” has been imposed on the cataloging world.

The original post on the OCLC-CAT list by Wojciech Siemaszkiewicz of the New York Public Library is behind a must-subscribe-and-authenticate form, but it has been copied out copied to an open website by Becky Yoose (thanks, Becky!). The subsequent discussion resulted in a Petition against the RDA Test by Jacqueline Byrd at Indiana University. The link to the position has been posted to the open AUTOCAT list, and there has been subsequent discussion there. (Hat tip to Kirsten Davis.)

Inside the Google Books Algorithm

Rich Results is the latest in a series of smaller front-end tweaks that have been matched by backend improvements. Now, the book search algorithm takes into account more than 100 “signals,” individual data categories that Google statistically integrates to rank your results. When you search for a book, Google Books doesn’t just look at word frequency or how closely your query matches the title of a book. They now take into account web search frequency, recent book sales, the number of libraries that hold the title, and how often an older book has been reprinted.

Alexis Madrigal article in TheAtlantic.com draws a comparison between the techniques and algorithms used for web search with those used for book materials. The need for relevant search results is the same, but books don’t have the same inter-page linking hints that drive the PageRank algorithm for web search. The use of anonymized circulation data in creating clustered bibliographic descriptions was mentioned at the ALA Midwinter ALCTS Forum on Mashups of Bibliographic Data, and apparently it is also used in the relevance ranking of Google Books search results. (Hat tip to Ron Murray.)

Google Releases mod_pagespeed

mod_pagespeed is an open-source Apache module that automatically optimizes web pages and resources on them. It does this by rewriting the resources using filters that implement web performance best practices. Webmasters and web developers can use mod_pagespeed to improve the performance of their web pages when serving content with the Apache HTTP Server. mod_pagespeed includes several filter that optimize JavaScript, HTML and CSS stylesheets. It also includes filters for optimizing JPEG and PNG images. The filters are based on a set of best practices known to enhance web page performance. Webmasters who set up mod_pagespeed in addition to configuring proper caching and compression on their Apache distribution should expect to see an improvement in the loading time of the pages on their websites.

Google has promoted best practices for improving the rate at which web pages load for a number of years. This week they introduced mod_pagespeed: an Apache web server module that brings these practices to bear by rewriting HTML, JavaScript, and Cascading Style Sheets on-the-fly. Since Google now includes the speed at which pages are rendered in a browser as a factor in ranking search results, this would seem to be a good module to explore for anyone running an Apache web server with public content. (Hat tip to Ed Summers.)

Google as a percentage of all internet traffic

Google Sets Internet Traffic Record

Google now represents an average 6.4% of all Internet traffic around the world. This number grows even larger (to as much as 8-12%) if I include estimates of traffic offloaded by the increasingly common Google Global Cache (GGC) deployments and error in our data due to the extremely high degree of Google edge peering with consumer networks. Keep in mind that these numbers represent increased market share — Google is growing considerably faster than overall Internet volumes which are already increasing 40-45% each year.

Craig Labovitz of Arbor Networks notes that if Google were an internet service provider, it would now be “the second largest carrier on the planet.” Wow! That is a lot of data sloshing around on its own internal network!

Within the span of a recent week we’ve had two views of the OCLC cooperative. In one we have a proposition that OCLC has gone astray from its core roots and in the other a celebration of what OCLC can do. One proposes a new mode of cooperation while the other extols the virtues of the existing cooperative. Both writers claim — independently — to “talk to librarians” and represent the prevailing mood of the profession. Can these two viewpoints be reconciled?

“Too Many Cooks?”

I spend quite a bit of time talking both to librarians and industry partners–publishers, booksellers, Web-technology providers, search engine companies–all kinds of people doing interesting things in our space. And in those talks, there is often a discussion of one of the following: content, technology or community. What I’ve come to realize, though, is that the best results come from places where all three come together.

Chip’s post is short but clear in its view that the community of OCLC members is something special and that it adds value to member libraries.

“The Cooperative We Need”

It appears to me that the interests of the OCLC we know today do not appear to be in total alignment with the needs and interests of its overall actual membership. Perhaps they are in alignment with the interests of the Board, Council, and other governing and administrative arms, but the feeling I get in talks with librarians is that it is not in alignment with what they want. As I talk to librarians, across the country today, I hear that what they want is an organization, a cooperative that is focused on developing and providing open and collaborative library content and services that are widely accessible by all in order that they (the librarians) can focus on re-establishing and/or maintaining the value of libraries in our society.

Carl goes on to propose the creation of a utility that aggregates the ratings and rankings of individual users into a database that can enhance the relevance ranking of the emerging generation of discovery layer products.

My Thoughts

Carl describes the problem in his post. When a not-for-profit vendor acquires a significant number of for-profit companies (and spins them back out again), how can we (members, vendors, and the library community in general) understand how the mix of commercial and non-commercial interests are playing out at the management level? Can the OCLC that is the bibliographic utility, the metadata switch between bibliographic-based services, and the R&D braintrust co-exist with the for-profit businesses, motivations, and operations? Or, to put it more sharply, does the negotiation of H.W. Wilson content for use on the subscription-based WorldCat database hinder the evolution of discovery layers that being developed by companies that don’t have the tax-advantaged not-for-profit status? (And don’t forget about the allegations of anti-competitive behavior in the SkyRiver/Innovative-versus-OCLC lawsuit.)

In closing this section, I want to pull out and emphasize another quotation from Carl’s post:

In the end, all of these business initiatives, and now resulting lawsuit, strongly work against OCLC being able to do what it does best—building collaboration, content, and related services as a non-profit entity to serve the larger profession.

Agreed.

Carl’s Grand Idea

I highly recommend reading Carl’s post and thinking about ways of answering the question “What Do We Want From OCLC?” I commend Carl for his courage and vision in articulating his points and proposing something new for the profession to drive towards.

On September 9th, OCLC filed its first substantial response with the court to the antitrust lawsuit file by SkyRiver and Innovative Interfaces. And in a motion where OCLC requests a change of venue from the Northern District of California to the Southern District of Ohio — something seemingly mundane — they certainly pulled no punches:

Through a lengthy recitation of inaccurate facts, Plaintiffs allege six claims against OCLC. In short, Plaintiffs allege that OCLC, a forty-year old non-profit entity, is making it difficult for Innovative and its one-year old sister-company, SkyRiver, to compete and gain market share in the ILL, ILS, and the online cataloging library world. Through a variety of uncited references in their Complaint to “prominent library-related internet blogs,” unnamed commentators, and unattributed articles and reports, as well as through creating an anti-OCLC website, Plaintiffs have levied a propaganda war on OCLC simply because Plaintiffs have been unable to compete successfully with OCLC’s membership base and bibliographic data which OCLC earned through forty years of dedicated service to its member libraries.SkyRiver Technology Solutions, LLC et al v. OCLC Online Computer Library Center, Inc. Filing: 16. Page 4. Retrieved from Justia Docs on 18-Sep-2010. (link added)

The question at hand seems to be a bit more than a propogada war between SkyRiver and OCLC. But the court is not yet at the meat of the matter.

The text accompanying the motion for change of venue, though, does not deal with the issues raised in the lawsuit. Instead, it requests the California court “transfer this action from this District to the Southern District of Ohio, Eastern Division, located in Columbus, Ohio.”

“Litigation should proceed where the case finds its center of gravity.” The “center of gravity” is determined by the location of key witnesses and documents. Here, the “center of gravity” is plainly the Southern District of Ohio, for these reasons:

• OCLC’s headquarters and virtually all of the key witnesses and documentary evidence are located in or near Central Ohio.
• OCLC has a relatively small presence in California, as compared to its much larger and longer-established presence in Ohio.
• OCLC made all decisions and actions operative to the allegations of Plaintiffs SkyRiver Technology Solutions, LLC (“SkyRiver”) and Innovative Interfaces, Inc. (“Innovative”) (collectively “Plaintiffs”) in Ohio.
• The State of California does not have an interest in this lawsuit beyond the fact that Plaintiffs are residents of California, whereas the State of Ohio has a great interest in this lawsuit because Plaintiffs have alleged that one of Ohio’s non-profit entities is abusing its non-profit status, an allegation that can impact other Ohio non-profit entities.

For these and other reasons discussed in more detail below, all parties and the Court will be better served by transferring this case to the Southern District of Ohio, Eastern Division. In making this Motion, OCLC reserves any defenses that it may have against Plaintiffs’ claims.SkyRiver Technology Solutions, LLC et al v. OCLC Online Computer Library Center, Inc. Filing: 16. Pages 1-2. Retrieved from Justia Docs on 18-Sep-2010. (Legal citations removed from text.)

The last point is probably the most interesting to the layperson watching this epic battle unfold. Pages 12 and 13 contain these statements:

The state of Ohio’s interest in adjudicating this matter within its borders also militates towards transferring this case. OCLC’s relevant policies and practices were developed and implemented in Ohio and the most important witnesses and evidence are located there.

Further, Plaintiffs have stated serious, albeit unfounded, allegations regarding OCLC’s non-profit status. Plaintiffs have stated that OCLC is “abusing its status as a tax exempt, non-profit entity and unfairly competes with for-profit companies, such as Innovative and SkyRiver, by using its non-profit status as leverage to monopolize the library services industry….” These baseless allegations could create serious implications for other Ohio non-profit entities, and Ohio courts have a greater interest in litigating these issues. While Ohio courts also have more experience applying Ohio’s laws, more importantly, they also have more experience with the routine customs and practices of non-profit entities in Ohio. In addition, the Ohio Attorney General has oversight over Ohio non-profit entities and would likewise have an interest in this lawsuit.

In contrast, California does not have an interest in litigating this action because none of the operative actions occurred in California. Though Plaintiffs asserted California state law claims, as explained below, Ohio courts will be equally skilled at applying and interpreting those laws.SkyRiver Technology Solutions, LLC et al v. OCLC Online Computer Library Center, Inc. Filing: 16. Pages 12-13. Retrieved from Justia Docs on 18-Sep-2010. (Legal citations removed from text.)

The question of OCLC’s tax exempt status is one that bubbles up on occasion. It would seem like OCLC’s legal team is willing to take this head on.

Accompaying the motion to transfer is a declaration by Bruce Crocco, Vice President, Library Services for the Americas for OCLC, and this is a more interesting document. It goes into the history of OCLC — its founding in 1967 on the Ohio State University campus, how OCLC revolutionized the production of paper cards for card catalogs and the movement into online catalogs, and the evolution of the WorldCat brand name.

Earlier this week, the judge in the case set this schedule for hearing from the parties on the motion:

This matter is set for a hearing on October 29, 2010 on Defendant OCLC Online Computer Library Center’s motion to transfer venue. The Court HEREBY ORDERS that an opposition to the motion shall be filed by no later than September 27, 2010 and a reply brief shall be filed by no later than October 4, 2010.

If the Court determines that the matter is suitable for resolution without oral argument, it will so advise the parties in advance of the hearing date. If the parties wish to modify this schedule, they may submit for the Court’s consideration a stipulation and proposed order demonstrating good cause for any modification requested.

To refresh your memory, the lawsuit was filed on July 28, 2010, and was assigned to Judge Jeffrey S. White on August 6, 2010. On August 12th, Judge White set a Case Management Conference for January 14, 2011, and on August 13th lawyers for OCLC filed a notice with their intent to request a change of venue. Stay tuned for an update as the case moves on…

Thursday will be a big day in the Google Book Search lawsuit settlement: the parties to the lawsuit, along with the objectors, supporters, and friends-of-the-court, will be in the courtroom of United States District Judge Denny Chin offering oral arguments in the final settlement/fairness hearing. In his order, Judge Chin recognized 26 parties that will speak for up to five minutes each on their positions in the settlement (21 in opposition, 5 in favor). The U.S. Department of Justice will also speak at the hearing. But I think we’re all eagerly awaiting to hear what the judge himself will say about the settlement agreement.

In the lead-up to the hearing, Associate Professor James Grimmelmann at the New York Law School has continued his efforts, along with the students from the Institute for Information Law and Policy at New York Law School, to make the documents and proceedings of the lawsuit accessible and understandable to non-lawyers. In the most recent court filings leading up to Thursday’s hearing are some interesting nuggets.

In his posting on the motion for attorneys fees, he notes that “counsel for the author sub-class are asking for the full $30 million in fees and reimbursement of their out-of-pocket costs.” The filing contains information about the number of hours and the billing rate for some of the lawyers working on the case. Some of the stuff is just really interesting, like one filing that included everything from 18 hours by a partner of a firm (who is also a law professor at NYU) at rate of $995/hour to an itemization of 51¢ for long distance calls by the firm related to the case. Whew!

More interesting to DLTJ readers would be Grimmelmann’s highlights of Dan Clancy’s declaration in support of the agreement. Dan Clancy is engineering director of the Google Book Search project, so he has a unique insight into the inner workings. Grimmlemann notes that Clancy states:

• To date, Google has Digitized over twelve million books, and intends to continue Digitizing books in the future.
• Google has received metadata from 48 libraries.
• Google pays approximately $2.5 million per year to license metadata from 21 commercial databases of information about books.
• Google has gathered 3.27 billion records about Books, and analyzed them to identify more than 174 million unique works.

The third bullet is interesting in that I think we can eliminate one of the “commercial databases” from the list. I can’t find it in my notes from ALA Midwinter, but I seem to recall hearing Jay Jordan (OCLC President) say something along the lines that OCLC was not receiving a monetary return from the sharing of bibliographic data with Google; the value OCLC gets for its membership comes from the links back to WorldCat from Google services. If I got this wrong, I hope someone from OCLC will call me out on it.

The last bullet is interesting, too: Google has identifying 174 million works in analyzing all of the sources of data coming into it. I tried to find some numbers in the descriptions of WorldCat to compare that to, but didn’t have any luck this evening. (There isn’t anything about statistics available on http://worldcat.org/?)

To Grimmelmann’s highlights I would add this statement that seems strangely out-of-place.

• Google has no interest in censorship. Indeed, Google’s mission is to organize the world’s information and make it universally accessible and useful.

Has anyone brought censorship into the discussion yet? Privacy for sure, but censorship?

Also:

• Google has developed algorithms to compare these numerous sources of metadata and identify the most accurate data about each book.

They certainly seem to have invested a lot of effort in this area. More info can be found in my summary of Kurt Groetsch’s presentation at ALA Midwinter 2010.

This year the ALCTS Forum at ALA Midwinter brought together three perspectives on massaging bibliographic data of various sorts in ways that use MARC, but where MARC is not the end goal. What do you get when you swirl MARC, ONIX, and various other formats of metadata in a big pot? Three projects: ONIX Enrichment at OCLC, the Open Library Project, and Google Book Search metadata.

Below is a summary of how these three projects are messin’ with metadata, as told by the Forum panelists. I also recommend reading Eric Hellman’s Google Exposes Book Metadata Privates at ALA Forum for his recollection and views of the same meeting.

ONIX Enrichment at OCLC

Renee Register, Global Product Manager for OCLC Cataloging and Metadata Services, was the first to present on the panel. Her talk looked at a new and evolving product at OCLC on the enhancement of ONIX records with WorldCat records, and vice versa. ¹

As libraries, Renee said “our instincts are collaborative” but “our data and workflow silos encourage redundancy and inhibit interoperability.” Beyond the obvious differences in metadata formats, the workflows of libraries differ dramatically from other metadata providers and consumers. In libraries (with the exception of CIP and brief on-order records) the major work of bibliographic production is performed at the end of the publication cycle and ends with the receipt of the published item. In the publisher supply chain, bibliographic data evolves over time, usually beginning months before publication and continuing to grow for months and years (sales information, etc.) after publication. Renee had a graphic showing the current flow of metadata around the broader bibliographic universe that highlighted the isolation of library activity relative to publisher, wholesaler, and retailer activity.

Diagram of the Process of Enhancing ONIX Records, from OCLC Services for the Publisher Supply Chain Webinar, August 2009

In her experience, Renee said that libraries need ways to enable our metadata to evolve over time and allow for publisher-created metadata to merge effectively with library-created metadata. The bibliographic record needs to be a “living, growing” thing throughout the lifecycle of a title and beyond. In concluding her remarks, she offered several resources to explore for further information: the OCLC/NISO study on Streamlining Book Metadata Workflow, the U.K. Research Information Network report on Creating Catalogues: Bibliographic Records in a Networked World, the Library of Congress Study of the North American MARC Records Marketplace, the Library of Congress CIP/ONIX Pilot Project, and the OCLC Publisher Supply Chain Website.

From MARC to Wiki with Open Library

Karen said right at the start that the Open Library project is different from most of what happens in libraries — it is “someone outside the library world making use of library data” — although the goal is arguably the same as others — “One web page for every book ever published.” As such, the Open Library isn’t a library catalog as librarians think of it in that it is not a representation of a libraries inventory. It has metadata for every book it can know about and a pointer to places where the book can be found, including all of the electronic books in Internet Archive (Open Content Alliance, Google Public Domain, etc.) as well as pointers back to OCLC WorldCat. Karen’s role for the project is that of “Library Data Informant.” The Internet Archive decided that they needed someone who understood library data in order to try to use it. From Karen’s perspective, she is trying to be a resource for project but not give them any guidance on how to implement the service. She is curious to see what the project would do when bibliographic data is viewed from a non-librarian perspective. If they have questions, or if they have assumptions about data that are wrong, then she intervenes.

Karen went on to briefly describe the Open Library system. Open Library doesn’t have records; rather, it has field types and data properties. In this way, it uses semantic web concepts. “Author” is a type, “Author birthdate” is another type, and so forth. There are no set field types, so if the project gets data from source for which a type doesn’t yet exist, it can create a new one. Each type can have data properties such as string, boolean, text, link, etc. Nothing is required and everything is repeatable. Everything — types, properties, and values — gets a URI (a URI is an identifier like a URL, but conceptually a superset of the universe of URLs). Titles, authors, subjects, author birthdates, and so on have URIs. Lastly, the underlying data structures are based on wiki principles: all edits are saved and viewable, anyone can edit any value, anyone can add new types or properties, anyone can develop their own displays, etc.

The data that is now in Open Library came from a variety of sources. They started with a copy of books from the Library of Congress, and continue to receive the weekly updates. They performed a crawl of Amazon’s book data. They have gotten some from publishers, libraries, and individual users. The last is perhaps the most interesting because it is mainly people outside the western world who are otherwise having trouble getting their works recognized.

Problems, Issues, Challenges, and Opportunities with the Data

Names -

“These library forms of names? Honestly no one but us can stand them.” Even something as simple as the form of last-name-comma-first-name is troublesome. No one else uses this form of the name: Amazon, Wikipedia, etc. In processing these, any information between parenthesis has been deleted, birth and death dates move into separate field types.

Titles -

In working with the Open Library developers, this is one place that Karen tried insisting on applying a library practice: knowing the initial article. For us, this is important for sorting books in alphabetical order. The developer response — why do we have to sort in alphabetical order? “Where else but library catalogs to we see things sorted in alphabetical order? Not in Google, not in Amazon, not anywhere. Alphabetical order is not in the mindset anymore.” They also found that the title might include extraneous data. Amazon, for instance, appends the series title in parenthesis to the main title. This is a demonstration of how other communities are not as concerned about strongly typing and separating information into fields. Amazon, of course, has reasons for series information into the main title: it helps sell books.

Product dimensions -

Publishers and distributors need to know characteristics of an item such as height, width, depth, and weight; they, of course, need to put it in a box and ship it. Libraries, concerned about placing the item on the shelf, record just height. Recording pagination is different, too: libraries use odd notations “ill. (some col)” and “xv, 200p.” versus simply “200 pages.”

Birthdates -

Librarians use birthdates to distinguish names; if there is no need to distinguish a name, birth and death dates are not added. Someone looking at this from the outside would ask ‘Why don’t all authors have birth and death dates?’ This can be useful information for viewing the context of an item, not just to distinguish author names. Open Library ran author names against Wikipedia to pick up not only birth and death years, but also the actual dates.

Subject headings -

Open Library using Library of Congress Subject Headings was out of the question. In processing the data, the Open Library developers just broke them apart into segments and used them. But because they were able to do data mining on the subject field types, they did find statistical relationships between the disassembled precoordinated headings and were able to present those to the user.

The View of the Data -

Rather than a traditional library view of long lists of author-title, the Open Library (in its next version coming in February) will have several different views into the mass of data: Authors; Books (what we would call FRBR ‘manifestations’); Works; Subjects; and eventually places, publishers, etc. For example, when searching for an author one would get the author page. On it would be all of the works from the author as well as other biographical information. It looks similar to a WorldCat identities page, except it is the actual user interface built into the system. Similarly, every work will have a page, and at the bottom of it one will see all of the editions of the work. Also, each subject will have a page, and one will see a list of works with that subject as well as authors who write on that subject. As Karen said, “The subject itself becomes an object of interest in the database, not just something that is just tacked on to the bottom of the library record.”

Data mining -

With the data in this format, it is possible to perform data mining actions against it. For instance, simple data mining such as country of publication, popular places that appear, etc. When they had the problem of author names — knowing when to reverse surname and forname — they ran the names against Amazon and Wikipedia and retained the ones where they found the order of the entry was the same. The Open Library developers are also experimenting with data mining to find publisher names. Publisher names, of course, vary dramatically, but by using ISBN prefixes they can pull together related items into a “publisher” view.

Karen suggested watching the Edward Betts’s site, one of the developers of the Open Library project with an eye on the data mining aspects. She said it is fun to look at our data when it can be viewed from this different point-of-view. She also said to watch out for a new version of the Open Library website coming in February.

Google Book Search Metadata

On the input side, Google is getting bibliographic metadata from over 100 sources in a variety of formats. MARC records are coming from libraries, union catalogs, commercial providers (OCLC), publishers/retails (one publisher supplies records in MARC format). Google also gets ONIX records from commercial providers (such as Ingram and Bowker), publishers, and retailers. Google is especially interested in data from non-U.S. retailers because it is a source of information about books published outside the United States; it helps facilitate discovery of items that they may not otherwise encounter in the publisher and library programs. Google also receives records in a variety of “idiosyncratic formats” — for example, publisher-contributed metadata (via the Publisher Partner Program); information associating books with jacket images; name authority records (from LC); reviews; popularity signals (sales data as well as anonymized circulation data from some library partners, useful for feeding into the relevancy ranking algorithm); and internally-generated metadata (for instance, whether a book is commercially available or not). Google processes all of this information to come up with a single record that describes a book. At this point they have over 800 million bibliographic records and one trillion bits of information in those records.

All of these records from all of these sources are processed and remixed with Google’s parsing algorithms about twice a week. The first step is to transform the incoming records into a “less verbose format” for storage and processing. It is a SQL-like structure that allows elements of the metadata to be queried. Records are then parsed to extract specific bits of information, transform the bits as necessary, and write the information to an internal “resolved records” data structure (a subset of the data coming from the input formats). In the presentation, Kurt had examples of how making inferences from data coming from both MARC and ONIX can be troublesome. Parsing also involves extracting “bibkeys” from the records to aid in matching across sources of data. Four types of identifiers are extracted from bibliographic records: OCLC numbers, LCCNs, ISBNs, and ISSNs. They provide usually useful signals when matching bibliographic and help with assertions that two records describe the same manifestation. Google also tries to parse item data when present in records representing multi-volume works, enumeration and chronology. They will also treat barcode as a form of a “bibkey” if they get it from a library. The parsing algorithm will also split records containing multiple ISBNs representing different product forms (e.g. hardback, paperback, etc.).

With all of this data parsed into records, Google starts its clustering process where records are examined and attached to each other. Bibkeys provide significant evidence for relating records to each other, but bibkeys are not always present in a record (non-U.S. records and older records frequently contain no bibkeys). The algorithms then fall back on text similarity matching using title, subtitle, contributor and other fields such as publisher and publication year. The results are clusters of records representing the same manifestation. An algorithm then attempts to derive the “best-of” record for a single cluster from all of the parsed input records. This is done in a field-by-field voting process based on the trustworthiness of individual fields from record sources.

Kurt went into some of the challenges facing the team building the clustering and best-of record creation algorithms. For instance, in dealing with multivolume works they know of 5 numbering schemas with 3 number types in 15 different languages. Enumeration is now showing in the public display, but the development team is still working with unparsable item data due to inconsistent cataloging practices between institutions…and sometimes inconsistencies within an institution. Another problem is non-unique identifiers. In the current data set ISBN 7899964709 is shared by 75 books and ISBN 7533305353 is associated with 1413 books. There are also poor quality or “junk records”. Kurt said his favorite was “The Mosaic Navigator” by Sigmund Freud published in 1939. These are hard to identify with an algorithm, and they rely on reports of problems that enable the developers to go in and “kill” the troublesome record. Another example is a book by Virginia Woolf where the incoming record had conflicting information; it had two 260 fields that contained different dates (1961, correct, and 1900) with fixed field information that strongly suggested that 1900 was the single date of publication. When the data problem is systematic, they can identify it and compensate for it. Kurt’s example for this case was “The United States Since 1945″ published in 1899. This one was highlighted in Geoffrey Nunberg’s criticism of Google Books metadata. In this case, there was a source of metadata from Brazil that when they didn’t know the date of publication would use 1899. When Google went back and looked at the date distribution of books there was a huge spike in 1899. Once Google knew about it they were able to go in and kill that information from that source of records. ³

In closing, Kurt said that Google is committed to engaging with the library community on improving metadata and metadata processing.

The text was modified to update a link from http://www.niso.org/publications/white_papers/Stream lineBookMetadataWorkflowWhitePaper.pdf to http://www.niso.org/publications/white_papers/StreamlineBookMetadataWorkflowWhitePaper.pdf on January 19th, 2011.

The text was modified to remove a link to http://www.oclc.org/speakers/bios/register_renee.htm on February 11th, 2011.

Footnotes

1. For those not familiar with ONIX, it is a suite of standards promulgated by EDItEUR for the interchange of information on books and serial publications. It is primarily used as the communication channel between the publishing industry through distribution chains to retail establishments.
2. By the way, it seems like BISAC is an acronym for “Book Industry Systems Advisory Committee”, the former name of the Book Industry Study Group.
3. A side note: Google isn’t the only one tripped up by this. If one searches for the ISBN of the item, 0195038487, you get to more than one site that has the same incorrect publication date. At least Google is attempting to clean up the data!

Jay Jordan’s remarks during the OCLC Update Breakfast and the discussion at the Developers Network table at that breakfast generated further fuel for my previous philosophical thoughts on “Who is a member of the OCLC Cooperative?” In the context of things like Developer Network API keys¹ this question of who is a member of OCLC the cooperative and who is not meets the on-or-off, ones-and-zeros nature of computers. One can’t “kinda” have an API Key unless that capability is programmed into the software (or a human chooses to override the established rules for who has a key).

The discussion around the table after Jay’s remarks tested some of the “edge cases” to the established hard-and fast rules. OCLC Governance, at least as I understand it, surrounds institutions who are members of the OCLC Cooperative. By implication, the benefits (and also the responsibilities) of membership in the OCLC Cooperative transfer to staff employed at those member institutions. People who work for governing (institutional) members of OCLC can get Developer Network API keys to create services for their library. And except for the fuzzy notion explored previously whether individuals are members of the OCLC cooperative, this all seems pretty clear.

But what about individuals in the employment of a library that is not a member of OCLC that want to create applications that benefit library users. Further, following Jay Jordan’s definition of partners OCLC wants to work with (as expressed during the OCLC Record Use Policy Council Update), that non-institutional-member individual can return value back to the cooperative (by contributing the ideas to the Developer’s Network Showcase, by contributing working code to a software repository, etc.). Isn’t that returning value to the cooperative? Should this person be granted a Developer Network API key even though their institution is not a member? Can we conceive of a process and guidelines by which this could happen? (Has it perhaps already happened in the human-created fuzz of OCLC-the-stewards overriding the established rules?)

I’ll state here that I would support the creating of a process by which anyone — from a member library or a consultant doing work on behalf of a member library or a 12-year-old kid) can request a Developer Network API key. What is needed is a clear understanding that those who get such keys gain some kind of adjunct membership in the OCLC-the-cooperative, and as so are expected to return value to the cooperative through their work with the API key. (There would need to be transparency from OCLC-the-steward on who in this category is getting keys, who is not, and for what reasons — at least in summary.) The area of consultants or other agents doing work on behalf of a library is an area that would need further exploration. I would offer that it is okay for one such agent to use an API key to help a particular client, but I also agree with Jay that it is not okay for that same agent to use that key to derive revenue from helping non-OCLC members with the same key. The latter fails the renumeration-back-to-the-cooperative test (either financial or intangible benefit) that Jay remarked on during Saturday’s Record Use Council session and one that the Record Use Council seems to be struggling with.

These are tough but interesting issues. We’ve left the days where the benefits of membership were tied to the delivery of shelf-ready cards and dedicated terminals for online original and copy cataloging. OCLC seems to be transforming itself from a library-used service to a world-used service. I give OCLC-the-stewards credit for listening to OCLC-the-membership about appropriate ways to make use of the shared resource that is WorldCat, and credit to OCLC-the-membership for pushing OCLC-the-stewards into creating ways to expand access to that shared resource.

The text was modified to update a link from http://dltj.org/article/alamw10-oclc-record-use to http://dltj.org/article/alamw10-record-use-policy/ on December 31st, 2010.

The text was modified to update a link from http://dltj.org/article/alamw10-oclc-record-use to http://dltj.org/article/alamw10-record-use-policy/ on December 31st, 2010.

Footnotes

1. “API” is an acronym for Application Programming Interface. In summary, an API is the set of rules by which one program can task another program for data or to perform a service. An “API key” is the mechanism through which a requesting application establishes the right to be able to use data and services of another application. In this context, the holder of an OCLC Developer Network API key can access the wealth of data and services being offered by OCLC.

I think it is a statistical anomaly that many of the meetings I attended during ALA Midwinter were somehow related to OCLC. That statistical anomaly has certainly played out in postings here on DLTJ of my impressions of Midwinter meetings. Continuing with this thread of OCLC events, I attended the OCLC Update Breakfast Sunday morning for a membership-dues-paid croissant and orange juice, and to listen to Jay Jordon’s biannual update on the past, present and future of OCLC. What follows are highlights that I found interesting in the course of his remarks, but certainly not a comprehensive report of what was said. Video of Jay’s remarks where recorded and are to be posted at some point on the OCLC website (roughly six to eight weeks from now, if my memory of past events can be any guide).

WorldCat Growth since 1998

New “Search Engines”

OCLC Services in the Cloud

OCLC is also looking to help with collection management as a cloud-available tool. Working with the New York University Libraries, OCLC is bringing analytics to bear on collection management and space allocation decisions by helping with data about the location of items in the campus library, in the library’s “ReCAP” remote storage, and what is available digitally in HathiTrust. And speaking of HathiTrust, the public interface to 7.5 million volumes digitized largely through the Google Book Search partnership, OCLC is working with project participants to aid the metadata description of items in HathiTrust, to ensure that items in HathiTrust have records in WorldCat, and to add WorldCat Local as an interface to the HathiTrust collection.

Recent problems for Cataloging Partners

On Saturday morning of ALA Midwinter 2010, Dr. Jennifer Younger moderated a session on the progress of the OCLC Record Use Policy Council. The meeting started with an introduction to the reasons behind the creation of the Record Use Council, the charge of the Council from the board of trustees, and how the framing of the discussion of the policy is guided by the values and history of OCLC the cooperative. There wasn’t much new here for those that have been following the progress of the policy discussion, so I am skipping over it most of it with the exception of a few notable topics. After that, I’m focusing on the lengthy question and answer session that followed Dr. Younger’s background presentation.

Highlights of the Background Presentation

WorldCat itself is now made up of 170 million bibliographic records and 1.5 billion statements of holdings from libraries. A policy is needed to create a viable business plan for sustaining this resource.

What the policy will cover: rights and responsibilities of members that have created WorldCat — the rights of members to use elements of WorldCat and the shared responsibilities to the members of the cooperative that go along with the rights; identifying acceptable use by third parties; what are OCLC’s rights to use the records on behalf of the members; and a process for collective participation in reviewing and modifying the policy over time. It will also have a “rather robust” preamble that answers the question of why a policy is needed, what problem is the policy is trying to solve, and what it is about WorldCat that necessitates a policy.

An Aside: What’s In a Name — OCLC-the-membership and OCLC-the-stewards

The View from the Database Level

1. As a supply of bibliographic records.
2. The ability to represent library holdings — the collective collection of libraries and the capability to reveal what libraries have in places like Google Book Search.
3. Knowledge organization pieces: taking the shared contribution of libraries and makes something more from it using authority control, terminologies, the Dewey Decimal classification system, FRBR work sets, etc.

It was interesting to note a non-U.S. perspective that the council has heard regarding the value of WorldCat. While most North American libraries strongly value WorldCat as a supply of bibliographic records (copy cataloging), the national libraries outside of North America are joining because adding their records to WorldCat gives greater visibility to their holdings. So the second and third value propositions above carry more weight than the first, which is arguably the most valuable aspect for North American libraries.

The challenge the Council said it is facing is to put enough controls in place to protect the value and viability of WorldCat while allowing enough flexibility for members, non-members, and OCLC to experiment and derive new, valuable services. One of the questions the review council is grappling with is how can the Cooperative use “community norms” to ensure the responsibilities assigned to the members are followed so we govern ourselves.

In taking this database-wide view, the council has set aside issues of individual record ownership and copyright of data in records and focused on what is valuable about the collection of records as a whole to the membership. WorldCat as a whole collective is copyrighted. As explained in the follow-up discussion with members of the council, the intellectual property law surrounding WorldCat records extends across many juristictions, so the council chose to focus at the database level.

Third Parties

In licensing WorldCat data to others, OCLC-the-steward is looking for remuneration of some sort for OCLC-the-cooperative. If the business use by a non-member third party is one that will harm the value and viability of the WorldCat Network, then the policy council wants to see it governed in some way. Remuneration can be in monetary form, where that external party pays a fee for the data. Or it can be in a non-monetary form, such as the quid pro quo with the internet search engines that have WorldCat data and in return drive traffic back to local libraries through linkage on WorldCat.org. As Jay Jordan put it sucinctly, “I’ll do a contract with anyone that returns value to the cooperative. Oftentimes, that is not cash.” It was stated that there have been attempts to download the entire WorldCat database. In order to be able to legally stop that, there must be a policy in place that prohibits it.

WorldCat as Linked Data

Questions

1. In Jennifer’s introduction, she talked about not only the value of the bibliographic records but also the value of the holdings. Has the Council looked a differing policies for bibliographic information versus holdings?
2. The discussion of linked data was incomplete due to time constraints. Has there been a discussion about a differentiation of value for different types or views of data? Machine access version human-oriented access? Linked data of some portion of the bibliographic record? Is the representation of the benefit of the world in general being taken into account in drafting policy guidelines?