Threads this week without commentary. (It has been a long week that included only one flight of four that actually happened without a delay, cancellation, or redirection.) Big announcements are one from the Library of Congress to re-envision the way bibliographic information travels, one from Douglas County (Colorado) Library’s experiment with taking ownership of ebooks and applying its own digital rights management, and a study on the ecosystem of spam.

Feel free to send this to others you think might be interested in the topics. If you find these threads interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the right. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments and tips, as always, are welcome.

Transforming our Bibliographic Framework: A Statement from the Library of Congress

Spontaneous comments from participants in the US RDA Test show that a broad cross-section of the community feels budgetary pressures but nevertheless considers it necessary to replace MARC 21 in order to reap the full benefit of new and emerging content standards. The Library now seeks to evaluate how its resources for the creation and exchange of metadata are currently being used and how they should be directed in an era of diminishing budgets and heightened expectations in the broader library community.

- Transforming our Bibliographic Framework: A Statement from the Library of Congress, Bibliographic Framework Transition Initiative

Also see John Mark Ockerbloom’s Open data’s role in transforming our bibliographic framework for more details and links to other posts talking about the Bibliographic Framework Transition Initiative.

Douglas County Library to Distribute Ebooks with its own DRM

We are pleased to announce a partnership between the Colorado Independent Publishers Association (CIPA), and two Colorado libraries: Red Rocks Community College Library, and Douglas County Libraries.

Many members of CIPA have entered the world of digital publishing. By June of 2011, Red Rocks Community College Library and Douglas County Libraries will not only offer eBooks from CIPA’s authors for checkout through their library catalogs, but will also allow click-through purchases of these titles.

- New e-book partnership, Douglas County Libraries

There are more details on a post on the ALA Presidential Task Force on Equitable Access to Electronic Content blog along with an earlier post about that library’s experiments with Adobe Content Server.

Study Says Spam Can Be Cut by Blocking Card Transactions

For years, a team of computer scientists at two University of California campuses has been looking deeply into the nature of spam, the billions of unwanted e-mail messages generated by networks of zombie computers controlled by the rogue programs called botnets. They even coined a term, “spamalytics,” to describe their work.

Now they have concluded an experiment that is not for the faint of heart: for three months they set out to receive all the spam they could (no quarantines or filters need apply), then systematically made purchases from the Web sites advertised in the messages.

- Study Says Spam Can Be Cut by Blocking Card Transactions, by John Markoff, New York Times

Thanks to everyone for participating in the first Code4Lib Virtual Lightning Talks on Friday. In particular, my gratitude goes out to Ed Corrado, Luciano Ramalho, Michael Appleby, and Jay Luker being the first presenters to try this scheme for connecting library technologists. My apologies also to those who couldn’t connect, in particular to Elias Tzoc Caniz who had signed up but found himself locked out by a simultaneous user count in the presentation system. Recordings of the presentation audio and screen capture video are now up in the Internet Archive.

Lessons Learned

Second, some comments I got were about cranky Java applets and applications. LYRASIS has two conference tools at its disposal — Java-based Centra and Flash-based Acrobat Connect — and I chose Centra because running Flash on LINUX is an issue. Maybe this will need to be revisited (or maybe there is another Java-based conference system that can do better).

Third, since we were not limited by space and other timing constraints, can the five-minutes-per-presenter limit be relaxed? I have mixed feelings about this; I think defined time limits promote better presentations, but the four presentations this first go-around went to the end of the five minute time limit and there was no opportunity for questions or audience interaction.

On the whole, it seemed like a positive experience from my perspective and from that of the feedback I’ve received so far. I’m going to start a conversation thread in Code4LibCon (where all of the Code4Lib meeting planning discussion takes place) to see if it is worthwhile to do again and to identify what should be done differently. If you are interested, please consider joining and contributing to the discussion. Or e-mail me privately and I’ll reflect your comments into the group discussion.

Footnotes

1. “In addition to the continuous maintenance of the standard described above, a comprehensive review of a database standard at regular intervals may be necessary which is organized in accordance with the rules in the ISO/IEC Directives and the ISO Supplement for the systematic review process.” Procedure for the development and maintenance of standards in database format. Annex ST of the ISO supplement to the ISO/IEC Directives.

Eric Morgan posted a message to the Next Generation Catalog for Libraries mailing list this morning that points to a announcement by the University of Florida library that they are now applying a Creative Commons Public Domain Dedication statement to MARC records they create. Their announcement says:

Beginning March 2011, the University of Florida Smathers Libraries implemented a policy to include a Creative Commons license in all of its original cataloging records. The records are considered public domain with unrestricted downstream use for any purpose.

The following MARC 588 field (Source of Description Note) is added to new records contributed to WorldCat. It has not been added retrospectively to University of Florida original records in WorldCat.

588::|a This bibliographic record is available under a Creative Commons CC0 license. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.

Their announcement page also provides links to some examples from the OPAC. Scroll down to the bottom of the page to see the CC0 graphic and declaration.

• Latinoamericanismo : historia intelectual de una geografía inestable
• Mapa Everest de carreteras, España y Portugal
• The Song of Ceylon

The University of Florida joins University of Michigan in making original cataloging records available under CC0. To refresh your memory CC0 “Public Domain Dedication” statement (it isn’t a license!) says:

The person who associated a work with this deed has dedicated the work to the public domain by waiving all of his or her rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.

You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.

Now this is an interesting development because it is a kind of viral declaration of the same sort that OCLC proposed with the withdrawn draft of the record use policy. There OCLC was going to add a 996 field to all records exported from WorldCat that would say:

This would do something similar except that it would make viral the public domain declaration on records added to OCLC WorldCat. (It is probably also an oversight that the 996 field documentation is still on OCLC’s site.) Does this begin to segment WorldCat into records that can and cannot be used? Or is it redundant since some think that MARC records, as a recitation of facts, cannot be copyrighted anyway?

It wasn’t too long ago that the music industry was in an uproar about stories of how easy it was to copy digital audio files and make digital copies with high fidelity. It was predicted that we would see the same thing in other media forms, and this week’s DLTJ Thursday Threads has two stories on the topic of book publishing. First is news of another inexpensive and simple (and now to be commercially produced) book digitizing system. Although the process of “ripping” a book from its physical medium might take longer than an audio track, these kind of devices are emerging that will make it simple to do. What happens with the digital copy after that? The second Thursday Threads pointer is to an interview with the founder of book publishing industry consultant about the state of book piracy, how it is measured, and why digital rights management software is a poor way to stop it. The last entry this week is a short excerpt of a brief summary of a study conducted by OCLC last year on the usage of MARC tags in cataloging records.

As a side note, apologies to DLTJ readers that had problems reading some of the content here over the past couple of weeks. A series of problems with my personal server — driven by the fact, I believe, that the server was first set up about 10 years ago and all the patches, tweaks, and updates over the decade have finally driven performance into the ground — prompted me to migrate this blog to Amazon’s Web Services cloud. It is now running on a micro Elastic Cloud Computing (EC2) virtual machine backed by Simple Storage Service (S3) and the CloudFront content distribution network. I’ve also been optimizing the snot out of configuration — employing all sorts of new tricks for reducing the time it takes to deliver pages to your browser. I have another blog post in draft with the details for when anyone (even me!) wants to replicate it. Given enough personal time, watch for that in the next week or so.

All of that said, if you are seeing things that don’t look or function right, please let me know.

Book Saver – A personal book digitization setup from ION

Picture and Demonstration Video of the Book Saver from ION

Book Saver has two cameras that take separate images in rapid succession of each page within an open book. Both cameras of Book Saver also have a flash for allowing the page to be fully illuminated during the scanning process. Book Saver’s cradle, where the book is placed during the scanning process, is also angled as to not require you to hold pages down to get a flat, even surface. While similar devices require up to seven seconds per one page, Book Saver takes only one second per two pages!

News of the new Book Saver product comes from Fiacre O’Duinn. It is a hand-held device for digitizing book materials. The promotional literature says it takes about 15 minutes to digitize a 200-page book. The product was announced in time for the Consumer Electronics Show earlier this month, but is not yet available. It is expected to ship this summer with a manufacturer’s suggested retail price of $189 (I’m already seeing price points of $149 mentioned).

One of the “Key Features” listed on the product page is that the device “eliminates the need to purchase electronic versions of reading material you already own.” As Fiacre points out in his post, this really brings down the cost (in equipment and in effort) of digitally reproducing books. Are we about to see a new wave of personal book sharing/piracy? And what will the impact on libraries be? In the higher education arena, it is already being mentioned as a way to digitize textbooks. It is conceivable that students would borrow textbooks from our libraries, digitize them in an afternoon, and return them — or maybe just digitize them in the library. Do we need to get ahead of devices like this with education and policy initiatives?

Book Piracy: Less DRM, More Data

As digital book publishing continues to expand at a rapid pace to meet reader demands, piracy rears its head at the forefront of many a discussion in publisher circles. Many publishers respond to the perceived threat with strict digital rights management (DRM) software. But is this the best solution? And does it even provide protection from piracy?

In the following interview, Magellan Media founder and TOC 2011 speaker Brian O’Leary (@brianoleary) discusses the current state of book piracy, how measurement data isn’t sufficient to determine its impact, and why DRM is a poor anti-piracy tool.

The same arguments in favor of digital rights management for the music sector are now being made in the book publishing sector. This interview comes from the perspective of why DRM is the wrong answer to the perceived problem of book piracy. The backdrop is O’Reilly Media’s Tools of Change for Publishing conference to be held next month in New York City.

Core Bibliographic Description

Those “outliers” can be categorized according to three general purposes:

• Provenance and Identity: identifiers (e.g. ISBN, OCLC, etc.) and cataloging source (040)
• Elements useful for discovery: title statement (245), personal names (100, 700) and subject (650)
• Elements useful for understanding and evaluation: publication statement (260), physical description (300), and notes (500)

That’s it. In a nutshell you have the very core of bibliographic description as defined by librarians over the last century or so.

This post by Roy Tenant briefly summarizes the work of OCLC Research staff member Karen Smith-Yoshimura. The research work was to gather evidence to inform changes in MARC metadata practices, and that project page includes a 72 page report [PDF] and an Excel spreadsheet of data tables along with audio and video of a one hour webinar on the report. In my FriendFeed posting of Roy’s article, Walt Crawford noted a similar finding in his 1986 Bibliographic displays in the online catalog. As Walt notes, “somehow it’s not surprising that it’s still true in 2010.”

In preparation for the last webinar of the three-part series “Using RDA: Moving into the Metadata Future“, I’m reading again Karen Coyle‘s “Library Data in a Modern Context” — the first chapter ofUnderstanding the Semantic Web: Bibliographic Data and Metadata. Right at the start she has a clear and useful definition of this thing we call “metadata.”

The most common definition of metadata is “data about data.” This short, catchy definition is worthy of a successful advertising campaign. Unfortunately, it doesn’t really help us understand metadata, and is actually somewhat incorrect. A more useful definition is decidedly less snappy, but can help us understand the helpful role that metadata can play in facilitating information access. In fact, a functional definition gives us a viable roadmap for our own studies of metadata utility and quality.

So here it goes—metadata is constructed, constructive, and actionable:

• Constructed: Metadata is not found in nature. It is entirely an invention; it is an artificiality.
• Constructive: Metadata is constructed for some purpose, some activity, to solve some problem. The proliferation of metadata formats that seem similar on the surface is often evidence of different definitions of needs or of different contexts. We may dream of a universal set of metadata for some set of things, like biological entities, printed books, or a calendar of events, but are likely to be disappointed in practice.
• Actionable: The point of metadata is to be useful in some way. This means that it is important that one can act on the metadata in a way that satisfies some needs.¹

A little further on Karen focuses on the actionablity of metadata. I have a heightened awareness of the need for other-than-visual access to information based on the last few months of activity with my previous employer, so I reread this section with “new eyes” (so to speak):

…today’s metadata must be in a form that can be processed by computers, and the sense that it is “actionable” really needs to be interpreted as being “actionable by electronic machines.” Even when the final goal is to display the data to humans in an understandable form, the data will undergo some machine processing on the way to its destination on a screen [or] in printed form or when read aloud by a screen reader.

I added that last part. The transformation of the meaning of the metadata into a visual form is but one possible sensory input across the human-computer divide. It is important to also design interfaces that bring meaning to data by supplying labels to values in alternate ways. For the MARC 300 field, it is the difference between “ix, 74 p. : ill. ; 23 cm” and “9 pages of introductory material followed by 74 numbered pages. Includes illustrations. 23 centimeters high.” If the only way to transmit this information was auditory, which one of these would you like spoken to you? Is it: “eye-ex, seventy four pee. ill. twenty three cem”?

Now let’s try to engineer that backwards. Is the auditory version easier to do with:

300    |aix, 74 p. :|bill. ;|c23 cm

or something like this made-up, MODS-like markup:

<physicaldescription>
  <extent>
    <pagination>
       <pages type="introductory">9</pages>
       <pages type="numbered">74</pages>
    </pagination>
    <illustration />
    <height unit="cm">23</height>
  </extent>
</physicaldescription>

With the second, we can produce something like the first — or even the abbreviated display version. But it is considerably more difficult to create the auditory version from the first, particularly with the wide variation of punctuation encoding ISBD offers. It just isn’t machine actionable, which makes it difficult to transform, reuse, and translate that data in another context.

I’m reminded too of this recent quote from Jonathan Rochkind: “Of course, our legacy environment is even worse, with the ‘data model’ being supplied by an unholy combination of ISBD … and MARC….” It would be good to stop doing our data entry in the language of the computer (e.g. MARC). Based on the chat from the first webinar in the series, we wouldn’t expect catalogers to type out the XML fragment above. There should be computer-assisted workflows to capture the data and store it with all the required semantics. That XML would be used for machine-to-machine communication and transformation into the output desired by the user — be it a short-hand visual display or an auditory reading of information.

Footnotes

1. Coyle, Karen. “Library Data in a Modern Context.” Library Technology Reports 46.1 (2010): 5-13.

This week I sat in on the first of the three “Using RDA: Moving into the Metadata Future” webinars being hosted by ALA. This one was hosted by Karen Coyle with the title New Models of Metadata where she talked about library-specific efforts such asRDA and FRBR as well as the linked data effort in the wider world of information. There was a great deal of concern expressed in the chat window by participants about the future of cataloging, of cataloguers, and of MARC. The latter brought up memories of Roy Tennant‘s “MARC Must Die” declaration. My take away, though, isn’t that MARC is dead as much as MARC is a dead end.

Cover art from 'Library of the Dead' audio book

MARC, not dead yet?

What trips up our community even more, I think, is that we have a tendency to equate this communications format with mental model of how we describe things from a bibliographic point of view. We think of discrete records that describe these things rather than a network (or, more accurately, a graph) of interrelated nodes. This forces us to focus on the textual content of fields and not on the relationships between things. And in doing so, we are not making the best use of our limited efforts to describe the things in our curatorial care.

MARC may not be dead, but it is a dead end.

This is definitely becoming a habit…welcome to the fourth edition of DLTJ‘s Thursday Threads. If you find these interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the left. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments, as always, are welcome.

Defining Linked Data By Analogy

RDF is the grammar for a language of data. URIs are the words of that language. As in natural language, these words (i.e., the URIs) belong to grammatical categories. RDF properties (such as “isReferencedBy”) function a bit like verbs, RDF classes like nouns.

As in natural languages, where utterances are meaningful only if they follow a sentence grammar, RDF statements follow a simple and consistent three-part grammar of subject, predicate, and object. Analogously to paragraphs, RDF statements are aggregated into RDF graphs.

This is a posting from Thomas Baker on the W3C Library Linked Data exploratory group mailing list. It compares RDF to natural languages using analogies of grammar, words, sentences, and paragraphs. I think this is a useful way to think about RDF and linked data, although as initial introduction to the topic, you might want to see the presentation below.

RDF For Librarians presentation recording

The RDF model underlying Semantic Web technologies is frequently described as the future of structured metadata. Its adoption in libraries has been slow, however. This is due in no small part to fundamental differences in the modeling approach that RDF takes, representing a “bottom up” architecture where a description is distributed and can be made up of any features deemed necessary, whereas the record-centric approach taken by libraries tends to be more “top down” relying on prespecified feature sets that all should strive to make the best use of. This presentation will delve deeply into the differences between these two approaches to explore why the RDF approach has proven difficult for libraries, look at some RDF-based initiatives that are happening in libraries and how they are allowing different uses of this metadata than was previously possible, and pose some questions about how libraries might best.

Jenn Riley gave this hour-long presentation to the Indiana University Digital Library Brown Bag earlier this month. The URL to the slides synchronized to the audio recording is http://breeze.iu.edu/p48776227/. The presentation slides and the handout from the session are available as well. I highly recommend spending an hour with this presentation to learn about how linked data compares and contrasts with MARC records. (via Diane Hillmann)

The Future of the Federal Depository Libraries

[ProPublica's Dafna] Linzer’s expose of government tampering with a court docket is an example of the problem on which the LOCKSS Program has been working for more than a decade, how to make the digital record resistant to tampering and other threats. The only reason this case was detected was because Linzer created and kept a copy of the information the government published, and this copy was not under their control. Maintaining copies under multiple independent administrations (i.e. not all under control of the original publisher) is a fundamental requirement for any scheme that can recover from tampering (and in practice from many other threats).

David Rosenthal summarizes a story about how a published document from the U.S. government was changed and why we need highly-distributed copies of government documents to detect and recover from tampering. There are big implications here for the future of government documents depository programs.

ProPublica’s Guide to Mechanical Turk

Amazon Mechanical Turk – or mTurk – is an online marketplace, set up by the online shopping site Amazon, where anyone can hire workers to complete short, simple tasks over the Internet. Amazon originally developed it as an in-house tool, and commercialized it in 2005. The mTurk workforce now numbers more than 100,000 workers in 200 countries, according to Amazon. At ProPublica, we use it for tasks like collecting, reformatting, and de-duplicating data. This is a guide to journalists looking to use Mechanical Turk in their data projects. It’s meant for users who are already familiar with mTurk and are looking for ways to improve their results.

Do you have repetitive digital conversion or analysis jobs that can be broken down into manageable-sized chunks? ProPublica published this guide on using Amazon’s Mechanical Turk service to outsource this activity.

This year the ALCTS Forum at ALA Midwinter brought together three perspectives on massaging bibliographic data of various sorts in ways that use MARC, but where MARC is not the end goal. What do you get when you swirl MARC, ONIX, and various other formats of metadata in a big pot? Three projects: ONIX Enrichment at OCLC, the Open Library Project, and Google Book Search metadata.

Below is a summary of how these three projects are messin’ with metadata, as told by the Forum panelists. I also recommend reading Eric Hellman’s Google Exposes Book Metadata Privates at ALA Forum for his recollection and views of the same meeting.

ONIX Enrichment at OCLC

Renee Register, Global Product Manager for OCLC Cataloging and Metadata Services, was the first to present on the panel. Her talk looked at a new and evolving product at OCLC on the enhancement of ONIX records with WorldCat records, and vice versa. ¹

As libraries, Renee said “our instincts are collaborative” but “our data and workflow silos encourage redundancy and inhibit interoperability.” Beyond the obvious differences in metadata formats, the workflows of libraries differ dramatically from other metadata providers and consumers. In libraries (with the exception of CIP and brief on-order records) the major work of bibliographic production is performed at the end of the publication cycle and ends with the receipt of the published item. In the publisher supply chain, bibliographic data evolves over time, usually beginning months before publication and continuing to grow for months and years (sales information, etc.) after publication. Renee had a graphic showing the current flow of metadata around the broader bibliographic universe that highlighted the isolation of library activity relative to publisher, wholesaler, and retailer activity.

Diagram of the Process of Enhancing ONIX Records, from OCLC Services for the Publisher Supply Chain Webinar, August 2009

In her experience, Renee said that libraries need ways to enable our metadata to evolve over time and allow for publisher-created metadata to merge effectively with library-created metadata. The bibliographic record needs to be a “living, growing” thing throughout the lifecycle of a title and beyond. In concluding her remarks, she offered several resources to explore for further information: the OCLC/NISO study on Streamlining Book Metadata Workflow, the U.K. Research Information Network report on Creating Catalogues: Bibliographic Records in a Networked World, the Library of Congress Study of the North American MARC Records Marketplace, the Library of Congress CIP/ONIX Pilot Project, and the OCLC Publisher Supply Chain Website.

From MARC to Wiki with Open Library

Karen said right at the start that the Open Library project is different from most of what happens in libraries — it is “someone outside the library world making use of library data” — although the goal is arguably the same as others — “One web page for every book ever published.” As such, the Open Library isn’t a library catalog as librarians think of it in that it is not a representation of a libraries inventory. It has metadata for every book it can know about and a pointer to places where the book can be found, including all of the electronic books in Internet Archive (Open Content Alliance, Google Public Domain, etc.) as well as pointers back to OCLC WorldCat. Karen’s role for the project is that of “Library Data Informant.” The Internet Archive decided that they needed someone who understood library data in order to try to use it. From Karen’s perspective, she is trying to be a resource for project but not give them any guidance on how to implement the service. She is curious to see what the project would do when bibliographic data is viewed from a non-librarian perspective. If they have questions, or if they have assumptions about data that are wrong, then she intervenes.

Karen went on to briefly describe the Open Library system. Open Library doesn’t have records; rather, it has field types and data properties. In this way, it uses semantic web concepts. “Author” is a type, “Author birthdate” is another type, and so forth. There are no set field types, so if the project gets data from source for which a type doesn’t yet exist, it can create a new one. Each type can have data properties such as string, boolean, text, link, etc. Nothing is required and everything is repeatable. Everything — types, properties, and values — gets a URI (a URI is an identifier like a URL, but conceptually a superset of the universe of URLs). Titles, authors, subjects, author birthdates, and so on have URIs. Lastly, the underlying data structures are based on wiki principles: all edits are saved and viewable, anyone can edit any value, anyone can add new types or properties, anyone can develop their own displays, etc.

The data that is now in Open Library came from a variety of sources. They started with a copy of books from the Library of Congress, and continue to receive the weekly updates. They performed a crawl of Amazon’s book data. They have gotten some from publishers, libraries, and individual users. The last is perhaps the most interesting because it is mainly people outside the western world who are otherwise having trouble getting their works recognized.

Problems, Issues, Challenges, and Opportunities with the Data

Names -

“These library forms of names? Honestly no one but us can stand them.” Even something as simple as the form of last-name-comma-first-name is troublesome. No one else uses this form of the name: Amazon, Wikipedia, etc. In processing these, any information between parenthesis has been deleted, birth and death dates move into separate field types.

Titles -

In working with the Open Library developers, this is one place that Karen tried insisting on applying a library practice: knowing the initial article. For us, this is important for sorting books in alphabetical order. The developer response — why do we have to sort in alphabetical order? “Where else but library catalogs to we see things sorted in alphabetical order? Not in Google, not in Amazon, not anywhere. Alphabetical order is not in the mindset anymore.” They also found that the title might include extraneous data. Amazon, for instance, appends the series title in parenthesis to the main title. This is a demonstration of how other communities are not as concerned about strongly typing and separating information into fields. Amazon, of course, has reasons for series information into the main title: it helps sell books.

Product dimensions -

Publishers and distributors need to know characteristics of an item such as height, width, depth, and weight; they, of course, need to put it in a box and ship it. Libraries, concerned about placing the item on the shelf, record just height. Recording pagination is different, too: libraries use odd notations “ill. (some col)” and “xv, 200p.” versus simply “200 pages.”

Birthdates -

Librarians use birthdates to distinguish names; if there is no need to distinguish a name, birth and death dates are not added. Someone looking at this from the outside would ask ‘Why don’t all authors have birth and death dates?’ This can be useful information for viewing the context of an item, not just to distinguish author names. Open Library ran author names against Wikipedia to pick up not only birth and death years, but also the actual dates.

Subject headings -

Open Library using Library of Congress Subject Headings was out of the question. In processing the data, the Open Library developers just broke them apart into segments and used them. But because they were able to do data mining on the subject field types, they did find statistical relationships between the disassembled precoordinated headings and were able to present those to the user.

The View of the Data -

Rather than a traditional library view of long lists of author-title, the Open Library (in its next version coming in February) will have several different views into the mass of data: Authors; Books (what we would call FRBR ‘manifestations’); Works; Subjects; and eventually places, publishers, etc. For example, when searching for an author one would get the author page. On it would be all of the works from the author as well as other biographical information. It looks similar to a WorldCat identities page, except it is the actual user interface built into the system. Similarly, every work will have a page, and at the bottom of it one will see all of the editions of the work. Also, each subject will have a page, and one will see a list of works with that subject as well as authors who write on that subject. As Karen said, “The subject itself becomes an object of interest in the database, not just something that is just tacked on to the bottom of the library record.”

Data mining -

With the data in this format, it is possible to perform data mining actions against it. For instance, simple data mining such as country of publication, popular places that appear, etc. When they had the problem of author names — knowing when to reverse surname and forname — they ran the names against Amazon and Wikipedia and retained the ones where they found the order of the entry was the same. The Open Library developers are also experimenting with data mining to find publisher names. Publisher names, of course, vary dramatically, but by using ISBN prefixes they can pull together related items into a “publisher” view.

Karen suggested watching the Edward Betts’s site, one of the developers of the Open Library project with an eye on the data mining aspects. She said it is fun to look at our data when it can be viewed from this different point-of-view. She also said to watch out for a new version of the Open Library website coming in February.

Google Book Search Metadata

On the input side, Google is getting bibliographic metadata from over 100 sources in a variety of formats. MARC records are coming from libraries, union catalogs, commercial providers (OCLC), publishers/retails (one publisher supplies records in MARC format). Google also gets ONIX records from commercial providers (such as Ingram and Bowker), publishers, and retailers. Google is especially interested in data from non-U.S. retailers because it is a source of information about books published outside the United States; it helps facilitate discovery of items that they may not otherwise encounter in the publisher and library programs. Google also receives records in a variety of “idiosyncratic formats” — for example, publisher-contributed metadata (via the Publisher Partner Program); information associating books with jacket images; name authority records (from LC); reviews; popularity signals (sales data as well as anonymized circulation data from some library partners, useful for feeding into the relevancy ranking algorithm); and internally-generated metadata (for instance, whether a book is commercially available or not). Google processes all of this information to come up with a single record that describes a book. At this point they have over 800 million bibliographic records and one trillion bits of information in those records.

All of these records from all of these sources are processed and remixed with Google’s parsing algorithms about twice a week. The first step is to transform the incoming records into a “less verbose format” for storage and processing. It is a SQL-like structure that allows elements of the metadata to be queried. Records are then parsed to extract specific bits of information, transform the bits as necessary, and write the information to an internal “resolved records” data structure (a subset of the data coming from the input formats). In the presentation, Kurt had examples of how making inferences from data coming from both MARC and ONIX can be troublesome. Parsing also involves extracting “bibkeys” from the records to aid in matching across sources of data. Four types of identifiers are extracted from bibliographic records: OCLC numbers, LCCNs, ISBNs, and ISSNs. They provide usually useful signals when matching bibliographic and help with assertions that two records describe the same manifestation. Google also tries to parse item data when present in records representing multi-volume works, enumeration and chronology. They will also treat barcode as a form of a “bibkey” if they get it from a library. The parsing algorithm will also split records containing multiple ISBNs representing different product forms (e.g. hardback, paperback, etc.).

With all of this data parsed into records, Google starts its clustering process where records are examined and attached to each other. Bibkeys provide significant evidence for relating records to each other, but bibkeys are not always present in a record (non-U.S. records and older records frequently contain no bibkeys). The algorithms then fall back on text similarity matching using title, subtitle, contributor and other fields such as publisher and publication year. The results are clusters of records representing the same manifestation. An algorithm then attempts to derive the “best-of” record for a single cluster from all of the parsed input records. This is done in a field-by-field voting process based on the trustworthiness of individual fields from record sources.

Kurt went into some of the challenges facing the team building the clustering and best-of record creation algorithms. For instance, in dealing with multivolume works they know of 5 numbering schemas with 3 number types in 15 different languages. Enumeration is now showing in the public display, but the development team is still working with unparsable item data due to inconsistent cataloging practices between institutions…and sometimes inconsistencies within an institution. Another problem is non-unique identifiers. In the current data set ISBN 7899964709 is shared by 75 books and ISBN 7533305353 is associated with 1413 books. There are also poor quality or “junk records”. Kurt said his favorite was “The Mosaic Navigator” by Sigmund Freud published in 1939. These are hard to identify with an algorithm, and they rely on reports of problems that enable the developers to go in and “kill” the troublesome record. Another example is a book by Virginia Woolf where the incoming record had conflicting information; it had two 260 fields that contained different dates (1961, correct, and 1900) with fixed field information that strongly suggested that 1900 was the single date of publication. When the data problem is systematic, they can identify it and compensate for it. Kurt’s example for this case was “The United States Since 1945″ published in 1899. This one was highlighted in Geoffrey Nunberg’s criticism of Google Books metadata. In this case, there was a source of metadata from Brazil that when they didn’t know the date of publication would use 1899. When Google went back and looked at the date distribution of books there was a huge spike in 1899. Once Google knew about it they were able to go in and kill that information from that source of records. ³

In closing, Kurt said that Google is committed to engaging with the library community on improving metadata and metadata processing.

The text was modified to update a link from http://www.niso.org/publications/white_papers/Stream lineBookMetadataWorkflowWhitePaper.pdf to http://www.niso.org/publications/white_papers/StreamlineBookMetadataWorkflowWhitePaper.pdf on January 19th, 2011.

The text was modified to remove a link to http://www.oclc.org/speakers/bios/register_renee.htm on February 11th, 2011.

Footnotes

1. For those not familiar with ONIX, it is a suite of standards promulgated by EDItEUR for the interchange of information on books and serial publications. It is primarily used as the communication channel between the publishing industry through distribution chains to retail establishments.
2. By the way, it seems like BISAC is an acronym for “Book Industry Systems Advisory Committee”, the former name of the Book Industry Study Group.
3. A side note: Google isn’t the only one tripped up by this. If one searches for the ISBN of the item, 0195038487, you get to more than one site that has the same incorrect publication date. At least Google is attempting to clean up the data!

At ALA Midwinter, ALCTS sponsored a panel discussion about sharing library-created data inside and outside the library community, with a particular focus on cataloging data. I was honored to be ask to speak on the topic from the perspective of a consortial office. This is the second and final post in a series that represents an approximation of what I said on the panel.

The first part examined the nature of surrogate records that we create as a means to get users to content. The post looked at where we get records, how humans and machines can create them, and the rights associated with component data that makes up the records.

Right to reuse records without restrictions

The first area where is important is with discovery layers. A new generation of discovery layers are taking surrogates from a variety of sources – catalogs, publishers, index/abstract services, etc. – and performing actions such as consolidating records and building relationships between surrogates. These derived surrogates are presented to users in new interfaces or new portals into existing interfaces. Examples of these systems are the Extensible Catalog project at the University of Rochester and the newly announced Serials Solutions Summon product. OhioLINK recently solicited responses from vendors where this kind of capability is a key product of a new discovery layer. Other projects (such as subject-specific portals) also seek to re-purpose the data – mix it up with other sources of data to create new uses and views that are specific to a particular user community. Anything other than a permissive-for-all-by-default will put up roadblocks and cause builders of these systems to seek data from other services.

In addition to presenting the surrogates to users in new ways, libraries are also investigating new forms of workflow and collaborative activities surrounding the creation and maintenance of bibliographic records. One of the strong desires of many involved in the OLE Project is cooperative purchasing and cooperative technical services. OhioLINK has also recently issued an RFI seeking new options for highly collaborative workflows in the maintenance of surrogate records. Old models of charging for use of records can hinder the ability of cooperating institutions to optimize costs and efforts of back-room library options.

The elephant in the room is the recently proposed OCLC Records Use Policy. Setting aside the debatable legal framework under which OCLC asserted the right to set a usage policy on records from the cooperative, there were clauses in the proposed policy that jeopardize the usability of records, and as a consequence the viability of the cooperative as a whole. Actions that restrict use of data or create uncertainty around the use of data lessen the value of that data. I think few would argue that value can be created by aggregating services on top of the data; the activities in the WorldCat Grid and Developer’s Network point to that. Revenue could be generated in fees charged to non-cooperative members. It is conceptually important to separate the hosting of the surrogates from the layered services on top of them – WorldCat Local, mediated ILL, collection analysis, by way of example.

Parting thoughts

Post-panel Thoughts

Thanks to Charles Wilt, Executive Director of ALCTS, for inviting me to speak at the ALCTS Forum and to Karen Calhoun for facilitating the invitation. My appreciation also goes out to my co-panelists: Karen Calhoun (who has posted her slides online), John Mark Ockerbloom (who also posted his slides and approximate speech transcript), and Brian Schottlaender (who eloquently summarized statements from the other panelists and took point in fielding questions from the audience).

The text was modified to remove a link to http://lisnews.org/node/add/blog on January 13th, 2011.