It is an all e-books edition of DLTJ Thursday Threads this week. The biggest news was the announcement of the policy change by HarperCollins for ebooks distributed through OverDrive. Beyond that, though, was an announcement of a new sharing model and program through the Internet Archive. Lastly is a slidecast recording of a presentation by David Lewis on the future of library collections.

Before continuing, a quick apology and explanation. E-mail readers received a pair of extra Thursday Threads messages and RSS subscribers got a dump of unrelated posts; I’m sorry. The cause was an update of this blog’s WordPress software to version 3.1 and a conflict (maybe this one) with the SimpleTags plugin. I believe all is well, but I won’t know until this post is published.

Feel free to send this to others you think might be interested in the topics. If you find these threads interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the right. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments and tips, as always, are welcome.

HarperCollins Puts 26 Loan Cap on Ebook Circulations

In the first significant revision to lending terms for ebook circulation, HarperCollins has announced that new titles licensed from library ebook vendors will be able to circulate only 26 times before the license expires.

Mention of the new terms was first made in a letter from OverDrive CEO Steve Potash to customers yesterday. He wrote [emphasis in original]:

[W]e have been required to accept and accommodate new terms for eBook lending as established by certain publishers. Next week, OverDrive will communicate a licensing change from a publisher that, while still operating under the one-copy/one-user model, will include a checkout limit for each eBook licensed. Under this publisher’s requirement, for every new eBook licensed, the library (and the OverDrive platform) will make the eBook available to one customer at a time until the total number of permitted checkouts is reached.

Though the letter leaves the publisher unnamed, HarperCollins confirmed today to [Library Journal] that it is the publisher referred to.

In an odd one-two punch, this past week saw a disturbance in the status quo of e-book licensing. The first punch came in the letter from OverDrive [PDF] (part of which is quoted in the Library Journal article excerpted above). The second in that Library Journal article when we learned that the publisher pushing for the change of terms is HarperCollins. Since then it has been the source of a great deal of discussion by librarians and a few authors, much of it in the form of tweets with the hash-tag “#hcod” (short for HarperCollinsOverDrive). Damage control comes in the form of open letters from OverDrive and HarperCollins. There has been a call for a boycott. Bobbi Newman, one of the first to jump on the story, is maintaining a list of news articles and commentary.

As you can see, much has already been said about the issue, and since collection development is not my specialty, you probably shouldn’t look to me for an informed opinion. (If pressed, I will suggest that it is a perfectly reasonable collection development policy to not buy access to material with terms that are not in the library’s and patron’s best interest.) Instead, there is so much oddness in this new policy that I find I can’t put myself in HarperCollins’ shoes. First, using OverDrive as a proxy for announcing this policy change seems wrong (and, frankly, unfair to OverDrive). Then as word spreads, you don’t make your own announcement, but rather talk to a reporter from Library Journal. Then as news spreads through the day, you send a single tweet. In fact, you don’t really publicly respond until five days after the twitter universe and biblio-blogosphere have been talking about it. And it is pretty much a non-engaging, public relations response. (You do get credit, though, for allowing open comments on your blog post. But some of that credit is taken back because you aren’t using a company branded blog. Really? A typepad.com blog? One of the tenants of most information literacy courses I’ve seen is to look for the source of the information, and it requires extra effort to take this blog seriously because it isn’t in the harpercollins.com domain space.)

If I were to guess, this seems like a trial balloon that was badly floated. I certainly can’t fault HarperCollins for trying something new in the ebook licensing world, but this one has fallen flat.

Internet Archive and Library Partners Develop Joint Collection of 80,000+ eBooks To Extend Traditional In-Library Lending Model

Today [February 22, 2011], a group of libraries led by the Internet Archive announced a new, cooperative 80,000+ eBook lending collection of mostly 20th century books on OpenLibrary.org, a site where it’s already possible to read over 1 million eBooks without restriction. During a library visit, patrons with an OpenLibrary.org account can borrow any of these lendable eBooks using laptops, reading devices or library computers. This new twist on the traditional lending model could increase eBook use and revenue for publishers. …

Any OpenLibrary.org account holder can borrow up to 5 eBooks at a time, for up to 2 weeks. Books can only be borrowed by one person at a time. People can choose to borrow either an in-browser version (viewed using the Internet Archive’s BookReader web application), or a PDF or ePub version, managed by the free Adobe Digital Editions software. …

Publishers selling their eBooks to participating libraries include Cursor and OR Books. Books purchased will be lent to readers as well as being digitally preserved for the long-term. This continues the traditional relationship and services offered by publishers and libraries.

This press release from the Internet Archive largely went unnoticed on the eve of the #hcod onslaught. It was covered in the Chronicle of Higher Education’s Wired Campus blog and in Library Journal. The latter has a few more helpful details: “IA founder Brewster Kahle and director Peter Brantley also told LJ that small independent publishers Cursor, OR Books, and Smashwords will donate ebooks license-free to the Open Library for lending to all Open Library members. With this venture, IA hopes to establish a “first-sale precedent” for e-lending, according to Brantley.” One must be from one of the participating libraries to check out books. My experience with most Internet Archive efforts is that the initial announcement is very subtle and not picked up widely, then slowly grows to something substantial. I expect this project will follow much the same path and will have a noticeable imprint on the profession in a few years.

Slidecast of David Lewis’ “Collections Futures” Talk

• Context
  • The Big Shift
  • Interlude with Clay Shirky
  • A Bit of Disruptive Innovation Theory
• Collections in “A Strategy for Academic Libraries in the First Quarter of the 21st Century”
• What Will Be Easy and What Will Be Hard

So far in DLTJ Thursday Threads I’ve intentionally avoided pointing to items inside this blog — preferring to link to events, resources, and conversations elsewhere. I’m going to make sort-of-an-exception in this case because what I’m ultimately pointing to is not my work. It is a slidecast (recorded audio synchronized to slides) of David Lewis’ presentation at the 2010 Annual RLG Partnership Meeting. Starting with a foundation from John Hagel III, John Seely Brown and Lang Davison called the “Shift Index” [PDF], Clay Shirky’s “How Social Media Can Make History” TED Talk, and Clayton Christensen’s disruptive innovation theories, David walks through the possibilities for three strategic issues facing academic libraries: Complete the migration from print to electronic collections; Retire legacy print collections; and Migrate the focus of collections from purchasing materials to curating content. The slidecast is about 75 minutes long and well worth the time as a thought-provoking view of what libraries should be doing to survive the next few decades.

The highlights of the past week are around publishing — first with a model proposed by Eric Hellman in which consumers can pool enough money to pay publishers to “set a book free” under a Creative Commons license, then with an announcement by the University of Pittsburgh offering free hosting of open access e-journals. Since we have to be able to describe and find this content, their bibliographic descriptions are important; John Wilkin proposes a model for open access to elements of bibliographic descriptions. Rounding out this week’s topics are a report of a master’s degree program in business using Facebook, and tips for planning an unconference meeting.

Paying Publishers to Set their Content Free

[Eric] Hellman’s new model is something he calls GlueJar. He proposes to “unglue” e-books from their publishers so that they can be available to the world, DRM-free and under Creative Commons license. Here’s the model: publishers sign on with works that they want to “unglue.” They determine what they are willing to be paid for ungluing each work. Users contribute money towards the ungluing. When the threshold amount is reached for a given title, that title is unglued: it appears in all contributors’ e-book reader libraries and in repositories used for online public library access. The publisher is paid, and GlueJar takes a commission.

In other words, publishers just need to determine a price for content being taken off their hands, and if the public is willing to pay that price, it happens. (Users aren’t charged until works they want to unglue are unglued.) No more transaction costs; anyone can distribute the content to anyone else. Publishers could possibly retain subsidiary rights to the content, such as print on demand or derivative work rights.

Bill Rosenblatt of the Copyright and Technology blog looks at the problem publishers have of finding good content creators and having a model that makes that content widely available. Towards the end of his post, he summarizes Eric Hellman’s proposed model for “ungluing ebooks” in a way that makes sense for creators, publishers, and consumers. So far as I know, no one has taken Eric up on a trial of his model, but I think it would be interesting to see if it was practical. [Found via OCLC Research's Above the Fold.]

University of Pittsburgh Library System Offers Free E-Journal Publishing Service

Pitt’s University Library System (ULS) is now offering free e-journal publishing services to help academic journals make their content available to a global audience while eliminating the cost of print production. 

The E-journal Publishing Program—part of ULS’ D-Scribe Digital Publishing Program, which partners with the University of Pittsburgh Press—“is in keeping with the ULS’ commitment to free and immediate access to scholarly information and its mission to support researchers in the production and sharing of knowledge in a rapidly changing publishing industry,” said Rush G. Miller, Hillman University Librarian and director of the ULS. 

The ULS trains a journal’s editorial staff in the use of Open Journal Systems (OJS) software, which channels the flow of scholarly content from initial author submissions through peer review and final online publication and indexing. OJS provides the tools necessary for the layout, design, copy editing, proofreading, and archiving of journal articles. The platform provides a vast set of reading tools to extend the use of scholarly content through RSS feeds and postings to Facebook and Twitter. E-journal articles can be discovered via blogs, databases, search engines, library collections, and other means. 

The University of Pittsburgh announced that it is offering the infrastructure for managing and hosting electronic journals with an at-cost print-on-demand supplement. Since the cost of the digital publishing platform is absorbed by the University of Pittsburgh and since peer review is typically done at no cost, what’s left on the expense side of the balance sheet? Paying the editorial staff? Marketing and advertising the journal? Has the University of Pittsburgh tipped the equation enough to make this model viable?

Open Bibliographic Data: How Should the Ecosystem Work?

In the conversations about openness of bibliographic data, I often find myself in an odd position, vehemently in support of it but almost as vehemently alarmed at the sort of rhetoric that circulates about the ways that data should be shared.

The problem with both the arguments OCLC makes and many of the arguments for openness seem to be predicated on the view that bibliographic data are largely inert, lifeless “records” and that these records are the units that should be distributed and consumed.

Nothing could be further from the truth.

The above quote is just one small piece of a posting by John Wilkin on the Open Knowledge Foundation blog. In it he plants a flag for the library profession to drive towards with bibliographic data that is published in a fine-grained, easily recombined manner. In being too focused on silos of “lifeless records” (WorldCat, local ILSs, Open Library, etc.), he suggests that the profession is missing out on ways we (and our users!) can combine and enhance bibliographic data. John’s statement is in parallel with a growing movement towards linked data, a movement that encompasses a reinvigorating of bibliographic description using FRBR and RDA (the current and progressive best thinking of the library community) with the foundational elements of the “semantic web” vision. For more on the latter, see the work of the W3C-supported Library Linked Data Incubator Group and the work of Karen Coyle and Diane Hillman, among others.

On a related note, the JISC community in the UK has also published the Open Bibliographic Data Guide. “It is about the business cases for Open Bibliographic Data – releasing some or all of a library’s catalogue records for open use and re-use by others.”

Poking, Tagging and Now Landing an M.B.A

But thanks to a pair of young British entrepreneurs, students who do want both a business education and the credential to prove it can now pursue their studies at the same time as they “poke” their friends, tag photos, update their relationship status or harvest their virtual crops on FarmVille.

The London School of Business and Finance Global M.B.A. bills itself as “the world’s first internationally recognized M.B.A. to be delivered through a Facebook application.”

Hmm — meet the students where they are? This story from the New York Times outlines an MBA program that is fully immersed in the Facebook environment. I wonder if the completion rate of a Facebook-based program will be higher than that of other online systems because users spend more time in the Facebook environment. [Via Steven Bell]

How I Planned a Successful Unconference in 6 hours – and You Can Too

Last Friday I ran WhereCamp5280 in Denver, which attracted over 70 people (many from out of state and a couple from Canada), used thousands of dollars from top-tier sponsors and was organized in probably less than six hours total. An unconference is a conference in the loosest of terms. People show up, we build our own agenda and then go for it. Here I’ll describe how it was run.

Steve Coast, a guest author for ReadWriteWeb, give this how-to guide for planning an unconference. An unconference is a relatively new style of event where the content of the meeting is defined by the people who show up and participate. The common guidelines for such meetings¹ are: 1) The people who come are the best people who could have come; 2) Whatever happens is the only thing that could have happened; 3) It starts when it starts; 4) It’s over when it’s over; and 5) Exercise the Law of Two Feet. The last might take some more explanation; it means: “If you are not learning or contributing to a talk or presentation or discussion it is your responsibility to find somewhere where you can contribute or learn.”

In my experience, the unconference format is great if you want a group to brainstorm around a central idea or if you want to promote professional networking connections among a group. If you are looking for a particular outcome or have a specific agenda, this format does not work well.

Footnotes

1. These rules are common, but I found them most clearly expressed at the Scratchpad Wikia.

Did you know that Amazon offers a facility to make corrections to its catalog? Somewhere in the past few months someone mentioned this to me and I tried it out. (Unfortunately, it has been long enough now that I’ve forgotten who told me; if you are the one, please fess up in this post’s comments section. It was Ron Murray from the Library of Congress. Thanks, Ron!) And it works! Is this a model for crowdsourced corrections to library data?

Here is how it looks from a user’s perspective.

Step 1. Finding something to correct

Amazon page for 'How to Talk to Anyone' with typo

Step 2. Making the Correction

Excerpt of Amazon product information page with the 'update product info' link highlighted

Excerpt of Amazon Catalog Update Form

Preview of Amazon Catalog Updates

Submission confirmation page from Amazon Catalog Update service

Step 3. Getting Confirmation from Amazon

From: “gfix-noreply@amazon.com”
To: “peter@OhioLINK.edu”Subject: Your Amazon.com Catalog Update Request

==== This is an automated response message - please do not reply ====

Thank you for using the Catalog Update Form to send suggestions for

How to Talk to Anyone: 62 Little Tricks for Big Sucess in Relationships (ASIN 1593160267)

Your update has been accepted and processed. It will appear online within the next two to three business days.
Attribute: Title
Current value:
How to Talk to Anyone: 62 Little Tricks for Big Sucess in Relationships

Your suggestion:
How to Talk to Anyone: 62 Little Tricks for Big Success in Relationships

Data accuracy is highly important to us. We appreciate the time you have taken to submit your updates to us.

Best regards,

Catalog Department
www.amazon.com

And if you go to this product page now you’ll see the title has been corrected.

Would this Work for Libraries?

If we were to do it, I think it might have to be done by a bibliographic utility like OCLC who has ways to push the updated records to member libraries. Otherwise we run the risk of diluting the corrections across many individual library catalogs. Interestingly, this sort of user-generated correction facility one that the Open Library already provides. (Open Library is a wiki-like service that offers the ability for anyone to make changes to its records, much like how anyone can edit articles on Wikipedia.) So between Amazon and Open Library there is a continuum of workflows of mediated corrections to unmediated corrections for us to consider. This scheme, of course, begs us to consider the notion of distributed version control systems for handling our bibliographic data so that changes can be merged across many sources.

Lots to think about…

This year the ALCTS Forum at ALA Midwinter brought together three perspectives on massaging bibliographic data of various sorts in ways that use MARC, but where MARC is not the end goal. What do you get when you swirl MARC, ONIX, and various other formats of metadata in a big pot? Three projects: ONIX Enrichment at OCLC, the Open Library Project, and Google Book Search metadata.

Below is a summary of how these three projects are messin’ with metadata, as told by the Forum panelists. I also recommend reading Eric Hellman’s Google Exposes Book Metadata Privates at ALA Forum for his recollection and views of the same meeting.

ONIX Enrichment at OCLC

Renee Register, Global Product Manager for OCLC Cataloging and Metadata Services, was the first to present on the panel. Her talk looked at a new and evolving product at OCLC on the enhancement of ONIX records with WorldCat records, and vice versa. ¹

As libraries, Renee said “our instincts are collaborative” but “our data and workflow silos encourage redundancy and inhibit interoperability.” Beyond the obvious differences in metadata formats, the workflows of libraries differ dramatically from other metadata providers and consumers. In libraries (with the exception of CIP and brief on-order records) the major work of bibliographic production is performed at the end of the publication cycle and ends with the receipt of the published item. In the publisher supply chain, bibliographic data evolves over time, usually beginning months before publication and continuing to grow for months and years (sales information, etc.) after publication. Renee had a graphic showing the current flow of metadata around the broader bibliographic universe that highlighted the isolation of library activity relative to publisher, wholesaler, and retailer activity.

Diagram of the Process of Enhancing ONIX Records, from OCLC Services for the Publisher Supply Chain Webinar, August 2009

In her experience, Renee said that libraries need ways to enable our metadata to evolve over time and allow for publisher-created metadata to merge effectively with library-created metadata. The bibliographic record needs to be a “living, growing” thing throughout the lifecycle of a title and beyond. In concluding her remarks, she offered several resources to explore for further information: the OCLC/NISO study on Streamlining Book Metadata Workflow, the U.K. Research Information Network report on Creating Catalogues: Bibliographic Records in a Networked World, the Library of Congress Study of the North American MARC Records Marketplace, the Library of Congress CIP/ONIX Pilot Project, and the OCLC Publisher Supply Chain Website.

From MARC to Wiki with Open Library

Karen said right at the start that the Open Library project is different from most of what happens in libraries — it is “someone outside the library world making use of library data” — although the goal is arguably the same as others — “One web page for every book ever published.” As such, the Open Library isn’t a library catalog as librarians think of it in that it is not a representation of a libraries inventory. It has metadata for every book it can know about and a pointer to places where the book can be found, including all of the electronic books in Internet Archive (Open Content Alliance, Google Public Domain, etc.) as well as pointers back to OCLC WorldCat. Karen’s role for the project is that of “Library Data Informant.” The Internet Archive decided that they needed someone who understood library data in order to try to use it. From Karen’s perspective, she is trying to be a resource for project but not give them any guidance on how to implement the service. She is curious to see what the project would do when bibliographic data is viewed from a non-librarian perspective. If they have questions, or if they have assumptions about data that are wrong, then she intervenes.

Karen went on to briefly describe the Open Library system. Open Library doesn’t have records; rather, it has field types and data properties. In this way, it uses semantic web concepts. “Author” is a type, “Author birthdate” is another type, and so forth. There are no set field types, so if the project gets data from source for which a type doesn’t yet exist, it can create a new one. Each type can have data properties such as string, boolean, text, link, etc. Nothing is required and everything is repeatable. Everything — types, properties, and values — gets a URI (a URI is an identifier like a URL, but conceptually a superset of the universe of URLs). Titles, authors, subjects, author birthdates, and so on have URIs. Lastly, the underlying data structures are based on wiki principles: all edits are saved and viewable, anyone can edit any value, anyone can add new types or properties, anyone can develop their own displays, etc.

The data that is now in Open Library came from a variety of sources. They started with a copy of books from the Library of Congress, and continue to receive the weekly updates. They performed a crawl of Amazon’s book data. They have gotten some from publishers, libraries, and individual users. The last is perhaps the most interesting because it is mainly people outside the western world who are otherwise having trouble getting their works recognized.

Problems, Issues, Challenges, and Opportunities with the Data

Names -

“These library forms of names? Honestly no one but us can stand them.” Even something as simple as the form of last-name-comma-first-name is troublesome. No one else uses this form of the name: Amazon, Wikipedia, etc. In processing these, any information between parenthesis has been deleted, birth and death dates move into separate field types.

Titles -

In working with the Open Library developers, this is one place that Karen tried insisting on applying a library practice: knowing the initial article. For us, this is important for sorting books in alphabetical order. The developer response — why do we have to sort in alphabetical order? “Where else but library catalogs to we see things sorted in alphabetical order? Not in Google, not in Amazon, not anywhere. Alphabetical order is not in the mindset anymore.” They also found that the title might include extraneous data. Amazon, for instance, appends the series title in parenthesis to the main title. This is a demonstration of how other communities are not as concerned about strongly typing and separating information into fields. Amazon, of course, has reasons for series information into the main title: it helps sell books.

Product dimensions -

Publishers and distributors need to know characteristics of an item such as height, width, depth, and weight; they, of course, need to put it in a box and ship it. Libraries, concerned about placing the item on the shelf, record just height. Recording pagination is different, too: libraries use odd notations “ill. (some col)” and “xv, 200p.” versus simply “200 pages.”

Birthdates -

Librarians use birthdates to distinguish names; if there is no need to distinguish a name, birth and death dates are not added. Someone looking at this from the outside would ask ‘Why don’t all authors have birth and death dates?’ This can be useful information for viewing the context of an item, not just to distinguish author names. Open Library ran author names against Wikipedia to pick up not only birth and death years, but also the actual dates.

Subject headings -

Open Library using Library of Congress Subject Headings was out of the question. In processing the data, the Open Library developers just broke them apart into segments and used them. But because they were able to do data mining on the subject field types, they did find statistical relationships between the disassembled precoordinated headings and were able to present those to the user.

The View of the Data -

Rather than a traditional library view of long lists of author-title, the Open Library (in its next version coming in February) will have several different views into the mass of data: Authors; Books (what we would call FRBR ‘manifestations’); Works; Subjects; and eventually places, publishers, etc. For example, when searching for an author one would get the author page. On it would be all of the works from the author as well as other biographical information. It looks similar to a WorldCat identities page, except it is the actual user interface built into the system. Similarly, every work will have a page, and at the bottom of it one will see all of the editions of the work. Also, each subject will have a page, and one will see a list of works with that subject as well as authors who write on that subject. As Karen said, “The subject itself becomes an object of interest in the database, not just something that is just tacked on to the bottom of the library record.”

Data mining -

With the data in this format, it is possible to perform data mining actions against it. For instance, simple data mining such as country of publication, popular places that appear, etc. When they had the problem of author names — knowing when to reverse surname and forname — they ran the names against Amazon and Wikipedia and retained the ones where they found the order of the entry was the same. The Open Library developers are also experimenting with data mining to find publisher names. Publisher names, of course, vary dramatically, but by using ISBN prefixes they can pull together related items into a “publisher” view.

Karen suggested watching the Edward Betts’s site, one of the developers of the Open Library project with an eye on the data mining aspects. She said it is fun to look at our data when it can be viewed from this different point-of-view. She also said to watch out for a new version of the Open Library website coming in February.

Google Book Search Metadata

On the input side, Google is getting bibliographic metadata from over 100 sources in a variety of formats. MARC records are coming from libraries, union catalogs, commercial providers (OCLC), publishers/retails (one publisher supplies records in MARC format). Google also gets ONIX records from commercial providers (such as Ingram and Bowker), publishers, and retailers. Google is especially interested in data from non-U.S. retailers because it is a source of information about books published outside the United States; it helps facilitate discovery of items that they may not otherwise encounter in the publisher and library programs. Google also receives records in a variety of “idiosyncratic formats” — for example, publisher-contributed metadata (via the Publisher Partner Program); information associating books with jacket images; name authority records (from LC); reviews; popularity signals (sales data as well as anonymized circulation data from some library partners, useful for feeding into the relevancy ranking algorithm); and internally-generated metadata (for instance, whether a book is commercially available or not). Google processes all of this information to come up with a single record that describes a book. At this point they have over 800 million bibliographic records and one trillion bits of information in those records.

All of these records from all of these sources are processed and remixed with Google’s parsing algorithms about twice a week. The first step is to transform the incoming records into a “less verbose format” for storage and processing. It is a SQL-like structure that allows elements of the metadata to be queried. Records are then parsed to extract specific bits of information, transform the bits as necessary, and write the information to an internal “resolved records” data structure (a subset of the data coming from the input formats). In the presentation, Kurt had examples of how making inferences from data coming from both MARC and ONIX can be troublesome. Parsing also involves extracting “bibkeys” from the records to aid in matching across sources of data. Four types of identifiers are extracted from bibliographic records: OCLC numbers, LCCNs, ISBNs, and ISSNs. They provide usually useful signals when matching bibliographic and help with assertions that two records describe the same manifestation. Google also tries to parse item data when present in records representing multi-volume works, enumeration and chronology. They will also treat barcode as a form of a “bibkey” if they get it from a library. The parsing algorithm will also split records containing multiple ISBNs representing different product forms (e.g. hardback, paperback, etc.).

With all of this data parsed into records, Google starts its clustering process where records are examined and attached to each other. Bibkeys provide significant evidence for relating records to each other, but bibkeys are not always present in a record (non-U.S. records and older records frequently contain no bibkeys). The algorithms then fall back on text similarity matching using title, subtitle, contributor and other fields such as publisher and publication year. The results are clusters of records representing the same manifestation. An algorithm then attempts to derive the “best-of” record for a single cluster from all of the parsed input records. This is done in a field-by-field voting process based on the trustworthiness of individual fields from record sources.

Kurt went into some of the challenges facing the team building the clustering and best-of record creation algorithms. For instance, in dealing with multivolume works they know of 5 numbering schemas with 3 number types in 15 different languages. Enumeration is now showing in the public display, but the development team is still working with unparsable item data due to inconsistent cataloging practices between institutions…and sometimes inconsistencies within an institution. Another problem is non-unique identifiers. In the current data set ISBN 7899964709 is shared by 75 books and ISBN 7533305353 is associated with 1413 books. There are also poor quality or “junk records”. Kurt said his favorite was “The Mosaic Navigator” by Sigmund Freud published in 1939. These are hard to identify with an algorithm, and they rely on reports of problems that enable the developers to go in and “kill” the troublesome record. Another example is a book by Virginia Woolf where the incoming record had conflicting information; it had two 260 fields that contained different dates (1961, correct, and 1900) with fixed field information that strongly suggested that 1900 was the single date of publication. When the data problem is systematic, they can identify it and compensate for it. Kurt’s example for this case was “The United States Since 1945″ published in 1899. This one was highlighted in Geoffrey Nunberg’s criticism of Google Books metadata. In this case, there was a source of metadata from Brazil that when they didn’t know the date of publication would use 1899. When Google went back and looked at the date distribution of books there was a huge spike in 1899. Once Google knew about it they were able to go in and kill that information from that source of records. ³

In closing, Kurt said that Google is committed to engaging with the library community on improving metadata and metadata processing.

The text was modified to update a link from http://www.niso.org/publications/white_papers/Stream lineBookMetadataWorkflowWhitePaper.pdf to http://www.niso.org/publications/white_papers/StreamlineBookMetadataWorkflowWhitePaper.pdf on January 19th, 2011.

The text was modified to remove a link to http://www.oclc.org/speakers/bios/register_renee.htm on February 11th, 2011.

Footnotes

1. For those not familiar with ONIX, it is a suite of standards promulgated by EDItEUR for the interchange of information on books and serial publications. It is primarily used as the communication channel between the publishing industry through distribution chains to retail establishments.
2. By the way, it seems like BISAC is an acronym for “Book Industry Systems Advisory Committee”, the former name of the Book Industry Study Group.
3. A side note: Google isn’t the only one tripped up by this. If one searches for the ISBN of the item, 0195038487, you get to more than one site that has the same incorrect publication date. At least Google is attempting to clean up the data!

At ALA Midwinter, ALCTS sponsored a panel discussion about sharing library-created data inside and outside the library community, with a particular focus on cataloging data. I was honored to be ask to speak on the topic from the perspective of a consortial office. This is the second and final post in a series that represents an approximation of what I said on the panel.

The first part examined the nature of surrogate records that we create as a means to get users to content. The post looked at where we get records, how humans and machines can create them, and the rights associated with component data that makes up the records.

Right to reuse records without restrictions

The first area where is important is with discovery layers. A new generation of discovery layers are taking surrogates from a variety of sources – catalogs, publishers, index/abstract services, etc. – and performing actions such as consolidating records and building relationships between surrogates. These derived surrogates are presented to users in new interfaces or new portals into existing interfaces. Examples of these systems are the Extensible Catalog project at the University of Rochester and the newly announced Serials Solutions Summon product. OhioLINK recently solicited responses from vendors where this kind of capability is a key product of a new discovery layer. Other projects (such as subject-specific portals) also seek to re-purpose the data – mix it up with other sources of data to create new uses and views that are specific to a particular user community. Anything other than a permissive-for-all-by-default will put up roadblocks and cause builders of these systems to seek data from other services.

In addition to presenting the surrogates to users in new ways, libraries are also investigating new forms of workflow and collaborative activities surrounding the creation and maintenance of bibliographic records. One of the strong desires of many involved in the OLE Project is cooperative purchasing and cooperative technical services. OhioLINK has also recently issued an RFI seeking new options for highly collaborative workflows in the maintenance of surrogate records. Old models of charging for use of records can hinder the ability of cooperating institutions to optimize costs and efforts of back-room library options.

The elephant in the room is the recently proposed OCLC Records Use Policy. Setting aside the debatable legal framework under which OCLC asserted the right to set a usage policy on records from the cooperative, there were clauses in the proposed policy that jeopardize the usability of records, and as a consequence the viability of the cooperative as a whole. Actions that restrict use of data or create uncertainty around the use of data lessen the value of that data. I think few would argue that value can be created by aggregating services on top of the data; the activities in the WorldCat Grid and Developer’s Network point to that. Revenue could be generated in fees charged to non-cooperative members. It is conceptually important to separate the hosting of the surrogates from the layered services on top of them – WorldCat Local, mediated ILL, collection analysis, by way of example.

Parting thoughts

Post-panel Thoughts

Thanks to Charles Wilt, Executive Director of ALCTS, for inviting me to speak at the ALCTS Forum and to Karen Calhoun for facilitating the invitation. My appreciation also goes out to my co-panelists: Karen Calhoun (who has posted her slides online), John Mark Ockerbloom (who also posted his slides and approximate speech transcript), and Brian Schottlaender (who eloquently summarized statements from the other panelists and took point in fielding questions from the audience).

The text was modified to remove a link to http://lisnews.org/node/add/blog on January 13th, 2011.

At ALA Midwinter, ALCTS sponsored a panel discussion about sharing library-created data inside and outside the library community, with a particular focus on cataloging data. I was honored to be ask to speak on the topic from the perspective of a consortial office. This is the first in a series of posts that represents an approximation of what I said on the panel. (Also be sure to read the summary of the session by Norman Oder in Library Journal.)

I think it is important to step back and reflect on the nature of what we are talking about. We build bibliographic records as surrogates for the desired object, meaning that the surrogate is a means to an end – retrieving the described object – and not an end onto itself. We build indexes of these surrogates for patrons to use to discover information. All other factors held constant, the better the surrogate, the greater the chance the user will find the information they are seeking. The following discussion looks at the sources of records, the way they are built, and what it means to try to share them.

Sources of Records

The humans creating these surrogate records are typically called “catalogers” although I’m coming to prefer the term descriptionists as a more accurate portrayal of their activity (if, for no other reason, that the description activity engaged in by these professionals extends beyond what we traditionally consider “the catalog”). Using the tools of taxonomies and ontologies, the descriptionist creates the surrogate of the item to put into the catalog systems.

There is another way emerging, however, to create these surrogates: through computer algorithmic computation against the object itself. In his book Everything is Miscellaneous, David Weinberger talks about the fungible nature of metadata and data: “metadata” is what we know and “data” is what we want to find out. In a talk given at Google in May 2007, he gave an example of using something known — like a quote from a book — to find out something — like the author of that book. (Skip to 29 minutes and 25 seconds into the video.) The “metadata” (the quote) was used to find the “data” (the author) that was being sought.

This is Google’s big experiment in computational analytics. It is relying on an analysis of the text of the item itself to be the descriptive surrogate. It employs algorithms that look at every word — perhaps even every concept — in an item and weights it relative to those words and terms in other items in the corpus.

Amazon has done something similar for years with its “Key Phrases” techniques. For Amazon, Capitalized Phrases are people, places, events, or terms mentioned in a book. Statistically Improbable Phrases are the most distinctive phrases in a book as compared against the corpus of texts in its catalog. These become a form of index points – the surrogates – for finding the item.

Of human-generated or machine-generated, which of these is better – a measure of effectiveness in precision and recall as well as an assessment of the relative cost to create – is undoubtedly a topic of research and debate. For those of us more familiar in the ways of the descriptionists, however, it is undoubtedly time to become familiar with the ways of the computationalists to understand the strengths of each.

Ownership of records

Ownership of the data in records is even cloudier than what is outlined above. By my estimation, we are entering a world in which there are four types of attributes that make up our bibliographic records. The first is the recitation of the facts, whether copied from the item-in-hand or obtained as a feed of information from publishers. The second is the creative work of the descriptionist in adding value to the surrogate in the form of classification numbers, subject headings, abstracts, and the like. The third is the additions that the computationalists – or, more specifically, their algorithms – bring to the object’s description in the surrogate. This can take the form of the previous discussed Google Book Search algorithms as well as Amazon’s Capitalized Phrases and Statistically Improbable Phrases. It also includes actions specific to our traditional domain of human-generated surrogates; for instance, of the algorithms OCLC runs across the Worldcat dataset to merge records and improve records. And finally, thinking beyond the boundaries typical of AACR2 and MARC, there is a fourth type: user-contributed information. The creative efforts of users to add tags, reviews, summaries, and commentary add to the surrogate, and with that we kick wide open a door to the problem of who “owns” these surrogate records.

The issue of who “owns” the user-contributed information to records is handled in a wide variety of ways. In LibraryThing, for instance, the creator retains copyright over tags, reviews, summaries, and comments, and grants LibraryThing non-exclusive rights to make use of that creative effort. LibraryThing also has a function where users can add “Common Knowledge” (character names and occupations, locations, honors/awards, and quotations, among other details) about the item. Users add Common Knowledge augmentations with a Creative Commons “By, Share-Alike” license. The Open Library project asserts no rights over the data in its system; it declares the material in the open library database to be facts, and therefore in the public domain (at least under U.S. copyright law). The newly announced ‡biblios.net repository of records uses the Open Data Commons Public Domain Dedication and License, developed by Talis in the UK and Creative Commons. It, too, asserts that the subject data is in the public domain and offers suggested community norms surrounding the use of the data.

Ownership of records is also a continuing source of discussion among members of the OCLC cooperative. If a brief record from a publisher is added into the WorldCat database and is subsequently enhanced by one or more members, who “owns” the record. If a library systematically adds enhancements to records that matter to its own community – enhancements such as paper and binding types – who “owns” those records?

One way to handle this variety of who (if anyone) owns portions of the surrogate is to split records based on who contributed what. This would enable consuming systems to make decisions on what data to include based on the conditions the owners might put on their creative acts. (Keep in mind that legal precedent would seem to indicate that the facts expressed in the surrogate couldn’t be protected by copyright restrictions.) But our existing record structures do not give us a way to do this, nor does there seem to be any work going on to enable this to happen. Perhaps we are all just burying our collective heads in the sand and hoping it will go away.

More to come…

The text was modified to update a link from https://biblios.net/ to http://biblios.net/ on February 11th, 2011.

The text was modified to update a link from https://biblios.net/pddl to http://biblios.net/pddl on February 11th, 2011.

Earlier this week, Aaron Swartz of the Internet Archive announced the demonstration website of the Open Library project, a new kind of book catalog that brings together traditional publisher and library bibliographic data in an interface with the user-contributed paradigm of Wikipedia. Okay, I’ll pause for a moment while you parse that last sentence. Think you got it? Read — and watch — further.

Open Library has been mentioned a bit in the blogs this week, but not to the extent I thought was worthy of the magnitude of the project. So I recorded a screencast introduction (in Flash Video format below followed by a rough transcript) that looks at not only the browsing side of the system but also the record editing and record creation aspects of Open Library. As I say at the end of the recording, Open Library is one of those mind-bending, assumption-shattering projects that, at least for me, is challenging my thoughts about what library service could be and should be. Congratulations to the team at the Internet Archive, and I’m looking forward to future enhancements and directions for the project.

Rough transcript of the screen cast is below.

Introduction

Hello, and welcome to this screencast overview of the Open Library project. Open Library is an effort by the Internet Archive to create a comprehensive catalog of every book. As the project’s “about” page says, “Not every book on sale, or every important book, or even every book in English, but simply every book.” The about page goes on to describe the characteristics of Open Library project — that it is a project enabled by Internet technology because no physical space could hold it and that it aims to pull together records from publishers and libraries. It is also a project in the same vein as Wikipedia, meaning that any user can create and edit the records in the system.

In this overview, I’ll lead you through searching and browsing the Open Library’s demonstration website from the perspective of any modern library catalog interface. Then I’ll show you where it deviates from traditional library catalogs by exposing the underlying wiki nature of the database; we’ll examine the changes that users have made and we’ll even make a change ourselves. And finally I’ll show the process of creating entirely new records in the system. So let’s get started.

Searching

We’re looking at the home page of the Open Library project demonstration site. In the middle is a search box with a suggested search — “tom sawyer adventure”. That is a good suggestion so we’ll click on Go. Open Library returns a classic, relevance ranked list of matching records with some book covers along the left side and a faceted list of refinements along the right. So right away you can see that there are some authority control problems here in the author names — Twain comma Mark, Mark comma Twain, and Twain comma Mark with birth and death dates — and here in the language field. But I have high hopes that the developer team will find some intriguing ways to address these problems.

Back over here in the results area we have the various editions of Samuel Clemen’s “The Adventures of Tom Sawyer” — let’s pick the 1876 edition to see the full record display — there. We have the publisher, publication date and place, language, and a summary or review of sorts at the bottom. We also see signs of the availability of full text — over here in the options box there is a download from the Internet Archive link, a “Scan Sponsor” field here and a “View this book” graphic. This is one of the items scanned by the Open Content Alliance and made available by the Internet Archive through the Open Library project. A very nice interface for paging through the book. So one could imagine that the Open Library could become the primary vehicle by which Open Content Alliance materials are made available to the public.

So let’s go back here to the metadata page. Remember in the introduction that I said that the data was malleable in a wiki-like fashion. The Open Library developers created a system that allows for user-contributed updates (a la Wikipedia) to fielded data (like your classic bibliographic record). The two hints that the record is modifiable are this big edit button in the middle of the metadata and this more subtile “[history]” link near the top of the page. Let’s start with the history link to see what has been done to this record.

This page should look familiar to those who have worked with wikis before. It shows a listing of edits that were made to this record from most recent to the very first edit, who made the change (identified by IP addresses in this case because the people making the changes were not logged to an account at the time), an editor-supplied comment about what was done, and when the change was made. We can go back in time and see the page at a particular version through the links under the “When” column, or we can use the compare function to see the difference between two version. In the case of the changes between version 2 and version 3, we see that the editor added “Canada” as the place of publication. On this page you start to see the fielded nature of this wiki structure, but the best place to see it is look at the record edit screen itself.

These are all full-text fields on this page with no controlled vocabulary. You’ll note the absence of any MARC field names here, but as you scroll through you’ll see the evidence of MARC and AACR2 in the field labels. Down at the bottom is an edit summary to describe the changes made to the record, then save, preview and delete version buttons — all classic wiki functions.

Editing

Now, I’d like to show the full record editing process, but since I don’t have this Mark Twain book in hand, I’m going to bring up another record that I created yesterday — “Eric Meyer on CSS“. Before showing the editing process, let’s linger here a moment at the “options” box along the right side. Since this is a more modern book (as opposed to the Tom Sawyer book we saw first), there are additional options here for purchasing the book through these various vendors or borrowing the book through a very nice link into Open Worldcat and two web-based book trading sites.

But back to the metadata. There is one error and one omission in this record — perhaps this is a subtile demonstration of problems that creep in with user-generated content. First, the error, is that there is an extra digit in the ISBN-10 field, which is a big problem because the links in the options box use the ISBN as a linking field and at the time of this recording they don’t work. They will work in a moment, though. The second problem is that I forgot to put in the publication date. But hey, no problem, all I need to do is “Edit” this record.

So we are back to the edit screen, and I’m going to scroll down and fix the ISBN-10 field like so, then scroll down a little further and add the publication date. Then I’ll scroll all the way to the bottom and type in an edit summary — “Fixed the ISBN and added a publication date” — and hit save. We’re now back at the metadata display screen and the link to Open Worldcat now works. So, as an aside, one wonders what the folks in Dublin, Ohio, think about this. It is competition on the one hand since Worldcat is also aiming to be the most comprehensive catalog of books in the world. On the other hand, perhaps there is room for cooperation by somewhat getting vetted changes to Open Library records into the OCLC union catalog. Who knows?

Creating a New Record

Alright, back to current reality. Let’s add a record to Open Library, and in this case I’m going to use an ARL SPEC Kit that I wrote a number of years ago called “Library Patron Privacy”. First let’s run a search in Open Library to see if it is there, and no, it isn’t. The only way I’ve figured out how to enter a new item is to go to the URL where the page would be located and get the classic wiki “This page does not exist. Create it?” message.

One of the quirks I found in the system is that I have to create author wiki pages before book wiki pages — otherwise I’ll get a Python error message on the screen. I’ve reported this to the Open Library developers, but in the meantime just know authors need to be created before their books. Which is to say that authors have wiki pages in Open Library in addition to books. The structure of URLs to Open Library author pages is the letter “a” followed by a slash followed by the author’s last, first and middle names separated by underscore characters. So I’ll go to the URL of that form, then click on the “Create it” link.

Now here is one of the tricky parts of the existing interface. The page type starts as “type/page”, and as you can see it doesn’t have any of the fielded elements that we saw in previous examples. What you have do do is change the page type to “type/author” and then you get the fielded HTML form. So I’m going to go through here and fill in some of the parts. Then go down to the edit summary field and write a summary of this change, then click save. Now that Open Library knows who I am, let’s create the record for the book.

You’ve seen the structure of the URLs to book pages before — a “b” followed by a slash followed by the book title with spaces replaced by underscore characters. I’ll put that in the URL field and get the default page type. This needs to be changed to “type/edition” in order to get the bibliographic record fields. There. Now I’ll go through here and enter the data. When we get down to the author field we enter it in the same format that we used to create it — an “a” followed by a slash followed by the name with spaces replaced by underscores.

So we’ll just finish up here and come down to the edit summary field, put something in here, and hit save. This record is now in the system, and you can see the public display here along with the links on the right because I entered an ISBN. I haven’t quite figured out how to get a cover image into the system yet — I expect there is a file upload interface somewhere, but I haven’t found it.

Conclusions

So that’s all there is, and I don’t say that in a way to denigrate the work that has been done by the development team so far. As the URL and site banner indicate, it is a demonstration system — and a compelling demonstration it is. All sorts of questions immediately come to mind, of course — will there be a controlled vocabulary or authority control built into the system, can data be exported out of records — and, for that matter, can end-users bulk import data into the system, are there Web2.0 niceties like tagging and RSS feeds in the works, and so forth.

Even with all of those questions, Open Library is one of those mind-bending, assumption-shattering projects that, at least for me, is challenging my thoughts about what library service could be and should be. Congratulations to the team at the Internet Archive, and I’m looking forward to future enhancements and directions for the project.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer?m=history on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/adventures_of_Tom_Sawyer?b=3&a=2&m=diff on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/Eric_Meyer_on_CSS on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/a/Murray_Peter_E on January 19th, 2011.

The text was modified to remove a link to http://demo.openlibrary.org/b/Library_Patron_Privacy on January 19th, 2011.