One of the very relevant aspects of the Google Book Search Settlement Agreement to libraries is the provision that allows for free public access to the full text of books in public and academic libraries. The Notice of Settlement summary says: “Google will provide, on request, ‘Public Access’ licenses for free through a dedicated computer terminal at each public library building and through an agreed number of dedicated computer terminals at non-profit higher educational institutions located in the United States.” (Notice; Q9(F)(1)(c); p. 18) The details beyond the summary are quite a bit more interesting and, of course, have tidbits of useful information that isn’t in the summary.

Two notes before we get started. First, for readers who don’t normally follow DLTJ, this exploration of the Settlement is taken from a decidedly library (and library technologist) point of view. There are lots of bits that will probably be of greater interest to authors and publishers (like formula for determining how much compensation is due, etc.) that are not covered here. Second, this is a review of the document as it was submitted to the U.S. District Court for the Southern District of New York in the case of The Author’s Guild et al v. Google Inc. The court needs to review and approve it, and details may change between now and then. There are lots of holes in the document — notably dates and URLs — that will need to filled in.

In the summary below, references to the Proposed Class Action Settlement take the form of the word “Settlement” followed by the question number and possibly the paragraph number plus the page number. For instance: “(Settlement: 3.1, p. 21)”. Likewise, references to the summary Notice of Proposed Class Action Settlement use the word “Notice” in the citation.

The Most Interesting Points

• Each Public Library¹ can have one access point per building.
• Higher Education Institutions² are divided into two tiers, based on the Carnegie Classifications of Institutions of Higher Education. Institutions that qualify as “Associate’s Colleges” can get one access point per 4,000 FTE students; everyone else gets one access point per 10,000 FTE students.
• For higher education institutions, the definition includes the phrase “computer terminal may change from time to time”, but details about how to specify and/or change the access point are not included in the Settlement document.
• Printing can be enabled for public access service points for a per-page fee, but the institution must have a way to collect that per-page fee and remit it to Google for forwarding to the Book Rights Registry (which doesn’t yet exist) to be split among the rightsholders. The per-page fee will be set by the Registry, and there doesn’t appear to be a need to report what was printed by users. How Google will actually collect this money is also not specified in the Settlement.
• Google and the Registry may agree to make this public access service available to one or more public libraries or not-for-profit higher education institutions. This access can be free or for an annual fee; it isn’t clear how this would be different from an institutional subscription. (I’ll cover institutional subscriptions in a later DLTJ post.)

Since the per-page charge requires action by the Book Rights Registry, we probably shouldn’t expect the Public Access provision to be executed before the Registry is formed and its new board of directors has met. Formation of the Registry must be done by the “Effective Date” of the agreement, and the effective date of the agreement is after all parties have agreed to the settlement, the court dismisses the relevant actions against Google, and the time for filing appeals to the courts decision has passed. The date for the final approval hearing by the court has not been set yet.

There sure is a lot left undefined in this section about how such a program would work. How does an institution register? Who at an institution gets to decide which computer is used? A “computer terminal” (what I called an “access point”) presumably means a single IP address. One also presumes that the single IP address can’t be a proxy server of sorts that is enabling more than one user to have access at a time, but this isn’t explicitly stated. (And, believe me, there is a lot of stuff that is explicitly stated in the agreement.)

There is also some interesting mechanics. Are institutions going to be able to detect and charge more for prints from the public access Google Books station? Presumably institutions that have systems to charge for printing do so to recover the cost of consumables. Will such a system be able to charge for consumables and, under certain circumstances, charge more for the content itself?

Also tucked in this section is a bit about the possibility of a “Commercial Public Access Service” for “copy shops and other entities” (Settlement; 4.8(b); p. 61). Under the terms of the Settlement, the Registry and Google may choose to make this available for an annual fee based on concurrent users plus a fee per printed page. The notion of “print on demand” immediately leaps to mind, but that would seem to be covered on a special paragraph for print-on-demand (POD) under the heading of “New Revenue Models” (Settlement; 4.7(a); p. 59).

What the Settlement Says About “Public Access Service”

1. Public Access Service.
   1. Free Public Access Service. Google may provide the Public Access Service to each not-for-profit Higher Education Institution and Public Library that so requests at no charge (and without any payment to the Rightsholders, through the Registry or otherwise (other than as set forth in Section 4.8(a)(ii) (Printing)) as follows:
      1. in the case of not-for-profit Higher Education Institutions that do not qualify as Associate’s Colleges pursuant to the Carnegie Classification of Institutions of Higher Education, one computer terminal for every ten thousand (10,000) Full-Time Equivalency (i.e., full-time equivalent students) at each such institution (which computer terminal may change from time to time);
      2. in the case of not-for-profit Higher Education Institutions that qualify as Associate’s Colleges pursuant to the Carnegie Classification of Institutions of Higher Education, one computer terminal for every four thousand (4,000) Full-Time Equivalency (i.e., full-time equivalent students) at each such institution (which computer terminal may change from time to time); and
      3. in the case of each Public Library, no more than one terminal per Library Building.
   2. Printing. Google shall design the Public Access Service to enable users at a not-for-profit Higher Education Institution to print pages from Display Books for a per-page fee, and to enable users at a Public Library to print pages from Display Books for a per-page fee to the extent that such Public Library offers per-page printing services for a fee for other products and services. The Registry shall set a reasonable fee for such printing. Google shall collect all such printing fees and shall pay them to the Registry in accordance with the Standard Revenue Split for Purchases.
   3. Additional Public Access Service. The Registry and Google may agree that Google may make available the Public Access Service to one or more Public Libraries or not-for-profit Higher Education Institutions either for free or for an annual fee, in addition to the Public Access Service provided under Section 4.8(a)(i) (Free Public Access Service).
2. Commercial Public Access Service. The Registry and Google may agree to make a commercial public access service available to copy shops and other entities for an annual fee per concurrent user and a fee per printed page.

Footnotes

1. ‘Public Library’ means a library that (a) is accessible by the public, (b) is, or is part of, a not-for-profit or government-funded institution other than a not-for-profit or government-funded institution that is classified under the Carnegie Classification of Institutions of Higher Education, and (c) allows patrons to take books and other materials off the premises but may also have non-circulating reference collections or provide other library services; however, ‘Public Library’ does not include any library primarily funded or managed by the federal government or an agency thereof. (Settlement; 1.119; p. 15)
2. ‘Higher Education Institution’ means an Institution of Higher Education, as defined by the Carnegie Classifications of Institutions of Higher Education from time to time or, if and when the Carnegie Foundation for the Classification of Teaching is no longer classifying colleges and universities in the United States, as such term or its successor term is defined by any successor classification system used to classify colleges and universities in the United States. (Settlement; 1.66; p. 9)

OhioLINK’s Meg Spernoga pointed our staff to a 30 minute audio documentary called Who Needs Libraries? from Soundprint.org:

As more and more information is available on-line, as Amazon rolls out new software that allows anyone to find any passage in any book, an important question becomes: Who needs libraries anymore? Why does anyone need four walls filled with paper between covers? Surprisingly, they still do and in this program Producer Richard Paul explores why; looking at how university libraries, school libraries and public libraries have adapted to the new information world. This program airs as part of our ongoing series on education and technology, and is funded in part by the U.S. Department of Education.

Produced by Richard Paul. Hosted by Lisa Simeone.

Some of the topics covered:

• Numbers of New/Renovated public libraries are steady
• Use of consortial depositories by academic libraries
• Licensed content that can’t be found in Google (pros — immediate access; and cons — preservation)
• Widespread digitization of content for online access (pros and cons)
• Impact of Gates Foundation money on public library services
• Changing ways libraries are being used

Thanks, Meg!

This is a continuation of the investigation about integrating the California Digital Library’s XTF software into the FEDORA digital object repository that started earlier. This analysis looks at the textIndexer module in particular, starting with an overview of how textIndexer works now with filesystem-based objects and ending with an outline of how this could with reading objects from a FEDORA repository instead.

XTF’s Native File System handler

Natively, XTF wants to read content out of the file system. The core of the processing is done in these two class files:

TextIndexer.java

The main() driver for ingesting content into the index. It reads commandline arguments (cfgInfo.readCmdLine( args, startArg );) to determine the various parameters, one of which is the top of the document source directory (String srcRootDir = Path.resolveRelOrAbs( xtfHomeFile, cfgInfo.indexInfo.sourcePath );). Assuming all goes well, it calls a method to open the Lucene index for writing, process files in the source directory, and close the Lucene index:
[java]
srcTreeProcessor.open( cfgInfo );
srcTreeProcessor.processDir( new File(srcRootDir), 0 );
srcTreeProcessor.close();
[/java]

SrcTreeProcessor.java

processDir() is called recursively on the directory structure to process files in that directory. For each directory, a docBuf XML-as-a-string buffer is consisting of an element for every directory entry. docBuf is fed into the SAXON processor along with the docSelector XSLT stylesheet. The resulting XML is read node-by-node looking for file entries that have an “indexFile” tag. For each matching node, it calls processFile() to index each entry.

processFile() will run the prefilter XSLT against the file content, build the Lazy Tree (if possible and requested), create the IndexSource version by running the source document through the appropriate file type “*IndexSource” method (e.g. PDFIndexSource(), XMLIndexSource, and MARCIndexSource()) and queue the content for indexing by the Lucene indexer.

Requirements for an Object Handler for textIndexer

1. Parse command line and configuration file parameters to create an IndexerConfig instance (guiding parameters for the indexer as a whole) and an IndexInfo instance (parameters specific to the identified index-name).
2. Specify a collection of objects that you want in index-name.
3. Open up a writable instance of the index-name’s Lucene index (a la srcTreeProcessor.open( cfgInfo );)
4. For each object to be put into index-name, do these things:
   1. Optionally, run the source object through a prefilter (an XSLT transformation used to restructure the source document just prior to indexing without changing the stored source document).
   2. Optionally, remove a DOCTYPE declaration in the source object before it is indexed.
   3. Set up an transformation object from the native file format to something that is XML and call textProcessor.checkAndQueueText() to add it to a queue to be processed.
5. Close index-name’s Lucene index (a la srcTreeProcessor.close();), which should have the side effect of processing the queued text (a la textProcessor.processQueuedTexts();) which will ultimately create the Lazy Tree (if specified) and add the object to the Lucene index.
6. Optionally, compare the collection of objects that you want in index-name with what is actually in index-name before you started, and remove anything that wasn’t in the specified collection.

Considering a FEDORA-based XTF handler

First off, in terms of dealing with “collections of source objects to be indexed” I think it would be best to have this start with one of our “collection aggregation” objects as the root level of a source collection. We’d perform an RDF “isMemberOf” query against the resource index using the FEDORA PID of the aggregation object (and optionally make an “isMemberOf” query recursively against the returned set — as if one was drilling down a file system).

Secondly, to get the XML content to be indexed, each object would have a getXML disseminator (see Thinking about Our FEDORA Disseminators for background) that would render to XTF an XML version of itself. If the source object is an XML-based object, it just returns the XML. If the source object is a PDF or Word document or something that can be rendered into a text-like form, the disseminator would handle that. If the source object is an image or audio clip, the disseminator can return the descriptive XML of the object. The point being, though, by the time the object gets to XTF’s textIndexer, it has already be rendered to XML, so just the XML transformation tool would be needed (as in this snipped from SrcTreeProcessor.java):
[java]
IndexSource srcFile = null;
if( format.equalsIgnoreCase(“XML”) ) {
InputSource finalSrc = new InputSource( systemId );
srcFile = new XMLIndexSource( finalSrc, srcPath, key,
preFilters, displayStyle, lazyStore );
if( removeDoctypeDecl )
((XMLIndexSource)srcFile).removeDoctypeDecl( true );
}
[/java]

Third, a FEDORA-aware driver that replaces TextIndexer.java and SrcTreeProcessor.java. Given a configuration file location and a starting PID, it would gather the objects to be indexed, “open” the Lucene index, run through the snippet of Java above for each object, and “close” the Lucene index.

The quick-and-dirty first implementation would copy the XML source to a directory on the hard drive (directory and subdirectory names would be the PID of the aggregation object containing the collection of objects), and have XTF use that local filesystem copy as the indexed source. Lazy Tree files for each object would also be created and stored locally. This means we have two copies (three, if you count the Lazy Tree) of the object laying around, so eventually I think we’d want to modify XTF to pull content directly from FEDORA using a REST-based URL. Eventually I think we may also want to store the Lazy Tree in something other than the local file system. Could that be another datastream in the FEDORA object?

The text was modified to remove a link to http://texts-stage.cdlib.org/xtf/javadoc/org/cdlib/xtf/textIndexer/IndexerConfig.html on December 31st, 2010.

The text was modified to remove a link to http://texts-stage.cdlib.org/xtf/javadoc/org/cdlib/xtf/textIndexer/IndexInfo.html on December 31st, 2010.

We’re experimenting pretty heavily now with the California Digital Library‘s XTF framework as a front-end to a FEDORA object repository. Initial efforts look promising — thanks go out to Brian Tingle and Kirk Hastings of CDL; Jeff Cousens, Steve DiDomenico, and Bill Parod from Northwestern; and Ross Wayland from UVa for helping us along in the right direction.

XTF into Eclipse How-To

Points of Integration

DynaXML

[java]
public InputSource getInputSource( String sourcePath,
boolean removeDoctypeDecl ) throws IOException
[/java]
…followed shortly by:

[java]
// If it’s non-local, load the URL.
if( sourcePath.startsWith(“http:”) ||
sourcePath.startsWith(“https:”) )
{
return new InputSource( sourcePath );
}
[/java]
where “InputSource” is the entry point into the SAX parser, which will accept a URI as a parameter.

Unfortunately, using DefaultDocLocator in this way negates the use of CDL’s “Lazy Trees” (a binary version of each XML document containing all the original contents of the document, plus an index telling XTF where each element starts and ends). Lazy Trees are a good thing because they speed up parsing of the XML document and the resulting rendering to the user.

When dealing with local files (as opposed to the URL method described above), DefaultDocLocator will build a Lazy Tree in its index directory the first time the XML document is called up. In implementing a FEDORA interface for XTF’s DynaXML, what is required is a mixture of URL (or, in the case of FEDORA, a PID plus API-A call) to get the document and then create/store its lazy tree in the XTF index directory for subsequent retrieval. This does seem pretty straight forward, does it not?

textIndexer

It is the responsibility of the Document Selector XSLT code to output an XML fragment that identifies which of the files in the directory should be indexed. This output XML fragment should take the following form:
[xml]

{format = "FileFormatID"}
{preFilter = "PreFilterPath"}
{displayStyle = "DocumentFormatterPath"}>

[/xml]

Now the trick seems to be to build an alternate Document Selector that will not use filenames but rather URIs to build the index. That’ll be the subject of the next round of investigations.

Comments and observations are welcome!

The text was modified to remove a link to http://drc-dev.ohiolink.edu/wiki/EclipseXTFHowTo on December 31st, 2010.

The text was modified to remove a link to http://xtf.sourceforge.net/WebDocs/HTML/XTF_Under_Hood/XTFUnderHood.html#LazyFiles on December 31st, 2010.

The text was modified to remove a link to http://xtf.sourceforge.net/WebDocs/HTML/XTF_Programming_Guide/XTFProgGuide.html#textIndexer_DocSelector_Prog on December 31st, 2010.

The text was modified to update a link from http://xtf.cvs.sourceforge.net/xtf/xtf/WEB-INF/src/org/cdlib/xtf/dynaXML/DocLocator.java?revision=1.5&view=markup to http://xtf.hg.sourceforge.net/hgweb/xtf/xtf/file/549e4167039e/WEB-INF/src/org/cdlib/xtf/dynaXML/DocLocator.java on January 28th, 2011.

The text was modified to update a link from http://xtf.cvs.sourceforge.net/xtf/xtf/WEB-INF/src/org/cdlib/xtf/dynaXML/DefaultDocLocator.java?revision=1.10&view=markup to http://xtf.hg.sourceforge.net/hgweb/xtf/xtf/file/de7d8a406bef/WEB-INF/src/org/cdlib/xtf/dynaXML/DefaultDocLocator.java on January 28th, 2011.