Two papers were published recently exploring the quality of Google Scholar and Google Books.

Google Scholar

The study shows deficiencies in the coverage and up-to-dateness of the [Google Scholar] index. Furthermore, the study points up which web servers are the most important data providers for this search service and which information sources are highly represented. We can show that there is a relatively large gap in Google Scholar’s coverage of German literature as well as weaknesses in the accessibility of Open Access content. Major commercial academic publishers are currently the main data providers.

We conclude that Google Scholar has some interesting pros (such as citation analysis and free materials) but the service can not be seen as a substitute for the use of special abstracting and indexing databases and library catalogues due to various weaknesses (such as transparency, coverage and up-to-dateness).

The authors performed a “brute force analysis” (their words) of the coverage of Google Scholar by comparing search results by journal title with five journal lists: ISI Arts & Humanities Citation Index, ISI Social Science Citation Index, ISI Science Citation Index, open access journals listed by DOAJ, and journals from the SOLIS database (mainly German-language journals from sociological disciplines). They queried Google Scholar using the “Return articles published in…” limiter on the advanced search screen, downloaded the first 100 records for each title, then parsed and analyzed each of the records. In total, 621,000 records from Google Scholar search results were analyzed.

The authors first determined the coverage of titles in the five journal lists in the Google Scholar database. The authors note surprise at the relative lack of coverage for open access titles listed in the DOAJ. I think this can be explained by the fact that many open access publishers are not using a systematic application to put their content on the internet. Of the 2,804 journals in the DOAJ directory, only 846 are searchable via DOAJ’s own article-level indexing service.² If the journals can’t be easily harvested at the article level, then they Google can’t add them to the Scholar article index.

Based on the semantics provided in each record, the authors divided the results into three categories (referred to in the paper as “document types”): links to complete descriptive records on an external (publisher’s or aggregator’s) site, citation-only records (no full-text and no link to more complete information at an external site), and direct access links to full text. The distribution of results is shown in the table to the right.

The paper also includes an analysis of the various publisher and portal sites that supply information to Google Scholar’s index.

Google Books

I don’t subscribe to that notion at all, but it is perhaps because I’ve been around enough technology and innovation to know that each new service needs to stand on its own. Tristram Shandy is in part an experiment in typography and layout by the author, as Duguid describes in detail in this article, that is unusual and atypical to the extreme, so I think many of the characterizations of the Google Books project, based on this one artifact, are unfair and short-sighted. When you strip away the false dichotomy of innovative-or-inherited-quality, the oddities surrounding the Tristram Shandy artifact, and various unnecessary pot-shots³ Duguid’s analysis does point to some apparent problems with Google’s scheme for digitizing and indexing books. The quality of some of the scans pointed out in the Tristram Shandy artifact and others are sources of concern. Substandard metadata is another:

Not a word is mentioned about multiple volumes or volume number. Indeed, a quick survey of the Google Book Project suggests that Google doesn’t recognize volume numbers. Not only are the different editions (Harvard’s from 1896, Stanford’s from 1904) given exactly the same name, but also the different volumes of this Stanford’s multivolume edition are labeled identically. Consequently, whatever algorithm Google uses to find the book, it is quite likely, as in this case, to offer volume II first.

Reservations aside, it is a good review the some of the problematic outcomes of the Google Books Library Project.

Footnotes

1. Judging from the citation listed on Philipp Mayr’s homepage, the article will appear in an upcoming issue of Online Information Review from Emerald Group Publishing.
2. Numbers from the DOAJ home page, as of 15-Aug-2007.
3. “A quick look at the online catalogue for Stanford’s library shows that the Stanford volume presented as your second choice by Google Books is actually tucked away in the Stanford Auxiliary library along with “infrequently–used” texts.”

Earlier this year the DOAJ began offering a new schema for registered articles that significantly improves the value of OAI-PMH harvested article content. Prior to this addition the only scheme available was Dublin Core, which as a metadata schema for describing article content is woefully inadequate. (Dublin Core, of course, was never designed to handle the complexity of the description of an average article.) The new schema (graphically represented here
— select thumbnail to see a larger version) includes elements for ISSN/eISSN, volume/issue, start/end page numbers, and author affiliation. There is also a <fullTextUrl> element that is a link to the article content itself (not the splash page of the article on the publisher’s site).

Article content using this schema is harvestable through the DOAJ OAI-PMH provider site (for instance, using a ListRecords verb with a doajArticle metadata prefix against the PMH URL). This is, in fact, the same schema journal publishers use to submit article content to the DOAJ article database. With these pieces in place, it is now conceivable to harvest open access journal article content through the DOAJ and add it to a local journal article repository (such as the Electronic Journal Center in the case of OhioLINK).

Thanks go out to the DOAJ folks for making this available!