Disruptive Library Technology Jester » web development

OCLC Introduces an API for Anyone to Access Book Data

Peter Murray — Wed, 16 Dec 2009 20:01:37 +0000

OCLC announced on Monday the availability of a new API for querying the WorldCat database: the WorldCat Basic API. This is a big deal — it opens up part of the metadata in WorldCat to anyone who wants it (for non-commercial purposes). Previously, access to WorldCat data through the developer network was limited to institutions that are members of OCLC. Data returned includes text strings for author names and book titles, ISBN and OCLC identifiers, a link to the HTML display of the record on WorldCat.org, and an optional HTML-formatted citation.

Getting access to the API is pretty easy. One signs up for an API key that plugs into the URL when making a request. The key are limited to 1,000 queries per day. Requests are made to the API using a REST-style URL, and results are returned in RSS or Atom format.

This is incredibly useful for finding out data when what you have is an ISBN. For example, an API request for ISBN 0-8161-1889-2 would look like this:

http://worldcat.org/webservices/catalog/search/worldcat/opensearch?: q=0816118892&; start=1&; count=10&; wskey=omitted&; format=atom&; cformat=mla

…returns a results set with this data:

    xmlns:oclcterms="http://purl.org/oclc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/"     xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">   OCLC Worldcat Search: 0816118892   http://worldcat.org/webservices/catalog/search/worldcat/opensearch?q=0816118892&start=1&count=10&format=atom&wskey=[omitted]   2009-12-16T13:54:55-05:00   Search results for 0816118892 at http://worldcat.org/webservices/catalog   2   1   10           href="http://worldcat.org/webservices/catalog/search/worldcat/opensearch?q=0816118892&start=1&count=10&wskey=[omitted]"      type="text/html"/>        href="http://worldcat.org/webservices/catalog/search/worldcat/opensearch?q=0816118892&start=1&count=10&wskey=[omitted]&format=atom"      type="application/atom+xml"/>        href="http://worldcat.org/webservices/catalog/search/worldcat/opensearch?q=0816118892&start=1&count=10&wskey=[omitted]&format=atom"      type="application/atom+xml"/>        href="http://worldcat.org/webservices/catalog/search/worldcat/opensearch?q=0816118892&start=2&count=10&wskey=[omitted]&format=atom"      type="application/atom+xml"/>        href="http://worldcat.org/webservices/catalog/opensearch.description.xml"/>                        Crawford, Walt.            MARC for library use : understanding integrated USMARC            http://worldcat.org/oclc/18909048      2008-04-03T16:22:32Z
Crawford, Walt.         MARC for Library Use: Understanding Integrated         USMARC. [Professional librarian series]. Boston: G.K. Hall, 1989.
     urn:ISBN:0816118876      urn:ISBN:9780816118878      urn:ISBN:0816118892      urn:ISBN:9780816118892      18909048                        Crawford, Walt.            MARC for library use : understanding integrated USMARC            http://worldcat.org/oclc/473475190      2009-12-11T02:48:57Z      Crawford, Walt.         MARC for Library Use: Understanding Integrated         USMARC. Boston, Mass: G.K. Hall, 1989.      urn:ISBN:0816118876      urn:ISBN:9780816118878      urn:ISBN:0816118892      urn:ISBN:9780816118892      473475190

From that, one can provide information about other editions of the same work and links back to OCLC for more information and to have people use the locate-in-a-library feature of WorldCat.org.

The WorldCat Basic API joins other free WorldCat services. Unlike the WorldCat Basic API, these services embed pre-formatted HTML into the page without having to understand any coding at all: the “list” widget for displaying a list of items already created in WorldCat; the “keyword search” widget for displaying the results of a search in WorldCat; the “search box” widget that lets users enter a search term and be taken to the full-blown WorldCat.org; and embedded search WorldCat links for pre-constructing a search that leads into the full-blown WorldCat.org. All of these are free for non-commercial use, as described by the OCLC Affiliate Services Terms and Conditions.

If I were to have a complaint about the WorldCat Basic API, it would not be about the API itself but rather how it is described on the WorldCat Affiliate Services page. Those that have been following OCLC’s efforts in providing developer API access into WorldCat data know about other services such as the WorldCat Search API, xISBN/xISSN, and the Registry Search. What isn’t clear from any of the pages is “What is the definition of affiliate?” and “Given different tiers of affiliate status, who has access to (how much of) what?” Since OCLC is a cooperative organization, does “Affiliate” mean a member of the cooperative? In a conversation with Roy Tennant on the code4lib IRC channel, he cleared this up this question: in this case “affiliate” means more of the Internet-era definition of the word where a site offers services for other developers to consume. One is still left with the second question, though, as to what services are available at what limits to which categories of consumers. For instance, the page describing the xISBN service lists several tiers: free for any non-commercial use up to 500 requests/day, free (presumably with no daily limit) to all OCLC cataloging members, and available on a subscription baseis for non-commercial use that exceeds 500 requests per day or for commercial use. There a similar tiers described in other services, but there isn’t an acknowledgement of these tiers on the main WorldCat affiliates page or a convenient list of what is possible in what tiers. I hope OCLC will consider revising the description of these services to include these details.

Fronting Tomcat with Apache HTTPD to Remove Ports and Context Paths

Peter Murray — Thu, 20 Sep 2007 02:31:33 +0000

In this How-To guide, I show a combination of software and configuration to clean up URLs by removing the port numbers of the Java servlet engine (Tomcat) and the context path of the application. The goal is to create “cool URLs” that are are short (removing the unnecessary context path) and follow conventions (using the default port “80″ rather than “8080″). OhioLINK also uses a custom access control module — built for Apache HTTPD — which makes the fronting of Apache HTTPD for Tomcat even more desirable.

Requirement

We’re making use of the latest line of development for the Apache HTTPD series: version 2.2.x. The inclusion of mod_proxy_ajp — replacing the custom “mod_jk” with a module that extends the httpd proxy engine — in the latest major release of HTTPD makes our task much easier. This solution also uses HTTPD’s mod_rewrite and an add-on module called mod_proxy_html. No additions or changes are needed to the stock Tomcat installation.

The Plan

There are two overall tasks that we’re going to ask the HTTPD server to do. First, receive the incoming HTTP request and proxy it to the Tomcat servlet engine using the AJP protocol. Second, rewrite the URL paths of the headers and the X/HTML body from the Tomcat servlet engine to eliminate any instances of the context path. In a visual sense, what we are trying to is rewrite the path so it can be processed by Tomcat (the green box) then remove the extraneous parts of the path in the resulting headers and X/HTML (the red box):

Public Request URLs:	http://e.com		/remaining/path?and=params
URLs sent to Tomcat:	http://e.com	:8080	/context_path	/remaining/path?and=params
URLs as output by Tomcat:	http://e.com	:8080	/context_path	/next/page
URLs as seen by browser:	http://e.com		/next/page

The first half of this problem, modifying a request as they come into the Apache HTTPD server, will be handled by a mod_rewrite rule that rewrites the request to something Tomcat can understand then internally redirects it Tomcat via the AJP proxy. (Note that we are not using simply ProxyPass here because we want to send the request through the AJP interface to the Tomcat server, and RewriteRule allows us to do that with a [P] flag at the end of the RewriteRule line.) The second uses a combination of ProxyPassReverse (a part of Apache-supplied mod_proxy extension that adjusts the URL in the Location, Content-Location and URI headers), ProxyPassReverseCookiePath (also a part of the Apache-supplied mod_proxy extension; it rewrites the path string in Set-Cookie headers), and ProxyHTMLURLMap (from mod_proxy_html, a third-party extension that rewrites URLs inside X/HTML documents).

Preparations

The ‘mod_proxy_html’ extension is likely new to your Apache HTTPD installation, so we need to download the source, compile it, and move it into the proper directory. Fortunately, this is rather straight forward:

wget 'http://apache.webthing.com/mod_proxy_html/mod_proxy_html-2.5.2.c'
apxs -c -I/usr/include/libxml2 -i mod_proxy_html-2.5.2.c

Note that we are not using the mod_proxy_html author’s 3.0 version here. In my set-up, the 3.0 version was causing Apache HTTPD to dump core on every request (whether proxied or not), and the prior release works just fine for our purposes. The apxs line will compile, link, and copy the resulting library to the Apache modules directory for us.

The Configuration

This is the contents a ‘tomcat-proxy.conf’ file that is placed in the ‘conf.d’ directory of the Apache HTTPD configuration directory (most likely /etc/httpd/conf.d/tomcat-proxy.conf, although your installation may vary).

#
#  Information about 'mod_proxy_html' can be found at 
#   http://apache.webthing.com/mod_proxy_html/
LoadFile    /usr/lib/libxml2.so
LoadModule  proxy_html_module    modules/mod_proxy_html-2.5.2.so
 
# DON'T TURN ProxyRequests ON!  Bad things will happen
# http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#access
# http://www.akadia.com/services/prevent_abuse_proxy.html
ProxyRequests off
 
# Necessary to have mod_proxy_html do the rewriting
RequestHeader      unset  Accept-Encoding
 
# Rewrite the URLs to proxy ("[P]") into the Tomcat server
RewriteEngine     on
RewriteRule ^(/.*)      ajp://localhost:8009/context_path/$1    [P]
 
# Be prepared to rewrite the HTML/CSS files as they come back
# from Tomcat
SetOutputFilter proxy-html
 
# Rewrite JavaScript and CSS files in addition to HTML files
ProxyHTMLExtended on
 
# Output Strict XHTML (add "Legacy" to the end of the line below
# to output Transitional XHTML)
ProxyHTMLDoctype XHTML 
 
# Rewrite HTTP headers and HTML/CSS links for everything else
ProxyPassReverse /context_path/ /
ProxyPassReverseCookiePath /context_path/ /
ProxyHTMLURLMap /context_path/ /

That’s pretty much all there is to it. You should note that mod_proxy_html, like any HTML scraper, requires modestly well-formed X/HTML. If the markup is bad, the output from mod_proxy_html is likely to be unpredictable.