In this How-To guide, I show a combination of software and configuration to clean up URLs by removing the port numbers of the Java servlet engine (Tomcat) and the context path of the application. The goal is to create “cool URLs” that are are short (removing the unnecessary context path) and follow conventions (using the default port “80″ rather than “8080″). OhioLINK also uses a custom access control module — built for Apache HTTPD — which makes the fronting of Apache HTTPD for Tomcat even more desirable.

Requirement

The Plan

The first half of this problem, modifying a request as they come into the Apache HTTPD server, will be handled by a mod_rewrite rule that rewrites the request to something Tomcat can understand then internally redirects it Tomcat via the AJP proxy. (Note that we are not using simply ProxyPass here because we want to send the request through the AJP interface to the Tomcat server, and RewriteRule allows us to do that with a [P] flag at the end of the RewriteRule line.) The second uses a combination of ProxyPassReverse (a part of Apache-supplied mod_proxy extension that adjusts the URL in the Location, Content-Location and URI headers), ProxyPassReverseCookiePath (also a part of the Apache-supplied mod_proxy extension; it rewrites the path string in Set-Cookie headers), and ProxyHTMLURLMap (from mod_proxy_html, a third-party extension that rewrites URLs inside X/HTML documents).

Preparations

wget 'http://apache.webthing.com/mod_proxy_html/mod_proxy_html-2.5.2.c'
apxs -c -I/usr/include/libxml2 -i mod_proxy_html-2.5.2.c

Note that we are not using the mod_proxy_html author’s 3.0 version here. In my set-up, the 3.0 version was causing Apache HTTPD to dump core on every request (whether proxied or not), and the prior release works just fine for our purposes. The apxs line will compile, link, and copy the resulting library to the Apache modules directory for us.

The Configuration

#
#  Information about 'mod_proxy_html' can be found at 
#   http://apache.webthing.com/mod_proxy_html/
LoadFile    /usr/lib/libxml2.so
LoadModule  proxy_html_module    modules/mod_proxy_html-2.5.2.so
 
# DON'T TURN ProxyRequests ON!  Bad things will happen
# http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#access
# http://www.akadia.com/services/prevent_abuse_proxy.html
ProxyRequests off
 
# Necessary to have mod_proxy_html do the rewriting
RequestHeader      unset  Accept-Encoding
 
# Rewrite the URLs to proxy ("[P]") into the Tomcat server
RewriteEngine     on
RewriteRule ^(/.*)      ajp://localhost:8009/context_path/$1    [P]
 
# Be prepared to rewrite the HTML/CSS files as they come back
# from Tomcat
SetOutputFilter proxy-html
 
# Rewrite JavaScript and CSS files in addition to HTML files
ProxyHTMLExtended on
 
# Output Strict XHTML (add "Legacy" to the end of the line below
# to output Transitional XHTML)
ProxyHTMLDoctype XHTML 
 
# Rewrite HTTP headers and HTML/CSS links for everything else
ProxyPassReverse /context_path/ /
ProxyPassReverseCookiePath /context_path/ /
ProxyHTMLURLMap /context_path/ /

That’s pretty much all there is to it. You should note that mod_proxy_html, like any HTML scraper, requires modestly well-formed X/HTML. If the markup is bad, the output from mod_proxy_html is likely to be unpredictable.

Rob Casson of Miami University announced this weekend the beta availability of their video catalog. In a subsequent posting, Rob describes the user interface elements. Rob and the crew at Miami are seeking feedback on the interface, so if you have some be sure to offer it to them.

A couple of notes on the mechanisms Rob is using. Apache Solr is an open source enterprise search server based on the Lucene Java search library (also an Apache project). You can think of Lucene as the raw indexing and search engine with Solr layered on top to provide a non-Java interface to a rich feature set. What Miami has done is extract all of the bibliographic and related item records out of their Innovative Interface system, written programs to transform that data into XML, indexed it with Solr/Lucene and created a search interface.

Now what makes this really interesting is how much useful information is in the MARC record that doesn’t currently find its way into the WebPAC interface. For instance, this snapshot shows the facets where the MARC record has fielded data that can be turned into browsable lists:

Example from Miami University’s video catalog showing the available fielded data.

The corresponding WebPAC pre-search limits (for keyword searching) only includes a subset of languages, media formats, locations and does not include topic, genre, LC/SuDoc classes, and coverage date. In other words, there is a whole lot of information in the MARC record that isn’t being exposed in the normal WebPAC interface. Since Miami is in full control over the data in the Solr-based index, though, they are free to include as much or as little in the end-user interface.

Combined with faceted browsing, this makes for a very simple and quick interface to narrow down a large set of records. At the time of writing this entry, Miami’s video library consisted of 10,538 records. In three clicks, one can narrow that down to the 12 French-language comedy films in DVD format:

Example from Miami University’s video catalog facets after browsing for French-language Comedies on DVD.

From this screen, to see what is available in all media formats one need just click the red ‘X’ to the right of “DVD”. Also note the “RSS Feed” symbol on the right side of this interface snapshot. The results of any search/browse are immediately available as an RSS feed — a very convenient way to receive notifications of new titles that match this search!

Congratulations, Rob and everyone else at Miami that brought this interface into existence. It is a nice model and something we all can learn from through your experiences. Please keep us updated as the project continues.

The text was modified to remove a link to http://foam.lib.muohio.edu/blog/?p=13 on February 11th, 2011.

The text was modified to remove a link to http://foam.lib.muohio.edu/blog/?p=14 on February 11th, 2011.

Well, something is still going wrong on dltj.org — despite previous performance tuning efforts, I’m still running into cases where machine performance grinds to a halt. In debugging it a bit further, I’ve found that the root cause is an apache httpd process which wants to consume nearly all of real memory which then causes the rest of the machine to thrash horribly. The problem is that I haven’t figured out what is causing that one thread to want to consume so much RAM — nothing unusual appears in either the access or the error logs and I haven’t figured out a way to debug a running apache thread. (Suggestions anyone?)

In any case, I whipped up this little ditty that is running every five minutes in cron as a way to gloss over the problem for the moment. Running as root, it looks into all of the processes in the virtual /proc file system, specifically in the ‘stat’ file, and using awk looks to see if the second space-delimited value is the name of the httpd process (this is the Gentoo Linux distribution, so the name of the process is apache2) and the 23rd space-delimited value (the virtual size of the process) is bigger than 800MB. If so, it prints out the PID of the process (the first value in the stat file) at which the bash script unceremoniously sends it a kill (‘-9′) signal. The script looks like this:

#!/bin/bash
 
for i in `/bin/ls -d /proc/[0-9]*`; do
        if [ -f $i/stat ]; then
                pid=`/bin/awk '{ if ($2 == "(apache2)" && $23 > 800000000) print $1}' $i/stat`
                if [ "$pid" != "" ]; then
                        echo "Killing $pid because of load average: `awk '{print $1}' /proc/loadavg`"
                        kill -9 $pid
                fi
        fi
done

If anyone has any suggestions as to how to narrow down what the problem might be, I’d appreciate hearing from you. I’ve tried eliminating WordPress plugins, recompiling WordPress and Apache, and attempted to catch the behavior with a network traffic sniffer, but have come up empty so far.