In a futile effort to fight link rot on DLTJ, I installed the Broken Link Checker plugin by “White Shadow”. I like the way it scans the entire content of this blog — posts, pages, comments, etc. — looking for pages linked from here that don’t respond with an HTTP 200 “Ok” status code. The dashboard of problem links has a nice interface for updating or deleting these links, including the ability to add a CSS style deleted links to note that they were formerly there. One of the things I wished it did, though, was to add a message to posts/pages that noted a link was changed or deleted. You know — just to document that something changed since the page was first published. Tonight I hacked into the code to add this function. And with apologies to the original author of this beautifully structured object-oriented PHP code, it is a gruesome hack.

The Patch

Index: modules/parsers/html_link.php
===================================================================
--- modules/parsers/html_link.php       (revision 304787)
+++ modules/parsers/html_link.php       (working copy)
@@ -127,6 +127,9 @@
                //Find all links and replace those that match $old_url.
                $content = $this->multi_edit($content, array(&$this, 'edit_callback'), $args);  
 
+               // Append a notation to the end of the HTML that a link was changed
+               $content .= '<p style="padding:0;margin:0;font-style:italic;">The text was modified to update a link from '.$args['old_url'].' to '.$args['new_url'].'.</p>';
+
                return array(
                        'content' => $content,
                        'raw_url' => $new_url,
@@ -163,6 +166,9 @@
                //Find all links and remove those that match $raw_url.                
                $content = $this->multi_edit($content, array(&$this, 'unlink_callback'), $args);
+               // Append a notation to the end of the HTML that a link was deleted
+               $content .= '<p style="padding:0;margin:0;font-style:italic;" class="removed_link">The text was modified to remove a link to '.$args['old_url'].'.</p>';
+
                return $content;
        }
 
@@ -342,4 +348,4 @@
        }
 }-?>
\ No newline at end of file
+?>

You can see how this appears to the end-user by looking at the notes added to the bottom of this post.

Initial Results

cforms contact form by delicious:days

The WordPress Codex has documentation for running the login, registration, and administration interfaces on an SSL server. There is even a plug-in that will do much of the heavy lifting for you. I have found both of these methods, by themselves, to be rather unsatisfactory, though, in that admin services that rely on AJAX calls back to WordPress break (such as the periodic saving of drafts). What happens is this:

1. Plugins will use the ‘siteurl’ and/or ‘home’ values in the Options → General admin page, and that value is typically set to the “http://” rather than “https://” address of the blog.
2. The URL that plugins construct to talk back to the WordPress installation will go to an “http” address instead of the SSL-encrypted “https” address.
3. The admin page, loaded in the browser from the “https” address, attempts to talk back to the WordPress installation on a “http” address and triggers a exception. In Firefox, the error looks like this: Error: [Exception... "'Permission denied to call method XMLHttpRequest.open' when calling method: [nsIDOMEventListener::handleEvent]"...]

The security model in the browser prevents scripts on a page from using XMLHttpRequest¹ back to any host on the internet except for the host where the script came from. In this case, the difference between “http://…” and “https://…” is enough to trigger the problem.

So I fixed it with plug-in that uses an undocumented hook in WordPress 2.3. If a plugin requests the value of ‘siteurl’ or ‘home’, a filter is called to check if the requested page is on the SSL server. If it is, the filter changes the URL from ‘http’ to ‘https’. In that way, plug-ins will use the proper form of the URL.

< ?php
/*
Plugin Name: Fix Admin SSL
Plugin Script: fix_admin_ssl.php
Plugin URI: http://dltj.org/tag/fix_admin_ssl
Description: Fix the 'siteurl' and 'home' option values to make the protocol 'https' rather than 'http' when the page was requested with SSL.
Version: 1.0
License: GPL
Author: Peter Murray
Author URI: http://dltj.org/about
 
=== RELEASE NOTES ===
2008-02-18 - v1.0 - first version
*/
 
function fix_admin_ssl($url) {
  if ($_SERVER['HTTPS'] == 'on') {
    $url=preg_replace('/^http:\/\//','https://',$url);
  }
  return $url;
}
 
add_action ('option_siteurl', 'fix_admin_ssl', 1);
add_action ('option_home', 'fix_admin_ssl', 1);
 
?>

One downside to this plug-in, though, is that it will appear to change the values of ‘siteurl’ and ‘home’ on the Options → General admin page. The values in the database are still the ‘http’ ones, but since the Options page is an admin page the filter will run when it pre-loads those form fields.

If there is interest, I can package up the above code into a legitimate plugin and submit it to the WordPress plugins list.

Footnotes

1. See http://en.wikipedia.org/wiki/XMLHttpRequest for more information on XMLHttpRequest.

At least I hope that is the correct headline. I’ve been having some problems with this installation of WordPress lately — in particular, I could no longer activate or deactivate plugins — and the only solution offered in the WordPress codex was to start with a fresh installation of WordPress. Now you know how I spent my free time this weekend. While doing so, I updated the Barthelme theme and along the way gained some really need Semantic Web coolness to the underlying XHTML of the blog pages. The version of Barthelme is still a heavily, heavily hacked one, but hopefully the clean up this weekend will make it possible to keep up with new versions of the underlying theme files without major headaches. I also updated all of the plugins and cleaned out lots of old cruft in the plugins directory and in the theme files. As a result, the pages seem to load faster. Maybe that is just my wishful thinking.

cforms contact form by delicious:days

Also, a word of warning for those that try to fix their own WordPress blog, the process of getting your posts, content, and taxonomy terms from your old instance to your new instance is really, really broken. The suggested way to move content is by Exporting to a WordPress-specific XML format on your old blog, then importing it into your new blog. This caused many problems. First, if you are using a 2.3.x version of WordPress, you’ll run into a bug where tags are given numbers rather than the names. The patch — a one line change to the import PHP script — works as advertised. But then you’ll find that, if you have ever deleted a post or term, the act of importing the content will reassign new, sequential IDs — skipping over the deleted IDs in the old database as if they were never there. That’s okay, except many other plugins (most notably in my case, In-Series) rely on the IDs remaining constant because the ID number is stored in other tables. So save yourself a lot of trouble and just copy over the required tables in your underlying database. I used PhpMyAdmin to copy the wp_comments, wp_posts, wp_terms, wp_term_relationships, and wp_term_taxonomy tables from the old database to the new. And everything just worked right after that.

Last night DLTJ was upgraded to WordPress 2.3. As far as I can tell, everything is working okay, but please let me know in the comments or the comment form if something doesn’t seem right. There were two tricky parts to the upgrade. (Well, three really, if you count the tasks necessary to extract the reminants of the Ultimate Tag Warrior (UTW) from the theme.) Fortunately, one of them was not the upgrade itself; after abandoning the Gentoo portage ebuild for WordPress, I switched to the Subversion update method. This was the first time I did an ‘svn switch‘ to get the new version, and it worked great.

The first is dealing with the new tag infrastructure in Worpress 2.3. For those that read DLTJ via the website (as opposed to reading it through the RSS feed exclusively), you know that at the bottom of each posting is a table of tags with links to postings in DLTJ as well as links to items with similar tags in Technorati, del.icio.us, and a search for the tag in the english Wikipedia. Previously I had hacked UTW to provide that format, but without UTW available I bit the bullet and created a simple plug-in to create the table. It was my first plug-in for WordPress and I was quite proud of myself right up to the point where I couldn’t figure out why the table wasn’t appearing when the plug-in was enabled. So I went into a recent post to verify the tags and saw that the problem wasn’t with my plug-in — the problem was that there were no tags in the posting! It seems that I had missed the step of importing the UTW tags into the new WordPress 2.3 tag structure. I finally found out about it via the WordPress support forums, so here is a clue for those attempting the same thing — you’ll find utilities for importing tags in the…wait for it…import admin page. It would have been nice if the “/wp-admin/upgrade.php” utility saw that I was using a previous tagging system and pointed me in the direction of the import utility.

The second problem was easier to solve, and that was the missing functionality of Extended Live Archive on the DLTJ homepage. Fortunately, someone went through the effort of figuring out what needed to be changed in Extended Live Archive to accomodate the tag structure and published the results. I don’t like how the tag page doesn’t follow the common practice of font sizes based on popularity of the use of the tag, but I can get along without it. The built-in “tag cloud” functionality does a nice job of doing that.

Schemes to add functionality to the web OPAC fall into four categories: web OPAC enhancements, web OPAC wrappers, web OPAC replacements, and integrated library system replacements. I’m outlining these four techniques in a report I’m editing for an OhioLINK strategic task force and a bit of a reality check on this categorization is desired, so if I’m missing anything big (conceptually or announcements of projects/products that fall into these categories), please let me know in the comments. Generally speaking, this list is ordered by cost/complexity to implement — from lowest to highest — as well as the ability to offer the described enhanced services from least likely to most likely.

Web OPAC enhancements are functions that are added to the existing web OPAC system. This most often entails additional product purchases from the automation vendor, such as the optional enhancements in WebPAC Pro for Millennium OPACs or content solutions in SirsiDynix. Enhancement can also be added through creative use of an existing web OPAC’s template functions, such as the method by which LibraryThing for Libraries can be added to OPAC displays.

Web OPAC wrappers use the existing web OPAC provided by the integrated library system as a source of information, but hide that information behind a completely new interface. The intervening system get that information from the integrated library system through a variety of mechanism. In some cases, it may be possible to use established protocols (such as Z39.50) or programming interfaces (such as an XML content server). In cases where such functionality is not available from the underlying integrated library system, a “screen-scraping HTML” technique may be required. ¹

One example of such a wrapper is the work at Ann Arbor Public Library on SOPAC. Short for “Social OPAC,” SOPAC is “a set of social networking tools integrated into the AADL catalog [that] gives users the ability to rate, review, comment-on, and tag items.”² It uses an open source content management system called Drupal as a structure through which the added functionality is provided. For example, when a user seeks the bibliographic information page for a catalog record, that request is made from the user’s browser to the Drupal software. The Drupal software in turn makes a request to the integrated library system for the bibliographic information it holds. The response from the ILS is parsed by the Drupal software for key information such as title, author, subjects, holdings, etc. This information is mixed with information stored in the Drupal database (ratings, tags, reviews, cover images, etc.) and a new web page is created and returned to the user’s browser.

Another example of a web OPAC wrapper is Scriblio (formerly called WPopac). Using the underlying framework of WordPress, Scriblio offers faceted browsing, tagging, and syndication feeds for the underlying Millennium WebOPAC. Scriblio is a project of Plymouth State University, supported in part by the Andrew W. Mellon Foundation. Both SOPAC and Scriblio are available under open source licenses.

Web OPAC replacements are new systems that completely replace the existing web OPAC. Unlike wrappers (which get their bibliographic data in real-time from the underlying web OPAC), these replacements operate on sets of records that are extracted from the ILS or come from another source. (In some cases, these replacements still rely on the underlying web OPAC as a source of item status information such as checked out status and due date.) The first notable OPAC replacement was at North Carolina State University when its library installed and configured the Endeca software to provide a faceted browse to the library catalog. By itself, an Endeca OPAC display does not enable tagging, annotation, or user aggregation services such as recommendation engines.  Other similar web OPAC replacements are Encore from Innovative Interfaces³, Primo from Ex Libris and Aquabrowser from Medialab Solutions. Miami University’s experiments with the open source Apache SOLR and the exported records from their Millennium system also fall into this category. Worldcat Local is also a form of web OPAC replacement noting that the source of bibliographic records is the OCLC Worldcat database rather than the local ILS.

ILS replacements offer the biggest opportunity for enhanced user services, particularly by adopting one of the open source solutions now available. At this time, neither of the open source solutions (Evergreen and Koha) offers more than faceted search and browsing. Unlike the commercial systems, however, the source code of the system can be modified to add these functions, and the modifications shared with other users of the same system.

[Update 20071015T1624 : Corrections made -- and the text improved! -- based on Betsy Graham's comment. Thanks, Betsy!]

The text was modified to update a link from http://www.iii.com/mill/webopac.shtml to http://web.archive.org/web/20071015145500/http://iii.com/mill/webopac.shtml on January 20th, 2011.

The text was modified to update a link from http://www.iii.com/encore/main_index2.html to http://web.archive.org/web/20080328163000/http://www.iii.com/encore/main_index2.html on January 20th, 2011.

The text was modified to update a link from http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content to http://web.archive.org/web/20071109170053/http://www.sirsidynix.com/Solutions/Products/portalsearch.php#content on January 28th, 2011.

Footnotes

1. Such a technique gets the information from the ILS using the existing web OPAC. Such schemes are generally fragile because changes to the underlying web OPAC can have detrimental affects on the content scraping process.
2. Blyberg, J. (2007). AADL.org Goes Social. blyberg.net. Retrieved October 12, 2007, from http://www.blyberg.net/2007/01/21/aadlorg-goes-social/
3. As Betsy Graham, Vice President of Product Management at Innovative Interfaces, notes in the comments, the Encore will perform real-time queries to a Millennium ILS for bibliographic data, and in such cases the data extract is not needed.

I don’t think this has been widely announced, but while waiting for an update to Gentoo‘s portage entry for WordPress to cover the latest security and bug fixes, I discovered in the comments of a bug in Gentoo’s bugzilla database that they are making no effort to support WordPress on Gentoo. I think this is really a poor move on Gentoo’s part. As one of the bug commenters noted, “WordPress is, in general, a good product with an extremely active user community and good upstream maintenance.” ¹

A GLSA was issued for WordPress vulnerabilities in March — problems that have since been fixed — with this ‘resolution’:

Due to the numerous recently discovered vulnerabilities in WordPress, this package has been masked in the portage tree. All WordPress users are advised to unmerge it.

Now I’ll admit that I should not be expecting updates to packages that have been hard masked, but really — is this any way to treat the world’s most popular blogging software? In any case, I’ve abandoned Gentoo’s ebuild for WordPress and have reverted to the Subversion method for updating a WordPress installation. So here is a heads-up to do something similar should you find yourself in the same situation.

[20070812T2046 update: The portage keepers bumped the version of WordPress to 2.2.2 in Portage yesterday.]

Footnotes

1. Comment #16 on bug #168529 from Stephen Ulmer dated 2007-03-17 16:09:26 0000

This is a list of links to other sites from within the postings of DLTJ:

###linkharvest###

DLTJ was upgraded to WordPress 2.1 tonight. Everything looks good so far; please let me know if you disagree…

Well, something is still going wrong on dltj.org — despite previous performance tuning efforts, I’m still running into cases where machine performance grinds to a halt. In debugging it a bit further, I’ve found that the root cause is an apache httpd process which wants to consume nearly all of real memory which then causes the rest of the machine to thrash horribly. The problem is that I haven’t figured out what is causing that one thread to want to consume so much RAM — nothing unusual appears in either the access or the error logs and I haven’t figured out a way to debug a running apache thread. (Suggestions anyone?)

In any case, I whipped up this little ditty that is running every five minutes in cron as a way to gloss over the problem for the moment. Running as root, it looks into all of the processes in the virtual /proc file system, specifically in the ‘stat’ file, and using awk looks to see if the second space-delimited value is the name of the httpd process (this is the Gentoo Linux distribution, so the name of the process is apache2) and the 23rd space-delimited value (the virtual size of the process) is bigger than 800MB. If so, it prints out the PID of the process (the first value in the stat file) at which the bash script unceremoniously sends it a kill (‘-9′) signal. The script looks like this:

#!/bin/bash
 
for i in `/bin/ls -d /proc/[0-9]*`; do
        if [ -f $i/stat ]; then
                pid=`/bin/awk '{ if ($2 == "(apache2)" && $23 > 800000000) print $1}' $i/stat`
                if [ "$pid" != "" ]; then
                        echo "Killing $pid because of load average: `awk '{print $1}' /proc/loadavg`"
                        kill -9 $pid
                fi
        fi
done

If anyone has any suggestions as to how to narrow down what the problem might be, I’d appreciate hearing from you. I’ve tried eliminating WordPress plugins, recompiling WordPress and Apache, and attempted to catch the behavior with a network traffic sniffer, but have come up empty so far.