Chris Wilper gave this presentation on behalf of the work that he and Aaron Birkland did to improve the performance of the Fedora Resource Index.

Version 2.0 of the Fedora digital object repository software added a feature called the Resource Index (RI). Based on Resource Description Framework (RDF) triples, the RI provided quick access to relationships between objects as well as to the descriptive elements of the object itself. After about two years of use using the Kowari software, the RI has pointed to a number of challenges for “triplestores”: scalability (few triplestores are designed for greater than 100 million triples); performance; and stability (frequent “rebuilds”).

The real motivation behind experimenting with a new triplestore, however, was the NSDL use case. The National Science Digital Library (NSDL) is a moderately large repository (4.7 million objects, 250 million triples) with a lot of write activity (driven by periodic OAI harvests; primarily mixed ingests and datastream modifications). The NSDL data model also includes existential/referential integrity constraints that must be enforced. Querying the RI to determine correct repository state proved to be difficult: Kowari is aggressively buffering triple, sometimes on the order of seconds, before writing them to disk. Flushing the buffer after every write is also computationally expensive (hence the drive to use buffers in the first place).

The NSDL team also encountered corruption under concurrent use and with abnormal shutdowns, forcing the rebuild of the triplestore. And the solution was not scaling well; performance was becoming notably worse. In looking for solutions other triplestores were considered but rejected. Using a RDBMS seemed attractive — efficient transactions, very stable, generally speedy — but a “one big table” paradigm to store all of the relations did not seem to give them a desired scalability.

NSDL developers observed that total number of distinct predicates is much lower than the number of predicates or objects; NSDL has about 50 distinct predicates. Based on this observation, their solution, called “Mapped Predicate Tables,” creates a table for every predicate in the triplestore. This has several advantages: a low computational cost for triple adds and deletes, queries for known predicates are fast, complex queries benefit from the relatively mature RDBMS planner having finer-granularity statistics and query plans, and flexible data partitioning to help address scalability. This solution comes with several disadvantages, however: one needs to manage predicate to table mapping, complex queries crossing many predicates require more effort to formulate, and with a naive approach simple unbound queries scale linearly with the number of predicates.

So the NSDL team created the MPTStore triplestore and contributed it back to the Fedora core developers for use by the community. MPTStore is a Java library that handles all of the predicate mapping and accounting behind the scenes. The basic API remains the same as for other triplestores, performing triple writes and queries, and the library hides all of the implementation details of translating queries from a particular language (SPO, SPARQL) into SQL statements. The library is also designed to expose transaction/connection semantics should the developer wish to have direct access to the predicate tables.

A solution like MPTStore is well suited for NSDL use case. The NSDL team was very familiar with the operations of RDBMS administration: performance tuning, backups, etc. The stored triplestore data is transparent and “hackable” — adhoc SQL queries and analysis are relatively simple. In fact, the RDBMS triplestore helped track down Fedora middleware bugs that resulted in an inconsistent state. Fixing these bugs also improved the performance of the Kowari-based RI.

[Updated 20070129T1447 to include links to Chris' presentation on SlideShare.]

During the cookies and lemonade break during JCDL this afternoon I surprised one of the well-respected elders of the field with this question: are we really making progress? are we winning a fight against entropy ¹? I wasn’t out for a quote for publication at the time so I won’t reveal the individual’s name, but I will report that there was a chuckle then the reply “cautiously optimistic.”

This person went on to say that access to raw information has improved much over the last five years — that the internet and its tools have increased the capacity to publish and retrieve information. ‘Sure,’ s/he went on to say, ‘we have a number of hard problems to solve — linking related object to each other and so forth — but we are making progress.’ I, too, offered a chuckle and agreed, and we went back to our cookies and lemonade.

Entropy and chaos are powerful forces, however, and it was just after this brief encounter that we heard from Carl Lagoze with a talk called Metadata aggregation and “automated digital libraries”: A Retrospective on the NSDL experience. Although the paper is a modestly dry report on the issues resolved and overcome in “running a relatively large-scale digital library (over a million objects) by collecting, processing, storing, and using metadata” ², the oral presentation was anything but dry. In fact, it offered a sobering reminder of how hard this is and the challenges before us. He did it with four questions:

1. What is a digital library anyway?
2. What is the role of metadata in a digital library?
3. What is “low barrier” technology? [This one was tied to the observation that OAI-PMH, while modestly simple compared to other protocols, still requires a lot of effort to get right. See reality lesson #4 below.]
4. Where should expensive and limited human energy be allocated?

… and seven reality lessons:

• Reality lesson #1: Metadata is not being created
  In truth, there is not a lot of funding set aside in projects to create metadata.
• Reality lesson #2: Participating as a metadata provider is complicated by a “knowledge gap”
  Doing so requires three skill sets that are frequently distinct: Domain expertise (e.g. “mathematics”); Metadata expertise (e.g. “Dublin Core”); and Technical expertise (e.g. encode it in XML and use a formal protocol).
• Reality lesson #3: Harvested metadata is not necessarily useful metadata
  “Correct” metadata is not necessarily “rich” metadata. The general problem of metadata quality remains unsolved — even the best automated/automatic transformations are not good enough.
• Reality lesson #4: OAI-PMH is not necessarily low-barrier and automatic
  Doing OAI-PMH right incorporates lots of details and assumed knowledge (UTF-8, XML schema validation, URL encoding, date stamping, resumption tokens, etc.). An even after sometimes months of hand-holding data provider, the initial success does not persist in the majority of cases; the failure rate of subsequent harvests is high. And the “incremental harvest” functionality is a nice concept but it doesn’t work: support for “deleted” records is inconsistent in data providers; less than 50% of providers claim to persist deletions and many persistent claims are faulty. Too often server failures and harvest failures require a full harvest ‘resync’.
• Reality lesson #5: Human cost of large-scale harvesting is high
  In the case of NSDL, their metrics show that they exchange 170 messages per year per provider and that it takes on average 98 message exchanged for first harvest to succeed (which, as previously noted, subsequently fails).
• Reality lesson #6: Matching individual metadata records of equivalent resources is hard
  I didn’t have anything in my notes about this, but as I recall his comments were about the lack of ways to uniformly handle these surrogate objects in the OAI-PMH protocol.
• Reality lesson #7: Lots of (even good) metadata does not make a complete digital library (and maybe not even a digital library that is highly useful for education)
  There is a real need to understand the value-add of a digital library: capturing the wisdom of the community served as well as focusing less on structured information and more on relationships among resources and user-derived relationships and annotations.

So what do I think? You know — I’m not sure. These are tough problems, and the world would be a better place if they were solved. We can demand answers, but sometimes there just isn’t enough of a shoulder to stand on from the giant below. Still, one can’t help but wonder if all of the energy put into the collective “digital library” problem so far has just dissipated into chaos.

Footnotes

1. Defined as: “Measure of disorganization or degradation in the universe that reduces available energy, or tendency of available energy to dwindle. Chaos, opposite of order.” Do you remember your Second Law of Thermodynamics?
2. Lagoze, C., Krafft, D. B., Cornwell, T., Dushay, N., Eckstrom, D., Saylor, J. 200y. Metadata aggregation and “automated digital libraries”: A retrospective on the NSDL experience. In Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (Chapel Hill, NC, USA, June 11 – 15, 2005). JCDL ’06. ACM Press, New York, NY, 231. [arXiv:cs.DL/0601125]