Disruptive Library Technology Jester » DRC

Position Announcement: OhioLINK Systems Developer

Peter Murray — Fri, 05 Sep 2008 17:59:25 +0000

The Ohio Library and Information Network (OhioLINK) is seeking a hard-working, analytical individual to participate in the creation and maintenance of our internationally recognized set of online library information services, with special focus on the Ohio Digital Resource Commons. OhioLINK serves the higher education population in the State of Ohio with over 85 college and university member institutions.

The position requires a four-year degree in Computer Science, or a graduate degree in Information or Library Science, or equivalent technical experience. The candidate should have strong programming skills in languages such as Java, and should be comfortable working in a Unix/Linux environment with open source software. Experience with the following is highly valued: Digital Repositories, Cocoon, Apache Tomcat, XML/XSLT, PostgreSQL. Experience with the following is desirable: DSpace/Manakin, HTML/CSS site design, metadata, Subversion, Perl, shell scripting.

Salary: $49,000 minimum

If you are interested in this position, please send a resume, a summary statement of experience, and an indication of your salary expectations to resume@ohiolink.edu.

The text was modified to update a link from http://www.ohiolink.edu/member-info/ to http://www.ohiolink.edu/members-info/ on January 13th, 2011.

Two Personal Repository Services

Peter Murray — Mon, 04 Jun 2007 03:03:23 +0000

This year has seen the release of two personal repository services: http://PublicationsList.org/ and the U.K. Depot. These two services have an admittedly different focus, but I think it is still interesting to compare and contrast them to see what we can learn.

The Depot

The Depot provides one-stop place for U.K.-based researchers to deposit refereed articles, book chapters, and conference papers. It is “one-stop” in that The Depot can forward the author to his/her institution-based repository or, in the case where the author’s institution does not have a repository, upload and host the content right from The Depot.

The deposit interface, for those putting content directly into the centralized Depot repository, has four main stages. First, the “Type” stage, specifying whether the object is an article, a book chapter, or a conference paper:

Next, the “Upload” stage, where one can upload the file and supply a few more properties:

Then the “Details” stage, where the descriptive metadata (minus the controlled vocabulary subjects — that comes in the next screen) is input:

And finally, the “Subjects” page, with an AJAX-driven expanding-and-collapsing hierarchy of subjects:

To retrieve contents from the repository, there is a “browse” interface for looking by ‘year’ or by ‘subject’ — no other browse facets and no search interface. The Depot was just formally released this month, so I would bet that functionality like that is in the works.

PublicationsList.org

PublicationsList is a commercial service with a free, limited-functionality version. Unlike The Depot (and similar institutional repository systems), the focus is on putting together and publishing a personal bibliography with the deposit function taking a secondary role (and only for paid subscribers of the service).

The single item entry page is a just-the-facts interface. Note that the content hosting service is only available to those who have upgraded to the “Publications List Professional” version (which costs £9.99, or approx $20/€15, per year).

The system can also accept a variety of citation manager file formats for bulk entry. (See snapshot to the right.) PublicationsList also has a built in search-and-select interface to PubMed for finding publications matching your name and automatically populating the metadata fields in your personal citation.

Then end result is a web-based bibliography with links to the publications (either hosted on PublicationsList or on other sites). The free version is hosted on PublicationsList.org (see the service founder’s page as an example) and the professional version can embed the publications list in your own page.

PublicationsList does provide discounts and additional functionality for groups (such as departments, research centers, etc.).

Observations

Both The Depot and PublicationsList provide interesting suites of features for academics seeking to get their content online, but neither really addresses the problems of getting academics to put their content online. ¹ The search-and-select interface for PubMed is very helpful in cutting down on the data entry required to populate a citation entry. If OhioLINK were to replicate this service, we could tap into not only PubMed but also the wide variety of index/abstract databases and electronic journals that we host. The automatic handling of various forms of citation management data is also nice. I don’t think PublicationsList offers an export feature, which would be good to have so that an author can add entries found through the search-and-select interface back into their personal bibliographic management software.

The one-stop, redirection service in The Depot is a good concept, too. If a researcher wanted to deposit their content in a repository and they weren’t sure if their institution had a repository to hold it, OhioLINK would be a natural place to look for a content hosting service in the state and we could redirect the author to the appropriate location on a campus. OhioLINK could also be playing the role of repository-of-last-resort for Ohio academic researchers by providing a space and services for published content, whether or not the institution in question has set up a formal repository space on the DRC.

Footnotes

For a really good discussion of that problem, see Davis, P.M., & Connolly, M.J.L. (2007). Institutional Repositories: Evaluating the Reasons for Non-use of Cornell University’s Installation of DSpace. D-Lib Magazine, 13(3/4). Retrieved March 14, 2007, from http://www.dlib.org/dlib/march07/davis/03davis.html.

Disseminators As the Core of an Object Repository

Peter Murray — Thu, 26 Apr 2007 13:58:32 +0000

I’ve been working to get JBoss Seam tied into Fedora, and along the way thought it would be wise to stop and document a core concept of this integration: the centrality of Fedora Disseminators in the the design of the Ohio Digital Resource Commons. Although there is nothing specific to JBoss Seam (a Java Enterprise Edition application framework) in these concepts, making an object “render itself” does make the Seam-based interface application easier to code and understand. A disseminator-centric architecture also allows us to put our code investment where it matters the most — in the repository framework — and exploit that investment in many places. So what does it mean to have a disseminator-centric architecture and have objects “render themselves”?

How It Works

This is a sequence diagram showing all of the pieces:

Browser: The user’s browser
DRCseam: A JBoss Seam application that generates the user interface and performs much of the business logic. DRCseam, however, does not render the objects or their metadata into browser-consumable artifacts. Read on!
Fedora: A basic Fedora digital object repository.
Disseminator: A simple servlet that performs various transformations on object datastreams to render content usable by the browser.

With these components in play, here is the description of a sequence to render a page showing the metadata for a repository item:

request item page: The browser follows a link to an item detail page.
API-A ObjectProfile: The interface application asks the repository for the ‘Object Profile’ of the item…
return object profile: …which the repository returns. The interface application now knows basic details about the object: that it exists, the creation and updated timestamps, and so forth.
API-A DatastreamDissemination for fullDisplay: The interface application needs the object’s metadata display, so it asks the object to “render itself” by making a call to the Fedora repository for the object’s “FullDisplay” disseminator.
call getFullDisplay: The Fedora repository in turn calls the object’s disseminator with the Persistent Identifier (PID) of the object as a parameter.
API-A Datastream for metadata: Using the object PID, the disseminator calls back to the Fedora repository for the descriptive metadata datastream (the DC datastream, in this case)…
XML metadata: …which the Fedora repository returns.
transform metadata: The disseminator performs some transformation or derivation on the descriptive datastream to create an XHTML representation…
XHTML fragment: …which it returns to the Fedora software…
XHTML fragment: …which is returned to the interface application…
XHTML page: …which inserts it at the appropriate place in the XHTML page it has built and returns the XHTML page to the browser.

Step #4 is where we diverge from previous architectures. Instead of making the interface application transform the metadata into a human-readable format, the interface application calls the object’s disseminator to do the job.

The Heart of It All: The Disseminator

The key to this architecture is asking the object to “render itself”. This puts the task of creating the appropriate representation at the object level. The object can be an image, a video, a spreadsheet, or a PDF file. More importantly, the object can be a PDF of a journal article or a PDF of a thesis; in both cases the metadata describing that PDF file will be different (journal/volume/issue in one case and department/degree/advisor in the other).

Rather than putting special case code in the interface application to render the description of the journal article one way and the thesis another way, that special case code is bound to the object in the form of a “disseminator”. The disseminator methods for the journal article and the thesis share the same name — getFullDisplay — but will return entirely different XHTML fragments — one for a journal article and one for a thesis. For both objects, though, the interface application will make a call to the object in the Fedora repository asking for the output of each getFullDisplay dissemination. In the case of a Dublin Core description, the dissemination output could look like this:

<table class="drc_dublinCore_table">
<tr class="drc_dublinCore_row drc_dublinCore_title">
<td class="drc_dublinCore_label drc_dublinCore_title">Title:</td>
<td class="drc_dublinCore_value drc_dublinCore_title">Jester Example</td>
</tr>
<tr class="drc_dublinCore_row drc_dublinCore_identifier">
<td class="drc_dublinCore_label drc_dublinCore_identifier">Identifier:</td>
<td class="drc_dublinCore_value drc_dublinCore_identifier">demo:exampleObject</td>
</tr>
</table>

You’ll note that there is a liberal application of CSS styles on all of the XHTML elements, allowing for the look of the dissemination to be further transformed in the browser via CSS stylesheets. A getFullDisplay dissemination for a journal article could look like this:

<table class="drc_ejc_table">
<tr class="drc_ejc_row drc_ejc_title">
<td class="drc_ejc_label drc_ejc_title">Article Title:</td>
<td class="drc_ejc_value drc_ejc_title">Taking Advantage of Fedora Disseminations</td>
</tr>
<tr class="drc_ejc_row drc_ejc_volume">
<td class="drc_ejc_label drc_ejc_volume">Volume:</td>
<td class="drc_ejc_value drc_ejc_volume">3</td>
</tr>
<tr class="drc_ejc_row drc_ejc_issue">
<td class="drc_ejc_label drc_ejc_issue">Issue:</td>
<td class="drc_ejc_value drc_ejc_issue">2</td>
</tr>
</table>

Looking at the Pieces

There is a demonstration system set up for a short period of time that shows all of the pieces. First, the disseminator:

http://drc-dev.ohiolink.edu:8080/BaseDisseminator/getFullDisplay/demo:exampleObject

Next, how this disseminator looks as accessed through the Fedora repository:

http://drc-dev.ohiolink.edu:8080/fedora/get/demo:exampleObject/demo:bDefExample/getFullDisplay/

And finally, how this result looks through the Seam-based interface application. (A note about this application — only this URL works at the moment even though there are other links on the page. This is also the ‘trunk’ version of our interface code, so it is likely to change and/or break and/or work better at any time.)

http://drc-dev.ohiolink.edu:8080/drc/item.seam?itemId=demo%3AexampleObject

Fedora Setup

In addition to the Seam-based interface application and the disseminator code, there is setup required at the Fedora repository — specifically, the creation of a Behavior Definition (bDef) that describes the disseminators that the objects share in common and the creation of a Behavior Mechanism (bMech) that describes the implementation of that definition for a particular object type. Below is a series of screen shots that show the steps to create the bDef and bMech.

Disseminator Behavior Definition (bDef)

Using the Fedora Admin client, under the “Builders” menu, select “Behavior Definition Builder”. The first pane, “General” parameters, use a specific PID of ‘demo:bDefExample‘ and put something in for the Behavior Object Name, Behavior Object Description, and one of the Dublin Core Metadata fields. (It doesn’t matter what you put in for these values.)

Under the “Abstract Methods” pane, create new definitions for each of the disseminator methods.

Under the “Documentation” pane, put something in the first entry. Again, it doesn’t matter what is put in for these values, but they are required.

Select “Ingest” at the bottom of the window, and the demo:bDefExample bDef will be created. Alternatively, you could import the demo:bDefExample saved in the DRC source code repository (choose “original format” at the bottom of that page).

Disseminator Mechanism Definition (bMech)

The bMech is a little more complicated. Under the “Builders” menu, select “Behavior Mechanism Builder”. The first pane, “General” parameters, use a specific PID of ‘demo:bMechExample‘ and put something in for the Behavior Object Name, Behavior Object Description, and one of the Dublin Core Metadata fields. (It doesn’t matter what you put in for these values.) In the “Behavior Definition Contract” pick the bDef just created (demo:bDefExample).

In the “Service Profile” pane, put in values in the “General” area (it doesn’t matter what). In the Service Binding area, make sure the Message Protocol is HTTP GET, put in text/html, text/xml for Input MIME Types and put in text/html, text/xml, text/plain for Output MIME Types.

Under the Service Methods pane, put in http://localhost:8080/BaseDisseminator for the Base URL. (The disseminator is also loaded in the same servlet as the Fedora repository and the Seam interface application, and it is loaded at the “/BaseDisseminator” context path in the servlet.) Create Service Method Definitions that correspond to the Abstract Methods in the bDef.

Select “Properties” for each one of the Service Method Definitions in turn. “echo” is a unique disseminator method that simply echos back the context parameters of the disseminator request. This is useful for seeing exactly what the Fedora server is going to give to the disseminator.

With the exception of “echo” all of the other Service Method Definitions are the same. The Method Binding consists of the disseminator method followed by a slash and the PID placeholder followed by a question mark and ‘dc’ equals the DC placeholder. Since the Method Binding field has two placeholders, there are two entries in the Method Parameter Definitions area. The first is for PID — a “Default” parameter that is required and passed by value to the disseminator. The default value is the special value $PID, which the repository software will replace with the PID of the object as the disseminator is called. The second is for DC, a “Datastream” parameter that is required and passed to the disseminator by URL reference. The disseminator doesn’t actually use this reference to a datastream, but it is a requirement that all bMechs pass a datastream of one sort or another to the disseminator.

If you have followed all of the steps so far, under the “Datastream Input” pane there will be one entry for DC in the table. The only thing that needs to be done here is adding “text/xml” in the MIMEType column.

Under the “Documentation” pane, put something in the first entry. Again, it doesn’t matter what is put in for these values, but they are required.

Select “Ingest” at the bottom of the window, and the demo:bMechExample bMech will be created. Alternatively, you could import the demo:bMechExample saved in the DRC source code repository (choose “original format” at the bottom of that page).

Sample Object

The last step is to add this disseminator bDef/bMech combination to an object. Edit any object in the repository and go to the “Disseminators” pane. If there are other disseminators already defined for this object, select “New” along the left side. Put in a label — any label will do. Next to “Behavior defined by…” select demo:bDefExample. Then next to “Mechanism” select demo:bMechExample. The admin client will prompt for a DC binding; select “Add” and choose the DC datastream in the pop-up window.

Select “Save Changes” at the bottom. The completed disseminator looks like this:

There is a sample object in the DRC source code repository that has the disseminator already defined.

Notes

Comments about this architecture are certainly welcome. I’m sure I’ll be writing about it more in the future, but here are some thoughts at this point:

Future Directions

In this case, I’m using an XSLT stylesheet to transform the Dublin Core XML into an XHTML table. That stylesheet is stored in the BaseDisseminator WAR file. The stylesheet could just as easily be a datastream of a special “formatting” object in the repository. One of the key distinctions of OhioLINK’s Fedora implementation is that institutions using the repository will be able to “brand” their content in any way they choose. Having the flexibility of storing metadata transformations just like any other object in the repository would seem to be of great advantage in that scenario.

On a related front, this style of implementation would be greatly enhanced by the work of the Fedora Content Model Dissemination Architecture (CMDA). Because disseminators must be bound to specific objects rather than classes of objects, management of the variety of bMechs in a scenario such as this will likely become difficult very soon. I’m heartened by the fact that the CMDA work is going on and will cut our management complexity dramatically when it becomes available.

Acknowlegements

These concepts are based in part on the work of the Digital Library Federation’s Aquifer Asset Actions technical working group and discussions among members of the OAI Object Reuse and Exchange technical committee as well as conversations with many Fedora developers and implementors. Thanks, everyone.

[Update 20070426T1147 : Fixed the sample object URL. Thanks, Jodi.]

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_exampleObject.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_bMechExample.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_bDefExample.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/drc/item.seam?itemId=demo%3AexampleObject on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/fedora/get/demo:exampleObject/demo:bDefExample/getFullDisplay/ on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/BaseDisseminator/getFullDisplay/demo:exampleObject on January 13th, 2011.

The text was modified to update a link from http://rama.grainger.uiuc.edu/assetActions/ to https://wiki.dlib.indiana.edu/display/DLFAquifer/Asset+Action+Project on January 19th, 2011.

Building an Institutional Repository Interface Using EJB3 and JBoss Seam

Peter Murray — Fri, 19 Jan 2007 04:00:49 +0000

This tour is designed to show the overall architecture of a FEDORA digital object repository application within the JBoss Seam framework while at the same time pointing out individual design decisions and extension points that are specific to the Ohio Digital Resource Commons application. Geared towards software developers, a familiarity with Java Servlet programming is assumed, although not required. Knowledge of JBoss Seam, Hibernate/Java Persistence API, EJB3 and Java EE would be helpful but not required; brief explanations of core concepts of these technologies are included in this tour.

The tour is based on revision 709 of /drc/trunk and was last updated on 18-Jan-2007.

This tour will also be incorporated into a presentation at Open Repositories 2007 on Tuesday afternoon.

Directory Layout

The source directory tree has four major components: ‘lib’, ‘resources’, ‘src’, and ‘view’.

lib – libraries required by the application. The lib directory contains all of the JAR libraries required by the application. Its contents is a mix of the Seam-generated skeleton (pretty much everything at the top level of the ‘lib’ directory) and JAR libraries that are specific to the DRC application (in subdirectories of ‘lib’ named for the library in use). For instance, the ‘commons-codec-1.3′ and the ‘hibernate-all’ and the ‘jboss-seam’ JAR files were all brought into the project via ‘seam-gen’ while ‘lib/commons-net-1.4.1/commons-net-1.4.1.jar’ library was added specifically for this project. A convention has been established whereby new libraries added to the project appear as entries in the lib.properties file which is used by series of directives in the build.xml file to setup the classpaths for compiling and for building the EJB JAR. This is done to make the testing and transition of new libraries into the application more explicit and easily testable. Note that the newly included library directory also includes a copy of any license file associated with that library; this is not only a requirement to use some libraries but is also a good practice to show the lineage of some of the lesser known libraries. (For an example of what is required, see the changes to build.xml and to lib.properties in order to bring the Apache Commons Net library into the application.)

resources – configuration files and miscellaneous stuff. The resources directory holds the various configuration files required by the application plus other files used for testing and demonstration. Much of this was generated by the Seam-generated skeleton as well. Some key files here are the import.sql file (SQL statements that are used to preload the RDBMS used by Hibernate as the mocked up repository system) and the test-datastreams directory which has sample files for each of the media types.

src – Java source code. The src directory contains all of the Java source code for the application. Everything exists in a package called ‘edu.ohiolink.drc’ with subpackages for classes handling actions from the view component of the MVC, entity beans (sometimes known as Data Access Objects — or DAOs — I think), exception classes (more on this below), classes for working with FEDORA (not currently used), media type handler classes (more on this below), unit test classes (not currently used), and utility classes.

view – XHTML templates, CSS files, and other web interface needs. The view directory holds all of the files for the “view” aspect of the Model-View-Controller paradigm. More information about the view components is below.

Entity Classes

The entity beans package has three primary entity beans defined: Item.java, Datastream.java, and Description.java. (The FedoraServer.java entity bean is not used at this time.) Item.java is the primary bean that represents an object in the repository. Datastream.java and Description.java are component beans that only exist in the lifecycle of an Item.java bean; Datastream.java holds a representation of a FEDORA object datastream and Description.java holds a representation of a Dublin Core datastream for that object.

The Datastream and Description objects are annotated with @Embedded in the Item.java source; this is Hibernate’s way of saying that these objects do not stand on their own. Item.java also has numerous methods marked with a @javax.persistence.Transient annotation meaning that this is information not stored in the backing Hibernate database; these methods are for the various content handlers, which will be outlined below.

Mock Repository

As currently configured, the entity beans pull their information from a static RDBMS using Hibernate rather than from an underlying FEDORA digital object repository. (You’ll need to go back to revision 691 to see how far we got with the FEDORA integration into JBoss Seam before we switched our development focus to the presentation ‘view’ aspects of the application.) As currently configured, Hibernate uses an embedded Hypersonic SQL database for its datastore. As part of the application deploy process, the Java EE container will instantiate a Hypersonic database and preload it with the contents of the import.sql file. (The import.sql file contains just three sample records at the moment: one each for a text file, a PDF file, and a graphic file.)

All of the data for a repository object is contained in a single table record. Hibernate manages the process for us of reading that record out of the database and creating the three corresponding Java objects: Item, Datastream and Description. (Hibernate could also handle the process of updating the underlying table record if we were to change a value in one of the Java objects.) The mapping of table column to Java object field is handled by the @Column(name="xx") annotations in the entity beans.

For Datastream, what is stored in the database is not the datastream content itself but rather a filename that points to the location of the datastream file. The file path in this field can either be absolute (meaning a complete path starting from the root directory of the filesystem) or a relative path. In the case of the latter, the path is relative to the deployed application’s WAR directory (something like “…/jboss-4.0.5.GA/server/default/deploy/drc.ear/drc.war/” for instance). Note that the getter/setter methods for the contentLocation are private — the rest of the application does not need to know the location of the datastreams; this will also be true when the DRC application is connected to a FEDORA digital object repository. The method marked public instead is getContent, and the implementation of getContent hides the complexity of the fact that the datastream is coming from a disk file rather than a FEDORA repository call. For the three records/repository-objects currently defined in ‘import.sql’ there are three corresponding demo datastreams in the test-datastreams directory.

In all likelihood, this representation of the FEDORA repository will be too simple for us to move forward much further. In particular, the current notion of one datastream per repository object is too simplistic. The Datastream embedded object will likely need to be broken out into a separate table and as a corresponding distinct Java applet. (We may reach the same point soon for the Description object as well.)

By using the Entity Beans as a buffer between the business logic and the view components of the rest of the application, I hope we can minimize/localize the changes required in the future in order to replace the mock repository with a real underlying FEDORA repository.

View Templates

The preferred view technology for JBoss Seam is Facelets, an implementation of Java Server Faces that does not require the use of Java Server Pages (JSP). Although the ‘.xhtml’ pages in the view directory bear a passing resemblance to JSP, behind the scenes they are radically different. Of note for us is the clean templating system used to generate pages. The home.xhtml file has a reference to the template.xhtml file in the ‘layout’ directory. If you read through the template.xhtml file, you can see where the Facelets engine will pull in other .xhtml files in addition to the content within the tag of home.xhtml.

Content Handlers

The paradigm of handling different media types within the DRC application is guided in large part by the notion of disseminators for FEDORA objects and the Digital Library Federation Aquifer Asset Actions experiments. The underlying concept is to push the media-specific content handling into the digital object repository and to have the presentation interface consume those content handlers as it is preparing the end-user presentation.

For instance, the DRC will need to handle content models for PDFs, images, video, and so forth. Furthermore, how a video datastream from the Digital Video Collection is offered to the user may be different than how a video datastream from a thesis is offered to the user. Rather than embedding the complexity of making those interface decisions into the front-end DRC application, this model of content handlers pushes that complexity closer to the objects themselves by encoding those behaviors a disseminators of the object. What the presentation layer gets from the object is a chunk of XHTML that it inserts into the dynamically generated HTML page at the right place.

There is work beginning on a framework for FEDORA disseminators at /BaseDisseminator/trunk in the source code repository; that work has been put on hold at the moment in favor of focusing on the presentation interface. In order to prepare for the time when the presentation behaviors are encoded as FEDORA object disseminators, the current presentation layer makes use of Content Handlers for each of the media types. The Handler interface defines the methods required by each handler and the TextHandler class, the ImageHandler class, and the PdfHandler class implement the methods for the three media types already defined.

Of these, TextHandler class is the most complete, so I’ll use it as an example.

The getRawDatastream method takes the datastream and sends it back to the browser with the HTTP headers that cause a File-Save dialog box to open.
The getFullDisplay method returns a chunk of XHTML that presents the full metadata in a manner that can be included in a full metadata display screen.
The getRecordDisplay method (currently unwritten) returns a chuck of XHTML used to represent the object in a list of records that resulted from a user’s search or browse request.
The getThumbnail method (currently unwritten) returns a static graphic thumbnail rendition of the datastream (e.g. a cover page, a key video frame, etc.).

By making these content handlers distinct classes, it is anticipated that the rendering code for each of these methods can be more easily moved to FEDORA object disseminators with minimal impact to the surrounding DRC interface application.

Exception Handling

The DRC application follows the practice suggested by Barry Ruzek in Effective Java Exceptions (found via this link on The Server Side). The article can be summarized as:

One type of exception is a contingency, which means that a process was executed that cannot succeed because of a known problem (the example he uses is that of a checking account, where the account has insufficient funds, or a check has a stop payment issued.) These problems should be handled by way of a distinct mechanism, and the code should expect to manage them.
The other type of exception is a fault, such as the IOException. A fault is typically not something that is or should be expected, and therefore handling faults should probably not be part of a normal process.
With these two classes of exception in mind, it’s easy to see what should be checked and should be unchecked: the contingencies should be checked (and descend from Exception) and the faults should be unchecked (and descend from Error).

All unchecked exceptions generated by the application are subclasses of DrcBaseAppException. (DrcBaseApplication itself is a subclass of RuntimeException.) For an example, see NoHandlerException. By setting up all of the applications exceptions to derive from this point, we have one place where logging of troubleshooting information can take place (although this part of the application has not been set up yet). Except when there is good reason to do otherwise, this pattern should be maintained.

At this point, no checked (or contingency) exceptions specific to the DRC have been defined. When they are needed, though, they will follow the same basic structure with a base exception derived from Exception.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/lib?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L65 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/changeset/699%23file3 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/changeset/699%23file2 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L116 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L101 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/lib/lib.properties?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/layout?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/drc-ds.xml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk?rev=691 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Description.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Datastream.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Item.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/utility?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/test?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions/DrcBaseAppException.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L140 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L132 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L92 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L63 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/PdfHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/ImageHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk?rev=691 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions/NoHandlerException.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/import.sql?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/layout/template.xhtml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/fedora?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/action?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/test-datastreams?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/Handler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/home.xhtml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://openrepositories.org/program/fedora#session4 on January 19th, 2011.

The text was modified to update a link from http://rama.grainger.uiuc.edu/assetActions/ to https://wiki.dlib.indiana.edu/display/DLFAquifer/Asset+Action+Project on January 19th, 2011.

The text was modified to update a link from http://dev2dev.bea.com/pub/a/2006/11/effective-exceptions.html to http://www.oracle.com/technetwork/articles/entarch/effective-exceptions-092345.html on January 20th, 2011.

Looking Forward to Version 2.2 of FEDORA

Peter Murray — Mon, 01 Jan 2007 01:46:39 +0000

Sandy Payette, Co-Director of the Fedora Project and Researcher in the Cornell Information Science department, announced a tentative date for the release 2.2 of the FEDORA digital object repository.

The Fedora development team would like to announce that Fedora 2.2 will be released on Friday, January 19, 2007.
This new release will contain many significant new features and enhancements, including [numbers added to the original for the sake of subsequent commentary]:
Fedora repository is now a web application (.war) that can be installed in any container
Fedora authentication has been refactored to use servlet filters (no longer Tomcat realms)
A new Fedora installer makes it easy to get started with Fedora (with both “quick” and “custom” install options)
GSearch service (backed by Lucene or Zebra) – flexible, configurable, indexes any datastream
Journaling service to create a backup/mirror repository
New checksum features for datastreams
Support for Postgres database configuration
Standard system logging with Log4J
Over 40 bug fixes
Many other enhancements
Be on the lookout for the release announcement the new year! Also, there will be opportunities to talk with the Fedora development team at Open Repositories 2007 (http://openrepositories.org/).

This is great news and a major step forward for the project. Here are some reasons why I think this is true.

1. Fedora repository is now a web application (.war)

To this point, the FEDORA repository application distribution has been pre-bundled inside a Tomcat Java servlet container. The binding has been pretty tight with certain dependencies written into the Tomcat configuration itself. That made it very difficult to install FEDORA into an organization’s existing servlet container (be it another installation of Tomcat or Jetty/JBoss/Glassfish, etc.). Even more problematic, there were reports of problems trying to get JSP-based applications to work inside the FEDORA-supplied container (we ran into this ourselves) meaning that organizations wanting to run both FEDORA and another servlet-based application needed to run two servlet containers; pretty inefficient. (OhioLINK was in this position in its early implementations of the Ohio DRC project.)

With release 2.2, the core developers have effectively turned the software distribution inside out. The primary output of the new build process is a standard Web ARchive (or WAR) file that can be put inside any servlet container. The new installation program (see #3 below) comes with a Tomcat distribution, should a new installation need it, but it is no longer required. There have been reports that the new WAR-based distribution works inside the Jetty servlet container; we’re hoping it will work in the JBoss Application Server as well (since that is what we’re using to build our next generation interface).

2. Fedora authentication has been refactored to use servlet filters

I’m not quite sure what this means, but I have hopes that it will make integration with Shibboleth easier. Can anyone else see the path between FEDORA and Shibboleth and comment on it?

3. A new Fedora installer makes it easy to get started with Fedora

From the start, FEDORA required a Java servlet container in order to run. To make the installation job easier for those that are not familiar with Java servlet containers, the FEDORA installation process did everything for you. Now that the relationship between the FEDORA application and the servlet container have been flipped around (see #1 above), the core developers devised an easy-to-use installation application that mimics the simplicity of the previous installation style while allowing others to make use of FEDORA as an integrated application within an existing servlet container.

4. GSearch service

The original FEDORA search service, the appropriately-named “basic search,” indexes only the Dublin Core (DC) datastream of each object. As has been mentioned on the Fedora-Users mailing list several times, the DC datastream is really meant as an administrative metadata datastream and not necessarily the full description of the object; that full description can be stored in other datastreams of a FEDORA object. Not only did basic search not index these other descriptive metadata streams, but it also wouldn’t index the full text of PDF, text, and other indexable datastreams.

GSearch — where “G” stands for “General” but could equally well stand for “Gert” Schmeltz Pedersen, its lead developer from the Technical University of Denmark — does all of the above as a new component in the FEDORA Service Framework. We extend our gratitude to Gert and his colleagues for contributing their work to the general FEDORA distribution as well as to DEFF, Denmark’s Electronic Research Library, which funded the GSearch project.

5. Journaling service

Like a journaling file system or a journaling database, this capability allows one to capture all of the transactions applied to the repository and replay them against a secondary repository instance or to restore a repository from backup.

6. Datastream checksums

As part of its ingestion and maintenance functions, the FEDORA software can now calculate, store, and verify checksums of datastreams. This helps ensure the integrity of the repository content, or at least detect when something goes wrong.

7. Support for PostgreSQL

In the battle between which relational database engine is best, FEDORA now supports most of the big ones out-of-the-box: Oracle, MySQL, and new PostgreSQL. Here at OhioLINK, we’ve started with MySQL but are considering a migration to PostgreSQL as our in-house, preferred RDBMS, so the timing of this announcement is great.

8. Standard system logging with Log4J

Put this one in the category of “playing nicely with others.” We’ve already reaped the benefit of the refactored logging code in the client JAR file in a pre-release version of the code.

9 and 10. Bug fixes and many other enhancements

The core code is evolving along a nice trajectory. This is good to see for the health of the overall project!

Version 2.2 represents another monumental step towards the vision of a Flexible, Extensible Digital Object Repository Architecture. Congratulations to the core developers for what sounds like is going to be a great release.

The text was modified to update a link from http://comm.nsdl.org/pipermail/fedora-users/2006-December/002330.html to http://article.gmane.org/gmane.comp.cms.fedora-commons.user/2330/ on January 19th, 2011.

The text was modified to remove a link to http://defxws2006.cvt.dk/fedoragsearch/ on January 19th, 2011.

Why FEDORA? Answers to the FEDORA Users Interview Survey

Peter Murray — Fri, 15 Sep 2006 22:48:40 +0000

The Fedora Outreach and Communications team is conducting a survey of the high-level sense of passion and commitment inherent in the Fedora community. I’ve posted some answers back to the FEDORA wiki on behalf of OhioLINK, and am also including the responses here as it fits into the “Why FEDORA?” series of blog postings. (If you are reading this through a RSS news reader, I think you’ll have to actually come to the DLTJ website and scroll down to the bottom of this post to see the table of contents of the series.) On with the responses!

How did you hear about Fedora?

I first remember hearing about FEDORA at a Coalition for Networked Information meeting in 2003. I only really remember it in passing because what was being presented was so radical that I didn’t appreciate what was being described.

I next encountered FEDORA during a conference call with the Internet2 Shibboleth core developers in mid-2004. The topic was enabling cross-repository access management — a topic that is still a challenge today (although the Shibboleth team is working on it). But that time I really started to catch on with what the FEDORA team was doing, and started paying closer attention.

Why did you chose Fedora?

When I arrived as project manager to the Ohio Digital Resource Commons (DRC) project in January 2005, OhioLINK was on the path to expand their existing Documentum installation to include a hosted institutional repository service. The Ohio DRC Steering Committee reviewed and accepted a proposal to use FEDORA as the foundation of this new hosted institutional repository service primarily because OhioLINK would be working with peers to develop the service (rather than working in isolation as likely would have happened with a Documentum-based solution).

Were there economic advantages to your project/org. in selecting Fedora?

The open source, free-to-license nature of FEDORA was definitely an advantage. It allowed us to turn grant funding that would have been used to pay for additional Documentum modules and licenses into to salary for temporary-hire programmers. In that way we felt that we had a better control over our destiny by creating the application code ourselves rather than relying on consultants.

What is Fedora’s unique role in your production system?

OhioLINK is beginning to look at the Service Oriented Architecture (SOAs) software design paradigm, and FEDORA fits right into that model as the content repository for all of our digital objects. If anything, FEDORA’s nature as a best-of-breed content repository — and nothing else — encourages us to think along the likes of sooner than we might otherwise have done.

Is there one specific Fedora attribute that enables your project/organization to accomplish your overall goals.

The fact that FEDORA is completely agnostic to what is contained in a datastream — be it audio, video, image, dataset, PDF, Dublin Core, MODS, EAD, FGDC, TEI, etc. — means that we can truly pursue a goal of managing all of our content in one place. The robustness of the content repository functions allows us to consider more interesting questions such as how this different content is ultimately presented to the end user.

Do you see yourself as an active member of the Fedora community? Why?

Yes. FEDORA represents the ability to take long-term control over the destiny of our digital objects. If, for some reason, the existing core developers at Cornell and UVa disappeared, a vibrant user community (OhioLINK included) can pick up the task of maintaining the software for the collective good. And if no one but OhioLINK is left in a “FEDORA community” our job of migrating out of it, should we desire to do so, is eased by the fact that we have the full view of the source code to help us move content and services to a new platform.

What would inspire you to become more involved?

It would take the existence of more hours in the day, I’m afraid!

What should be the mission of an ongoing Fedora organization?

A FEDORA community should first and foremost inspire communication among users of the FEDORA software. Almost all of us are working with extremely limited resources, and it weakens our collective effort if there is duplicated work underway. This communication should include not only developers but also users of the software.

The text was modified to remove a link to http://www.fedora.info/wiki/index.php/The_Fedora_Users_Interview_Survey on January 13th, 2011.

The text was modified to remove a link to http://www.fedora.info/wiki/index.php/Fedora_Outreach_User_Group on January 13th, 2011.

Previous in series

Best Practice Proposal for a DESCRIPTION Datastream

Peter Murray — Wed, 06 Sep 2006 19:58:42 +0000

OhioLINK is deep in the process of migrating content from our old Bulldog/Documentum-based system to, well, something else, and we’ve been talking about the treatment of the metadata in the course of the migration. I think it is safe to say that the Bulldog asset management system (and Documentum, which bought and integrated Bulldog into its product line about five years ago) is not really known for its rich handling of metadata. Or at least how the library community thinks of metadata: Dublin Core, MIX, MODS, MARC, VRA Core, PREMIS, FGCD, etc. — all at the same time in the same application engine with structured crosswalks between them. ¹ I think it is also safe to say that pure, unqualified Dublin Core, the only datastream that is required for every FEDORA object, does not completely encompass the descriptive fidelity needed for our objects. These observations, combined with reading a mid-term project report from the RepoMMan effort in the U.K., got me thinking about metadata and how we should store it in FEDORA objects. The outcome of that line of thinking is this proposal: “to establish a practice of creating an in-line XML datastream with the label ‘DESCRIPTION’ that contains the primary descriptive metadata for each object.”

Rationale

Although FEDORA mandates an unqualified Dublin Core datastream for every object, unqualified Dublin Core is not expressive enough to describe our objects. Therefore I recommend establishing this practice so subsequent agents/consumers of the objects (internal disseminators and external applications) will know the location of the most expressive metadata for the object.

Risks/Unknowns

FEDORA does not provide a mechanism to keep elements of the DESCRIPTION datastream in sync with the DC datastream. Do we store common data elements (e.g. “creator”) in both places? If so, our front-end applications would need to change the value of “creator” in two places and there is always the risk that they will get out of sync. How much real value is there in maintaining the FEDORA-mandated DC datastream?
There is no convention (that I know of) for a “primary descriptive metadata” datastream label in a FEDORA object, so “DESCRIPTION” is an arbitrary choice at this point. Future practices may go against this decision (although the choice does set us up to start using datastream labels like “PRESERVATION” for PREMIS metadata and so forth).

Background

In their “Experiences with Fedora” report, the RepoMMan team noted:

…working with Fedora’s compulsory Dublin Core (DC) datastream started one thinking about the metadata that a repository object would eventually need and how this might be mapped onto the Dublin Core fields. It was some considerable time later than an e-mail on the Fedora-users list made it clear that the inherent DC datastream was intended solely for Fedora’s internal use and not as the basis of external searches. ²

Even with our most simplest collection, we already know that unqualified Dublin Core will not be sufficient (most specifically, we had discussions about the lack of precision of “Date” and “Coverage” as compared to the field labels we already have in the Bulldog data dictionary). It is important that our metadata be parsable by machine processes, so I would advocate the proposed practice rather than trying to “shoe-horn” our descriptions into unqualified Dublin Core with text labels added the values and the like. And if we keep the machine parsable, we will have a wider variety of options for indexing the data and displaying it at the presentation layer.

The “in-line XML” part of this proposal means that the DESCRIPTION datastream would be “managed” by the FEDORA server (e.g. not external or referenced), so it would become part of the object in the content store.

Example

If we take for a moment what is displayed in the presentation layer for a sample object from the Forestry collection as the sum total of all of the descriptive metadata for an object of this collection, a corresponding DESCRIPTION datastream would look something like:
[xml]
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://drc.ohiolink.edu/schema/
http://drc.ohiolink.edu/schema/schema.xsd"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:dcterms="http://purl.org/dc/terms/">

Catalpa speciosa, bignonoides and Kampfera seeds.
Ohio Agricultural Experiment Station. Dept. of
Forestry.
Catalpa speciosa, bignonoides and Kampfera seeds.
Item #2
Ohio Agricultural Research and Development
Center
1908-12

2003-04-17T00:00:00

photographic prints
hdl:21151
2
Ohio
Copyright: Ohio State University

http://library.osu.edu/sites/dlib/terms.html

[/xml]

Comments?

Reactions to the proposal? A rational step forward, or is there a better way?

The text was modified to remove a link to http://worlddmc.ohiolink.edu/Science/Details?oid=4005859 on December 31st, 2010.

Footnotes

Reality check for those in the “library community” … do you think of metadata in this way?
Richard Green, “Experiences with Fedora during the project’s first year” Report D-D8, July 2006; page 8; retrieved 28-Aug-2006 from http://www.hull.ac.uk/esig/repomman/downloads/D-D8-fedora-exp-v10.pdf.

Previous in series Next in series

Analysis of CDL’s XTF textIndexer to Replace the Local Files with FEDORA Objects

Peter Murray — Tue, 22 Aug 2006 20:57:57 +0000

This is a continuation of the investigation about integrating the California Digital Library’s XTF software into the FEDORA digital object repository that started earlier. This analysis looks at the textIndexer module in particular, starting with an overview of how textIndexer works now with filesystem-based objects and ending with an outline of how this could with reading objects from a FEDORA repository instead.

XTF’s Native File System handler

Natively, XTF wants to read content out of the file system. The core of the processing is done in these two class files:

TextIndexer.java

The main() driver for ingesting content into the index. It reads commandline arguments (cfgInfo.readCmdLine( args, startArg );) to determine the various parameters, one of which is the top of the document source directory (String srcRootDir = Path.resolveRelOrAbs( xtfHomeFile, cfgInfo.indexInfo.sourcePath );). Assuming all goes well, it calls a method to open the Lucene index for writing, process files in the source directory, and close the Lucene index:
[java]
srcTreeProcessor.open( cfgInfo );
srcTreeProcessor.processDir( new File(srcRootDir), 0 );
srcTreeProcessor.close();
[/java]

SrcTreeProcessor.java

processDir() is called recursively on the directory structure to process files in that directory. For each directory, a docBuf XML-as-a-string buffer is consisting of an element for every directory entry. docBuf is fed into the SAXON processor along with the docSelector XSLT stylesheet. The resulting XML is read node-by-node looking for file entries that have an “indexFile” tag. For each matching node, it calls processFile() to index each entry.

processFile() will run the prefilter XSLT against the file content, build the Lazy Tree (if possible and requested), create the IndexSource version by running the source document through the appropriate file type “*IndexSource” method (e.g. PDFIndexSource(), XMLIndexSource, and MARCIndexSource()) and queue the content for indexing by the Lucene indexer.

Requirements for an Object Handler for textIndexer

Based on this analysis, if one were to replace the TextIndexer.java and SrcTreeProcessor.java “front end” of textIndexer, I think these would be the pieces that would be requried. (Note that some steps are skipped in this overview — any replacement of these two classes would need to be sure to do everything that those classes do now.)

Parse command line and configuration file parameters to create an IndexerConfig instance (guiding parameters for the indexer as a whole) and an IndexInfo instance (parameters specific to the identified index-name).
Specify a collection of objects that you want in index-name.
Open up a writable instance of the index-name’s Lucene index (a la srcTreeProcessor.open( cfgInfo );)
For each object to be put into index-name, do these things:
1. Optionally, run the source object through a prefilter (an XSLT transformation used to restructure the source document just prior to indexing without changing the stored source document).
2. Optionally, remove a DOCTYPE declaration in the source object before it is indexed.
3. Set up an transformation object from the native file format to something that is XML and call textProcessor.checkAndQueueText() to add it to a queue to be processed.
Close index-name’s Lucene index (a la srcTreeProcessor.close();), which should have the side effect of processing the queued text (a la textProcessor.processQueuedTexts();) which will ultimately create the Lazy Tree (if specified) and add the object to the Lucene index.
Optionally, compare the collection of objects that you want in index-name with what is actually in index-name before you started, and remove anything that wasn’t in the specified collection.

Considering a FEDORA-based XTF handler

So, all-in-all, that doesn’t seem too bad. Here is where we get to mix in some FEDORA pieces and see what we get in the end.

First off, in terms of dealing with “collections of source objects to be indexed” I think it would be best to have this start with one of our “collection aggregation” objects as the root level of a source collection. We’d perform an RDF “isMemberOf” query against the resource index using the FEDORA PID of the aggregation object (and optionally make an “isMemberOf” query recursively against the returned set — as if one was drilling down a file system).

Secondly, to get the XML content to be indexed, each object would have a getXML disseminator (see Thinking about Our FEDORA Disseminators for background) that would render to XTF an XML version of itself. If the source object is an XML-based object, it just returns the XML. If the source object is a PDF or Word document or something that can be rendered into a text-like form, the disseminator would handle that. If the source object is an image or audio clip, the disseminator can return the descriptive XML of the object. The point being, though, by the time the object gets to XTF’s textIndexer, it has already be rendered to XML, so just the XML transformation tool would be needed (as in this snipped from SrcTreeProcessor.java):
[java]
IndexSource srcFile = null;
if( format.equalsIgnoreCase(“XML”) ) {
InputSource finalSrc = new InputSource( systemId );
srcFile = new XMLIndexSource( finalSrc, srcPath, key,
preFilters, displayStyle, lazyStore );
if( removeDoctypeDecl )
((XMLIndexSource)srcFile).removeDoctypeDecl( true );
}
[/java]

Third, a FEDORA-aware driver that replaces TextIndexer.java and SrcTreeProcessor.java. Given a configuration file location and a starting PID, it would gather the objects to be indexed, “open” the Lucene index, run through the snippet of Java above for each object, and “close” the Lucene index.

The quick-and-dirty first implementation would copy the XML source to a directory on the hard drive (directory and subdirectory names would be the PID of the aggregation object containing the collection of objects), and have XTF use that local filesystem copy as the indexed source. Lazy Tree files for each object would also be created and stored locally. This means we have two copies (three, if you count the Lazy Tree) of the object laying around, so eventually I think we’d want to modify XTF to pull content directly from FEDORA using a REST-based URL. Eventually I think we may also want to store the Lazy Tree in something other than the local file system. Could that be another datastream in the FEDORA object?

The text was modified to remove a link to http://texts-stage.cdlib.org/xtf/javadoc/org/cdlib/xtf/textIndexer/IndexerConfig.html on December 31st, 2010.

The text was modified to remove a link to http://texts-stage.cdlib.org/xtf/javadoc/org/cdlib/xtf/textIndexer/IndexInfo.html on December 31st, 2010.

Previous in series Next in series

CDL’s XTF as a Front End to Fedora

Peter Murray — Tue, 22 Aug 2006 13:29:09 +0000

We’re experimenting pretty heavily now with the California Digital Library‘s XTF framework as a front-end to a FEDORA object repository. Initial efforts look promising — thanks go out to Brian Tingle and Kirk Hastings of CDL; Jeff Cousens, Steve DiDomenico, and Bill Parod from Northwestern; and Ross Wayland from UVa for helping us along in the right direction.

XTF into Eclipse How-To

As we get more serious about XTF, I wrote up a How-To document for bringing XTF into Eclipse so that it can be deployed as a dynamic web application. Let me know if you find it useful. Definitely let me know if you find it in error. We haven’t put a version of XTF into OhioLINK’s source code repository, but that might follow shortly.

Points of Integration

In its base configuration, XTF reads documents out of a “data” directory that is in the application’s Tomcat context directory. It looks like two of the XTF components will need to be modified to successfully converse with a FEDORA-based object repository: DynaXML and textIndexer. Of the two, DynaXML seems to be the most straight forward.

DynaXML

First I went looking for where XTF’s DynaXML reads documents and found the DocLocator interface with one implementation that looks into the file system. John Davison, one of the DRC programmers, figured out (with help from the CDL folks) that in fact it is possible to pass a FEDORA API-A URL to DefaultDocLocator and have it do the right thing. Its ‘getInputSource()’ method has this signature:

[java]
public InputSource getInputSource( String sourcePath,
boolean removeDoctypeDecl ) throws IOException
[/java]
…followed shortly by:

[java]
// If it’s non-local, load the URL.
if( sourcePath.startsWith(“http:”) ||
sourcePath.startsWith(“https:”) )
{
return new InputSource( sourcePath );
}
[/java]
where “InputSource” is the entry point into the SAX parser, which will accept a URI as a parameter.

Unfortunately, using DefaultDocLocator in this way negates the use of CDL’s “Lazy Trees” (a binary version of each XML document containing all the original contents of the document, plus an index telling XTF where each element starts and ends). Lazy Trees are a good thing because they speed up parsing of the XML document and the resulting rendering to the user.

When dealing with local files (as opposed to the URL method described above), DefaultDocLocator will build a Lazy Tree in its index directory the first time the XML document is called up. In implementing a FEDORA interface for XTF’s DynaXML, what is required is a mixture of URL (or, in the case of FEDORA, a PID plus API-A call) to get the document and then create/store its lazy tree in the XTF index directory for subsequent retrieval. This does seem pretty straight forward, does it not?

textIndexer

XTF’s textIndexer, on the other hand, really wants the XML it is indexing to be files on the local hard drive. The XTF programming guide speaks of a textIndexer Document Selector whose job it is to create a single XML file with the specifications of which documents to index and how to do it:

It is the responsibility of the Document Selector XSLT code to output an XML fragment that identifies which of the files in the directory should be indexed. This output XML fragment should take the following form:
[xml]

{format = "FileFormatID"}
{preFilter = "PreFilterPath"}
{displayStyle = "DocumentFormatterPath"}>

[/xml]

Now the trick seems to be to build an alternate Document Selector that will not use filenames but rather URIs to build the index. That’ll be the subject of the next round of investigations.

Comments and observations are welcome!

The text was modified to remove a link to http://drc-dev.ohiolink.edu/wiki/EclipseXTFHowTo on December 31st, 2010.

The text was modified to remove a link to http://xtf.sourceforge.net/WebDocs/HTML/XTF_Under_Hood/XTFUnderHood.html#LazyFiles on December 31st, 2010.

The text was modified to remove a link to http://xtf.sourceforge.net/WebDocs/HTML/XTF_Programming_Guide/XTFProgGuide.html#textIndexer_DocSelector_Prog on December 31st, 2010.

The text was modified to update a link from http://xtf.cvs.sourceforge.net/xtf/xtf/WEB-INF/src/org/cdlib/xtf/dynaXML/DocLocator.java?revision=1.5&view=markup to http://xtf.hg.sourceforge.net/hgweb/xtf/xtf/file/549e4167039e/WEB-INF/src/org/cdlib/xtf/dynaXML/DocLocator.java on January 28th, 2011.

The text was modified to update a link from http://xtf.cvs.sourceforge.net/xtf/xtf/WEB-INF/src/org/cdlib/xtf/dynaXML/DefaultDocLocator.java?revision=1.10&view=markup to http://xtf.hg.sourceforge.net/hgweb/xtf/xtf/file/de7d8a406bef/WEB-INF/src/org/cdlib/xtf/dynaXML/DefaultDocLocator.java on January 28th, 2011.

Next in series

Representing Collections In FEDORA

Peter Murray — Tue, 25 Jul 2006 19:28:44 +0000

One of the DRC developers had a question recently that sparked a discussion about what to do with collections of objects. In order to answer the question of how to represent the notion of a collection within the repository, we’re going to have to get pretty heavy into RDF: the Resource Description Framework. RDF is a language created by the Worldwide Web Consortium “for representing information about resources in the World Wide Web.” If you already know about RDF — or just want to see what a proposed solution is — you can skip down to the “RDF for Collections in FEDORA” heading.

At the preface, I have to say that I’m increasingly uncomfortable with the word “collection” because it has become so overloaded in library usage, and like Carl Lagoze prefer the term “aggregation” to describe in a general sense what we think a collection is and what it could become. I probably bounce back and forth between the terms here, but am aiming to use “aggregation” and “aggregation object” more often.

I’m going to be pulling a lot of examples and language from the “RDF Primer“, which I would recommend reading at some point. It is a very long, dense document, but if you can get through it you’ll have a very good understanding of what RDF is and what is does for us.

The Primer describes RDF this way: “It is particularly intended for representing metadata about Web resources, such as the title, author, and modification date of a Web page, copyright and licensing information about a Web document, or the availability schedule for some shared resource…. RDF is based on the idea that the things being described have properties which have values, and that resources can be described by making statements … that specify those properties and values.”

There are three parts to an RDF statement about an object. “[T]he part that identifies the thing the statement is about (the Web page in this example) is called the subject. The part that identifies the property or characteristic of the subject that the statement specifies (creator, creation-date, or language in these examples) is called the predicate, and the part that identifies the value of that property is called the object.”

These component make up what is called an “RDF triple.” When written in tabular form an RDF triple is conventionally written in the order subject, predicate, object. To represent RDF statements in a machine-processable way, RDF uses the Extensible Markup Language [XML]. RDF defines a specific XML markup language, referred to as RDF/XML, for use in representing RDF information, and for exchanging it between machines.

For instance, imagine trying to state the (nominally, Dublin Core) descriptive metadata about a web page called http://www.example.org/index.html. In natural language, the descriptive elements could be:

http://www.example.org/index.html has a creator whose value is John Smith
http://www.example.org/index.html has a creation-date whose value is Aug 16, 1999
http://www.example.org/index.html has a language whose value is English

In tabular form, this could look like:

SubjectPredicateObjecthttp://www.example.org/index.htmlcreatorJohn Smithhttp://www.example.org/index.htmlcreation-dateAug 16, 1999http://www.example.org/index.htmllanguageEnglish

In XML, this could look like:

          John Smith     Aug 16, 1999     English

Keep in mind, though, that we expressed the predicate here as Dublin Core; the predicate can be anything — even something you make up!

RDF for Collections in FEDORA

RDF is used throughout FEDORA — in fact, the Dublin Core properties can (and in our FEDORA configuration, are) expressed as RDF triples in an internal database and can be searched as such. But the RDF triples can be used to express more than just attributes about an object — it can be used to express /relationships/ between objects. There is a whole section of the FEDORA docs called “Fedora Digital Object Relationships” that goes into more detail. Quotations and examples in this section are drawn from that document.

“Fedora digital objects can be related to other Fedora objects in many ways. For example there may be a Fedora object that represents a collection and other objects that are members of that collection. Also, it may be the case that one object is considered a part of another object, a derivation of another object, a description of another object, or even equivalent to another object.”

FEDORA comes with a list of common relationships between objects, and other community or user-defined relationships may also be asserted. These relationships can be expressed in RDF notation:

SubjectPredicateObjectMyCatVideois a member of collectionGreatCatVideosdrc:101isMemberOfdrc:100drc:101isFromInstitutionmu3ug

                 mu3ug

“drc:100″ is an aggregation object (otherwise known as a “collection object”, but I’ve learned from others in the FEDORA community that “collection” is too loaded of a word) of which “drc:101″ is a member. To put it in terms that we may be familiar with:

drc:100 is the aggregation object for the “Charles E. Frohman Collection”
drc:101 is a digital image of a photograph with the title “Work Crew” that is part of the Charles E. Frohman collection
drc:101 is a digital image contributed by member institution “mu3ug”

Conclusions

So the issue becomes, I believe, to examine the pre-loaded set of relationships to match those against the existing relationships in the DMC and then do define any kind of unique relationships (such as “isFromInstitution”) that we would want to express about our objects.

Previous in series Next in series