As a part of work for an OhioLINK strategic task force, I have been exploring the creation and operation of regional/collaborative/shared digitization centers. This is a report of findings to date after an open call for information. The report is structured with questions to be explored when considering a regional digitization center followed by narratives from conversations with the Collaborative Digitization Program (formerly the Colorado Digitization Program), the Mountain West Digital Library, and the Ohio Historical Society. My thanks go out to Leigh Grinstead, Liz Bishoff, Karen Estlund, Angela O’Neal, and Phil Sager for their assistance.

I am still interested in talking with collaboratives about similar programs, both “on the record” and in private conversations. Please get in touch with me if you would like to chat.

Questions for Exploration

1. Who does the digital conversion? Is it staff at the hosting institution (where the equipment is located), or does the institution contributing the materials perform the scanning operation at the host institution?
2. Where and when is the description done? Description of the digitized item was universally done by the contributing institution, but the location (at the scanning center or at the contributing institution) and timing (before conversion, during conversion, or after conversion) vary. Answers to the first question — whether contributing staff members are performing the conversion — affect this question.
3. Is contribution of materials to the Ohio Digital Resource Commons required? If the regional scanning center is used to digitize materials (or in the case where the consortium is subsidizing a scanning service to perform the digital conversion), what conditions are put in place for contributing those materials to the central repository.
4. What is the cost structure for use of the regional scanning center? Options range from complete subsidy by consortium (most notably when the CDP funded the equipment at their regional scanning centers) to contributing institutions being charged at cost recovery rates by hosting institutions. There can be various cost structures applied, such as a consortial subsidy for equipment and training with labor supplied by contributing institution or contracted from hosting institution.
5. What training is offered? Topics for training range from optimal use of digitization equipment to digitization project planning to metadata creation standards. The training can be based on group instruction, one-on-one consultation with contributing institutions, or a combination of both.
6. Who will fill the role of “metadata editor”? The need for a collective expert in metadata creation was found in many of the projects. This person is typically charged with training contributing staff on the appropriate use of local metadata conventions, coaching individual staff on particular projects, and reviewing records that will become part of a consortial database.

Report from the CDP

The original climate that generated the interest in scanners was one where institutions did not want to outsource digitization because they didn’t want the materials leaving their direct control. The CDP, on the other hand, did not want institutions to buy cheap scanners that would result in lower-quality scans. The CDP purchased the scanners ($2,500 each, in 1999 dollars) plus desktop workstations and created a training program for the use of the equipment. The center provided the location for the equipment and hands-on assistance with using the equipment; it was incumbent on the staff at the institution with materials to be digitized to perform the scanning themselves. Before using the equipment, those performing the scanning had to attend a CDP “Introduction to Digital Imaging” training session on the proper use and techniques for obtaining high-quality images. Scanning projects that received CDP funding had priority over other users of the equipment, but there was little contention for the equipment at the centers. Descriptive metadata could be keyed at the time of the scan; CDP provided a web-based template (called “DC builder”) or staff could use their own system (an ILS, ContentDM, etc.). Contribution of the associated metadata to the CDP union catalog of metadata — Heritage West — was required for CDP-funded grant projects. Images were locally hosted.

In 2003, the CDP conducted an evaluation of the scan centers in the form of focus groups. They found two different responses. First, institutions on the western slopes of Colorado made much heavier use of their scan centers. These institutions tended to be smaller and/or economically disadvantaged, and the availability of the hardware and software to conduct the scanning operations was more critical. In front range libraries, where institutions tend to be better-funded, the centers were not as heavily used for projects and tended to be used for training and demonstration sites; these participants felt that the greatest value in the CDP came from the professional networking and training opportunities the locations, the grants that the CDP provided and the creation of the best practices and website that brought all this together.

The regional scan centers are not used today. The primary reason is the diffusion of experience within the community that was spurred by the early success of the centers. As funding cycles continued, institutions purchased their own equipment (generally replicating the equipment at the centers) so as avoid the need to transport materials and staff to a regional center for larger projects. The focus of grant funding within CDP also shifted from image-based collections to EAD and sound collections. There is still an “Introduction to Digital Imaging” workshop, but the focus is now on requirements for in-house equipment and/or what to seek from a vendor in an RFP. The imaging workshop is typically taught as part of a three-day workshop series with an “Introduction to Digital Project Management” (storage, preservation, handling socially sensitive materials, how to display them) before and “Introduction to Metadata” (focused on Dublin Core) after. The biggest issue facing the consortium now is oversized scanning; only the very largest university library would have this kind of equipment.

The Colorado model of teaching staff at the institution on the use of the equipment along with the creation of regional scan centers was picked up by several states: North Carolina; Alabama; Kansas; Missouri; and Wyoming (except that regional centers were not practical due to the wide population dispersion). Tennessee is working under a current IMLS grant to build three regional center plus a suite of mobile equipment.

Liz suggests that we should look for ways to support collaborative efforts within the institution with museum and archives on the campus. The Florida Center for Library Automation (FCLA) has a program in place where the local university library is the contact in partnerships with public libraries and local museums.

Leigh noted that in recent years, CDP projects include a “metadata editor” that is hired to look at record quality and work with individual institutions to improve records. Having a metadata editor, someone familiar with the CDP guidelines and the field in general, look at four or five records at the very early stage of a project is critical to the success of the quality of the rest of the project. In a recent project, nearly half of the partners were going to export records out of a local collection management system. It was discovered that the records were not consistent; local implementation/practice of cataloging standards has a higher overall impact than community best practice. Having a local cataloger participate on the team migrating the records from a local system to a central system is key to success.

Mountain West Digital Library

Content is ingested into the MWDL either through digital conversion centers at the regional institutions or through the efforts of the contributing institution. The regional institutions perform digital conversion for contributing institutions at cost-recovery rates. These regional centers use the equipment and staff at the hosting institution, and are very busy at times resulting in difficult choices to balance needs of the host institution with that of requests from other institutions.

The regional centers also conduct on-site training and technical education at contributing institutions about best practice for digital conversion. The on-site program includes a technical evaluation of the equipment to be used to ensure that it can produce conversions that meet the minimum requirements for the MWDL. In the course of this evaluation, staff at contributing institutions are taught some technical aspects of scanning such as the actual DPI scanning capabilities of the hardware versus interpolated resolutions (and why this is important). Staff also receive an introduction to Photoshop for de-skewing and other standard practices. They also are instructed in the Western Standard Metadata Best Practices. This hands-on approach — with the contributing institution’s people, equipment and materials — enables strong connections between the contributing institution and the hosting institution. It makes the contributing institution feel like a part of the larger program. Most contributing institutions with local digital conversion operations will FTP files to the ContentDM server at the regional institution; some institutions contribute materials through the transportation of a portable hard drive.

Each regional hosting center is responsible for the digital preservation of the material on their server. At this point there is no common agreement across the MWDL for standards on digital preservation; this is an area of work to be addressed in the near future by the cooperative.

The cooperative is starting work in several new areas. First is the digital conversion and posting of oral histories from contributing institutions. While it is anticipated that such resource will be highly valued, quick progress on this project is hampered by the same permission problems that face similar projects with relatively old audio. The cooperative also has a state LSTA grant for a Utah institutional repository with a portal connected to the MWDL. The state archives are also beginning a program to post state government documents, including items related to the Olympic Games held in Utah. The hands-on approach to training staff at contributing institutions is very labor-intensive and uneven across the project participants. MWDL is in the process of hiring a program director, and a component of that job description is to provide this kind of training across the project.

Ohio Memory Project (Ohio Historical Society)

OHS staff will work with contributing institutions on a one-on-one basis as well as conduct workshops on introduction to scanning. Many of the participants are from contributing institutions that have had staff changes and the new staff want to know how to use the equipment purchased during the grant-funded phase of the project. There is also a desire for more advanced training, such as Photoshop basics.

OHS staff recognize that a compromise is needed between institutional capabilities for scanning and the very best practices in the field. They have found it hard to establish and mandate an absolute standard for the parameters and quality of digital scans. Institutions that scan their own materials are responsible for their own storage and preservation.

This year has seen the release of two personal repository services: http://PublicationsList.org/ and the U.K. Depot. These two services have an admittedly different focus, but I think it is still interesting to compare and contrast them to see what we can learn.

The Depot

The deposit interface, for those putting content directly into the centralized Depot repository, has four main stages. First, the “Type” stage, specifying whether the object is an article, a book chapter, or a conference paper:

Next, the “Upload” stage, where one can upload the file and supply a few more properties:

Then the “Details” stage, where the descriptive metadata (minus the controlled vocabulary subjects — that comes in the next screen) is input:

And finally, the “Subjects” page, with an AJAX-driven expanding-and-collapsing hierarchy of subjects:

To retrieve contents from the repository, there is a “browse” interface for looking by ‘year’ or by ‘subject’ — no other browse facets and no search interface. The Depot was just formally released this month, so I would bet that functionality like that is in the works.

PublicationsList.org

The single item entry page is a just-the-facts interface. Note that the content hosting service is only available to those who have upgraded to the “Publications List Professional” version (which costs £9.99, or approx $20/€15, per year).

The system can also accept a variety of citation manager file formats for bulk entry. (See snapshot to the right.) PublicationsList also has a built in search-and-select interface to PubMed for finding publications matching your name and automatically populating the metadata fields in your personal citation.

Then end result is a web-based bibliography with links to the publications (either hosted on PublicationsList or on other sites). The free version is hosted on PublicationsList.org (see the service founder’s page as an example) and the professional version can embed the publications list in your own page.

PublicationsList does provide discounts and additional functionality for groups (such as departments, research centers, etc.).

Observations

The one-stop, redirection service in The Depot is a good concept, too. If a researcher wanted to deposit their content in a repository and they weren’t sure if their institution had a repository to hold it, OhioLINK would be a natural place to look for a content hosting service in the state and we could redirect the author to the appropriate location on a campus. OhioLINK could also be playing the role of repository-of-last-resort for Ohio academic researchers by providing a space and services for published content, whether or not the institution in question has set up a formal repository space on the DRC.

Footnotes

1. For a really good discussion of that problem, see Davis, P.M., & Connolly, M.J.L. (2007). Institutional Repositories: Evaluating the Reasons for Non-use of Cornell University’s Installation of DSpace. D-Lib Magazine, 13(3/4). Retrieved March 14, 2007, from http://www.dlib.org/dlib/march07/davis/03davis.html.

OhioLINK is engaged in building a “trusted digital repository” on behalf of its membership. As we build it, we want to have an understanding of what “trusted” means, and so we are engaging in an audit process to assess whether we can claim to be trustworthy. This process is panning out to have four major phases:

1. Research common and best practices for preservation.
2. Evaluate the OhioLINK policies and processes against common and best practices.
3. Perform a gap analysis between where we are now and where common and best practices suggest we should be.
4. Propose and adopt policies and processes that get us closer to the ideal common and best practices.

This is a report at the end of phase 1. Earlier this year, two major reports were released that address how one measures a “trustworthy repository.” The two reports are summarized below, followed by a recommendation.

Trustworthy Repositories Audit & Certification: Criteria and Checklist

The nestor working group says a trusted, “long-term digital repository is a complex and interrelated system” (nestor 2006). However, more than just the “digital preservation system” drives the management of the digital materials. In determining trustworthiness, one must look at the entire system in which the digital information is managed, including the organization running the repository: its governance; organizational structure and staffing; policies and procedures; financial fitness and sustainability; the contracts, licenses, and liabilities under which it must operate; and trusted inheritors of data, as applicable. Additionally, the digital object management practices, technological infrastructure, and data security in place must be reasonable and adequate to fulfill the mission and commitments of the repository.

A trusted digital repository will understand threats to and risks within its systems. As articulated by Rosenthal et al. (2005), these potential threats include media failure, hardware failure, software failure, communication errors, failure of network services, media and hardware obsolescence, software obsolescence, operator error, natural disaster, external attack, internal attack, economic failure, and organizational failure. Constant monitoring, planning, and maintenance, as well as conscious actions and strategy implementation will be required of repositories to carry out their mission of digital preservation. All of these present an expensive, complex undertaking that depositors, stakeholders, funders, the designated community(ies), and other digital repositories will need to rely on in the greater collaborative digital preservation environment that is required to preserve the vast amounts of digital information generated now and into the future.

The TRAC:CC contains 84 criteria broken out into three main sections: Organizational infrastructure; Digital object management; and Technologies, technical infrastructure, and security. Within each of these sections are various subsections and under the subsections are the criteria themselves.

1. Organizational infrastructure
   1. Governance & organizational viability
   2. Organizational structure & staffing
   3. Procedural accountability & policy framework
   4. Financial sustainability
   5. Contracts, licenses, & liabilities
2. Digital object management
   1. Ingest: acquisition of content
   2. Ingest: creation of the archivable package
   3. Preservation planning
   4. Archival storage & preservation/maintenance of AIPs
   5. Information management
   6. Access management
3. Technologies, technical infrastructure, and security
   1. System infrastructure
   2. Appropriate technologies
   3. Security

Some sample criteria are:

• A1.1 Repository has a mission statement that reflects a commitment to the long-term retention of, management of, and access to digital information.
• A2.2 Repository has the appropriate number of staff to support all functions and services.
• A3.5 Repository has policies and procedures to ensure that feedback from producers and users is sought and addressed over time.
• A5.4 Repository tracks and manages intellectual property rights and restrictions on use of repository content as required by deposit agreement, contract, or license.
• B2.5 Repository has and uses a naming convention that generates visible, persistent, unique identifiers for all archived objects (i.e., AIPs).
• B2.9 Repository acquires preservation metadata (i.e., PDI) for its associated Content Information.
• B3.4 Repository can provide evidence of the effectiveness of its preservation planning.
• B4.4 Repository actively monitors integrity of archival objects (i.e., AIPs).
• B5.3 Repository can demonstrate that referential integrity is created between all archived objects (i.e., AIPs) and associated descriptive information.
• C1.1 Repository functions on well-supported operating systems and other core
• infrastructural software.
• C1.5 Repository has effective mechanisms to detect bit corruption or loss.
• C1.7 Repository has defined processes for storage media and/or hardware change (e.g., refreshing, migration).
• C1.9 Repository has a process for testing the effect of critical changes to the system.
• C3.3 Repository staff have delineated roles, responsibilities, and authorizations related to implementing changes within the system.

TRAC:CC states that the “criteria are written to be applicable to any kind of digital repository or archives.” As such, criteria should be placed within the context of vision and goals of the Ohio DRC. One demonstrates compliance with the criteria through documentation (evidence), transparency (open examination of the evidence), adequacy (degree to which the evidence meets the vision/goals), and measurability.

Digital Repository Audit Method Based On Risk Assessment

The process has six stages, some with multiple tasks:

1. Identify organizational context
   • Specify mandate of your repository or the organization in which it is embedded
   • List goals and objectives of your repository
2. Document policy and regulatory framework
   • List your repository’s strategic planning documents
   • List the legal, regulatory, and contractual frameworks or agreements to which your repository is subject
   • List the voluntary codes to which your repository has agreed to adhere
   • List any other documents and principles with which your repository complies
3. Identify activities, assets and their owners
   • Identify your repository’s activities, assets and their owners
4. Identify risks
   • Identify risks associated with activities and assets of your repository
5. Assess risks
   • Assess the identified risks
6. Manage risks
   • Manage the risks

The report includes a catalog of risks taken from other checklists and repository audits that can be used to spur the thinking of the auditor.

Recommendation

I’ve been working to get JBoss Seam tied into Fedora, and along the way thought it would be wise to stop and document a core concept of this integration: the centrality of Fedora Disseminators in the the design of the Ohio Digital Resource Commons. Although there is nothing specific to JBoss Seam (a Java Enterprise Edition application framework) in these concepts, making an object “render itself” does make the Seam-based interface application easier to code and understand. A disseminator-centric architecture also allows us to put our code investment where it matters the most — in the repository framework — and exploit that investment in many places. So what does it mean to have a disseminator-centric architecture and have objects “render themselves”?

How It Works

• Browser: The user’s browser
• DRCseam: A JBoss Seam application that generates the user interface and performs much of the business logic. DRCseam, however, does not render the objects or their metadata into browser-consumable artifacts. Read on!
• Fedora: A basic Fedora digital object repository.
• Disseminator: A simple servlet that performs various transformations on object datastreams to render content usable by the browser.

With these components in play, here is the description of a sequence to render a page showing the metadata for a repository item:

1. request item page: The browser follows a link to an item detail page.
2. API-A ObjectProfile: The interface application asks the repository for the ‘Object Profile’ of the item…
3. return object profile: …which the repository returns. The interface application now knows basic details about the object: that it exists, the creation and updated timestamps, and so forth.
4. API-A DatastreamDissemination for fullDisplay: The interface application needs the object’s metadata display, so it asks the object to “render itself” by making a call to the Fedora repository for the object’s “FullDisplay” disseminator.
5. call getFullDisplay: The Fedora repository in turn calls the object’s disseminator with the Persistent Identifier (PID) of the object as a parameter.
6. API-A Datastream for metadata: Using the object PID, the disseminator calls back to the Fedora repository for the descriptive metadata datastream (the DC datastream, in this case)…
7. XML metadata: …which the Fedora repository returns.
8. transform metadata: The disseminator performs some transformation or derivation on the descriptive datastream to create an XHTML representation…
9. XHTML fragment: …which it returns to the Fedora software…
10. XHTML fragment: …which is returned to the interface application…
11. XHTML page: …which inserts it at the appropriate place in the XHTML page it has built and returns the XHTML page to the browser.

Step #4 is where we diverge from previous architectures. Instead of making the interface application transform the metadata into a human-readable format, the interface application calls the object’s disseminator to do the job.

The Heart of It All: The Disseminator

Rather than putting special case code in the interface application to render the description of the journal article one way and the thesis another way, that special case code is bound to the object in the form of a “disseminator”. The disseminator methods for the journal article and the thesis share the same name — getFullDisplay — but will return entirely different XHTML fragments — one for a journal article and one for a thesis. For both objects, though, the interface application will make a call to the object in the Fedora repository asking for the output of each getFullDisplay dissemination. In the case of a Dublin Core description, the dissemination output could look like this:

<table class="drc_dublinCore_table">
<tr class="drc_dublinCore_row drc_dublinCore_title">
<td class="drc_dublinCore_label drc_dublinCore_title">Title:</td>
<td class="drc_dublinCore_value drc_dublinCore_title">Jester Example</td>
</tr>
<tr class="drc_dublinCore_row drc_dublinCore_identifier">
<td class="drc_dublinCore_label drc_dublinCore_identifier">Identifier:</td>
<td class="drc_dublinCore_value drc_dublinCore_identifier">demo:exampleObject</td>
</tr>
</table>

You’ll note that there is a liberal application of CSS styles on all of the XHTML elements, allowing for the look of the dissemination to be further transformed in the browser via CSS stylesheets. A getFullDisplay dissemination for a journal article could look like this:

<table class="drc_ejc_table">
<tr class="drc_ejc_row drc_ejc_title">
<td class="drc_ejc_label drc_ejc_title">Article Title:</td>
<td class="drc_ejc_value drc_ejc_title">Taking Advantage of Fedora Disseminations</td>
</tr>
<tr class="drc_ejc_row drc_ejc_volume">
<td class="drc_ejc_label drc_ejc_volume">Volume:</td>
<td class="drc_ejc_value drc_ejc_volume">3</td>
</tr>
<tr class="drc_ejc_row drc_ejc_issue">
<td class="drc_ejc_label drc_ejc_issue">Issue:</td>
<td class="drc_ejc_value drc_ejc_issue">2</td>
</tr>
</table>

Looking at the Pieces

• http://drc-dev.ohiolink.edu:8080/BaseDisseminator/getFullDisplay/demo:exampleObject

Next, how this disseminator looks as accessed through the Fedora repository:

• http://drc-dev.ohiolink.edu:8080/fedora/get/demo:exampleObject/demo:bDefExample/getFullDisplay/

And finally, how this result looks through the Seam-based interface application. (A note about this application — only this URL works at the moment even though there are other links on the page. This is also the ‘trunk’ version of our interface code, so it is likely to change and/or break and/or work better at any time.)

• http://drc-dev.ohiolink.edu:8080/drc/item.seam?itemId=demo%3AexampleObject

Fedora Setup

Disseminator Behavior Definition (bDef)

Under the “Abstract Methods” pane, create new definitions for each of the disseminator methods.

Under the “Documentation” pane, put something in the first entry. Again, it doesn’t matter what is put in for these values, but they are required.

Select “Ingest” at the bottom of the window, and the demo:bDefExample bDef will be created. Alternatively, you could import the demo:bDefExample saved in the DRC source code repository (choose “original format” at the bottom of that page).

Disseminator Mechanism Definition (bMech)

In the “Service Profile” pane, put in values in the “General” area (it doesn’t matter what). In the Service Binding area, make sure the Message Protocol is HTTP GET, put in text/html, text/xml for Input MIME Types and put in text/html, text/xml, text/plain for Output MIME Types.

Under the Service Methods pane, put in http://localhost:8080/BaseDisseminator for the Base URL. (The disseminator is also loaded in the same servlet as the Fedora repository and the Seam interface application, and it is loaded at the “/BaseDisseminator” context path in the servlet.) Create Service Method Definitions that correspond to the Abstract Methods in the bDef.

Select “Properties” for each one of the Service Method Definitions in turn. “echo” is a unique disseminator method that simply echos back the context parameters of the disseminator request. This is useful for seeing exactly what the Fedora server is going to give to the disseminator.

With the exception of “echo” all of the other Service Method Definitions are the same. The Method Binding consists of the disseminator method followed by a slash and the PID placeholder followed by a question mark and ‘dc’ equals the DC placeholder. Since the Method Binding field has two placeholders, there are two entries in the Method Parameter Definitions area. The first is for PID — a “Default” parameter that is required and passed by value to the disseminator. The default value is the special value $PID, which the repository software will replace with the PID of the object as the disseminator is called. The second is for DC, a “Datastream” parameter that is required and passed to the disseminator by URL reference. The disseminator doesn’t actually use this reference to a datastream, but it is a requirement that all bMechs pass a datastream of one sort or another to the disseminator.

If you have followed all of the steps so far, under the “Datastream Input” pane there will be one entry for DC in the table. The only thing that needs to be done here is adding “text/xml” in the MIMEType column.

Under the “Documentation” pane, put something in the first entry. Again, it doesn’t matter what is put in for these values, but they are required.

Select “Ingest” at the bottom of the window, and the demo:bMechExample bMech will be created. Alternatively, you could import the demo:bMechExample saved in the DRC source code repository (choose “original format” at the bottom of that page).

Sample Object

Select “Save Changes” at the bottom. The completed disseminator looks like this:

There is a sample object in the DRC source code repository that has the disseminator already defined.

Notes

Future Directions

On a related front, this style of implementation would be greatly enhanced by the work of the Fedora Content Model Dissemination Architecture (CMDA). Because disseminators must be bound to specific objects rather than classes of objects, management of the variety of bMechs in a scenario such as this will likely become difficult very soon. I’m heartened by the fact that the CMDA work is going on and will cut our management complexity dramatically when it becomes available.

Acknowlegements

[Update 20070426T1147 : Fixed the sample object URL. Thanks, Jodi.]

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_exampleObject.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_bMechExample.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk/resources/foxml/demo_bDefExample.xml?rev=774 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/drc/item.seam?itemId=demo%3AexampleObject on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/fedora/get/demo:exampleObject/demo:bDefExample/getFullDisplay/ on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu:8080/BaseDisseminator/getFullDisplay/demo:exampleObject on January 13th, 2011.

The text was modified to update a link from http://rama.grainger.uiuc.edu/assetActions/ to https://wiki.dlib.indiana.edu/display/DLFAquifer/Asset+Action+Project on January 19th, 2011.

This tour is designed to show the overall architecture of a FEDORA digital object repository application within the JBoss Seam framework while at the same time pointing out individual design decisions and extension points that are specific to the Ohio Digital Resource Commons application. Geared towards software developers, a familiarity with Java Servlet programming is assumed, although not required. Knowledge of JBoss Seam, Hibernate/Java Persistence API, EJB3 and Java EE would be helpful but not required; brief explanations of core concepts of these technologies are included in this tour.

The tour is based on revision 709 of /drc/trunk and was last updated on 18-Jan-2007.

This tour will also be incorporated into a presentation at Open Repositories 2007 on Tuesday afternoon.

Directory Layout

The source directory tree has four major components: ‘lib’, ‘resources’, ‘src’, and ‘view’.

lib – libraries required by the application. The lib directory contains all of the JAR libraries required by the application. Its contents is a mix of the Seam-generated skeleton (pretty much everything at the top level of the ‘lib’ directory) and JAR libraries that are specific to the DRC application (in subdirectories of ‘lib’ named for the library in use). For instance, the ‘commons-codec-1.3′ and the ‘hibernate-all’ and the ‘jboss-seam’ JAR files were all brought into the project via ‘seam-gen’ while ‘lib/commons-net-1.4.1/commons-net-1.4.1.jar’ library was added specifically for this project. A convention has been established whereby new libraries added to the project appear as entries in the lib.properties file which is used by series of directives in the build.xml file to setup the classpaths for compiling and for building the EJB JAR. This is done to make the testing and transition of new libraries into the application more explicit and easily testable. Note that the newly included library directory also includes a copy of any license file associated with that library; this is not only a requirement to use some libraries but is also a good practice to show the lineage of some of the lesser known libraries. (For an example of what is required, see the changes to build.xml and to lib.properties in order to bring the Apache Commons Net library into the application.)

resources – configuration files and miscellaneous stuff. The resources directory holds the various configuration files required by the application plus other files used for testing and demonstration. Much of this was generated by the Seam-generated skeleton as well. Some key files here are the import.sql file (SQL statements that are used to preload the RDBMS used by Hibernate as the mocked up repository system) and the test-datastreams directory which has sample files for each of the media types.

src – Java source code. The src directory contains all of the Java source code for the application. Everything exists in a package called ‘edu.ohiolink.drc’ with subpackages for classes handling actions from the view component of the MVC, entity beans (sometimes known as Data Access Objects — or DAOs — I think), exception classes (more on this below), classes for working with FEDORA (not currently used), media type handler classes (more on this below), unit test classes (not currently used), and utility classes.

view – XHTML templates, CSS files, and other web interface needs. The view directory holds all of the files for the “view” aspect of the Model-View-Controller paradigm. More information about the view components is below.

Entity Classes

The entity beans package has three primary entity beans defined: Item.java, Datastream.java, and Description.java. (The FedoraServer.java entity bean is not used at this time.) Item.java is the primary bean that represents an object in the repository. Datastream.java and Description.java are component beans that only exist in the lifecycle of an Item.java bean; Datastream.java holds a representation of a FEDORA object datastream and Description.java holds a representation of a Dublin Core datastream for that object.

The Datastream and Description objects are annotated with @Embedded in the Item.java source; this is Hibernate’s way of saying that these objects do not stand on their own. Item.java also has numerous methods marked with a @javax.persistence.Transient annotation meaning that this is information not stored in the backing Hibernate database; these methods are for the various content handlers, which will be outlined below.

Mock Repository

As currently configured, the entity beans pull their information from a static RDBMS using Hibernate rather than from an underlying FEDORA digital object repository. (You’ll need to go back to revision 691 to see how far we got with the FEDORA integration into JBoss Seam before we switched our development focus to the presentation ‘view’ aspects of the application.) As currently configured, Hibernate uses an embedded Hypersonic SQL database for its datastore. As part of the application deploy process, the Java EE container will instantiate a Hypersonic database and preload it with the contents of the import.sql file. (The import.sql file contains just three sample records at the moment: one each for a text file, a PDF file, and a graphic file.)

All of the data for a repository object is contained in a single table record. Hibernate manages the process for us of reading that record out of the database and creating the three corresponding Java objects: Item, Datastream and Description. (Hibernate could also handle the process of updating the underlying table record if we were to change a value in one of the Java objects.) The mapping of table column to Java object field is handled by the @Column(name="xx") annotations in the entity beans.

For Datastream, what is stored in the database is not the datastream content itself but rather a filename that points to the location of the datastream file. The file path in this field can either be absolute (meaning a complete path starting from the root directory of the filesystem) or a relative path. In the case of the latter, the path is relative to the deployed application’s WAR directory (something like “…/jboss-4.0.5.GA/server/default/deploy/drc.ear/drc.war/” for instance). Note that the getter/setter methods for the contentLocation are private — the rest of the application does not need to know the location of the datastreams; this will also be true when the DRC application is connected to a FEDORA digital object repository. The method marked public instead is getContent, and the implementation of getContent hides the complexity of the fact that the datastream is coming from a disk file rather than a FEDORA repository call. For the three records/repository-objects currently defined in ‘import.sql’ there are three corresponding demo datastreams in the test-datastreams directory.

In all likelihood, this representation of the FEDORA repository will be too simple for us to move forward much further. In particular, the current notion of one datastream per repository object is too simplistic. The Datastream embedded object will likely need to be broken out into a separate table and as a corresponding distinct Java applet. (We may reach the same point soon for the Description object as well.)

By using the Entity Beans as a buffer between the business logic and the view components of the rest of the application, I hope we can minimize/localize the changes required in the future in order to replace the mock repository with a real underlying FEDORA repository.

View Templates

The preferred view technology for JBoss Seam is Facelets, an implementation of Java Server Faces that does not require the use of Java Server Pages (JSP). Although the ‘.xhtml’ pages in the view directory bear a passing resemblance to JSP, behind the scenes they are radically different. Of note for us is the clean templating system used to generate pages. The home.xhtml file has a reference to the template.xhtml file in the ‘layout’ directory. If you read through the template.xhtml file, you can see where the Facelets engine will pull in other .xhtml files in addition to the content within the <ui:define name="body"> tag of home.xhtml.

Content Handlers

The paradigm of handling different media types within the DRC application is guided in large part by the notion of disseminators for FEDORA objects and the Digital Library Federation Aquifer Asset Actions experiments. The underlying concept is to push the media-specific content handling into the digital object repository and to have the presentation interface consume those content handlers as it is preparing the end-user presentation.

For instance, the DRC will need to handle content models for PDFs, images, video, and so forth. Furthermore, how a video datastream from the Digital Video Collection is offered to the user may be different than how a video datastream from a thesis is offered to the user. Rather than embedding the complexity of making those interface decisions into the front-end DRC application, this model of content handlers pushes that complexity closer to the objects themselves by encoding those behaviors a disseminators of the object. What the presentation layer gets from the object is a chunk of XHTML that it inserts into the dynamically generated HTML page at the right place.

There is work beginning on a framework for FEDORA disseminators at /BaseDisseminator/trunk in the source code repository; that work has been put on hold at the moment in favor of focusing on the presentation interface. In order to prepare for the time when the presentation behaviors are encoded as FEDORA object disseminators, the current presentation layer makes use of Content Handlers for each of the media types. The Handler interface defines the methods required by each handler and the TextHandler class, the ImageHandler class, and the PdfHandler class implement the methods for the three media types already defined.

Of these, TextHandler class is the most complete, so I’ll use it as an example.

• The getRawDatastream method takes the datastream and sends it back to the browser with the HTTP headers that cause a File-Save dialog box to open.
• The getFullDisplay method returns a chunk of XHTML that presents the full metadata in a manner that can be included in a full metadata display screen.
• The getRecordDisplay method (currently unwritten) returns a chuck of XHTML used to represent the object in a list of records that resulted from a user’s search or browse request.
• The getThumbnail method (currently unwritten) returns a static graphic thumbnail rendition of the datastream (e.g. a cover page, a key video frame, etc.).

By making these content handlers distinct classes, it is anticipated that the rendering code for each of these methods can be more easily moved to FEDORA object disseminators with minimal impact to the surrounding DRC interface application.

Exception Handling

The DRC application follows the practice suggested by Barry Ruzek in Effective Java Exceptions (found via this link on The Server Side). The article can be summarized as:

One type of exception is a contingency, which means that a process was executed that cannot succeed because of a known problem (the example he uses is that of a checking account, where the account has insufficient funds, or a check has a stop payment issued.) These problems should be handled by way of a distinct mechanism, and the code should expect to manage them.

The other type of exception is a fault, such as the IOException. A fault is typically not something that is or should be expected, and therefore handling faults should probably not be part of a normal process.

With these two classes of exception in mind, it’s easy to see what should be checked and should be unchecked: the contingencies should be checked (and descend from Exception) and the faults should be unchecked (and descend from Error).

All unchecked exceptions generated by the application are subclasses of DrcBaseAppException. (DrcBaseApplication itself is a subclass of RuntimeException.) For an example, see NoHandlerException. By setting up all of the applications exceptions to derive from this point, we have one place where logging of troubleshooting information can take place (although this part of the application has not been set up yet). Except when there is good reason to do otherwise, this pattern should be maintained.

At this point, no checked (or contingency) exceptions specific to the DRC have been defined. When they are needed, though, they will follow the same basic structure with a base exception derived from Exception.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/lib?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L65 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/changeset/699%23file3 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/changeset/699%23file2 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L116 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/build.xml?rev=709#L101 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/lib/lib.properties?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/layout?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/drc-ds.xml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk?rev=691 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Description.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Datastream.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity/Item.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/utility?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/test?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions/DrcBaseAppException.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L140 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L132 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L92 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709#L63 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/PdfHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/ImageHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/TextHandler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/BaseDisseminator/trunk?rev=691 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions/NoHandlerException.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/import.sql?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/import.sql?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/layout/template.xhtml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/fedora?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/exceptions?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/entity?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/action?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/resources/test-datastreams?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/src/edu/ohiolink/drc/handler/Handler.java?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://drc-dev.ohiolink.edu/browser/drc/trunk/view/home.xhtml?rev=709 on January 13th, 2011.

The text was modified to remove a link to http://openrepositories.org/program/fedora#session4 on January 19th, 2011.

The text was modified to update a link from http://rama.grainger.uiuc.edu/assetActions/ to https://wiki.dlib.indiana.edu/display/DLFAquifer/Asset+Action+Project on January 19th, 2011.

The text was modified to update a link from http://dev2dev.bea.com/pub/a/2006/11/effective-exceptions.html to http://www.oracle.com/technetwork/articles/entarch/effective-exceptions-092345.html on January 20th, 2011.

Sandy Payette, Co-Director of the Fedora Project and Researcher in the Cornell Information Science department, announced a tentative date for the release 2.2 of the FEDORA digital object repository.

The Fedora development team would like to announce that Fedora 2.2 will be released on Friday, January 19, 2007.

This new release will contain many significant new features and enhancements, including [numbers added to the original for the sake of subsequent commentary]:

1. Fedora repository is now a web application (.war) that can be installed in any container
2. Fedora authentication has been refactored to use servlet filters (no longer Tomcat realms)
3. A new Fedora installer makes it easy to get started with Fedora (with both “quick” and “custom” install options)
4. GSearch service (backed by Lucene or Zebra) – flexible, configurable, indexes any datastream
5. Journaling service to create a backup/mirror repository
6. New checksum features for datastreams
7. Support for Postgres database configuration
8. Standard system logging with Log4J
9. Over 40 bug fixes
10. Many other enhancements

Be on the lookout for the release announcement the new year! Also, there will be opportunities to talk with the Fedora development team at Open Repositories 2007 (http://openrepositories.org/).

This is great news and a major step forward for the project. Here are some reasons why I think this is true.

1. Fedora repository is now a web application (.war)

With release 2.2, the core developers have effectively turned the software distribution inside out. The primary output of the new build process is a standard Web ARchive (or WAR) file that can be put inside any servlet container. The new installation program (see #3 below) comes with a Tomcat distribution, should a new installation need it, but it is no longer required. There have been reports that the new WAR-based distribution works inside the Jetty servlet container; we’re hoping it will work in the JBoss Application Server as well (since that is what we’re using to build our next generation interface).

2. Fedora authentication has been refactored to use servlet filters

3. A new Fedora installer makes it easy to get started with Fedora

4. GSearch service

GSearch — where “G” stands for “General” but could equally well stand for “Gert” Schmeltz Pedersen, its lead developer from the Technical University of Denmark — does all of the above as a new component in the FEDORA Service Framework. We extend our gratitude to Gert and his colleagues for contributing their work to the general FEDORA distribution as well as to DEFF, Denmark’s Electronic Research Library, which funded the GSearch project.

5. Journaling service

6. Datastream checksums

7. Support for PostgreSQL

8. Standard system logging with Log4J

9 and 10. Bug fixes and many other enhancements

Version 2.2 represents another monumental step towards the vision of a Flexible, Extensible Digital Object Repository Architecture. Congratulations to the core developers for what sounds like is going to be a great release.

The text was modified to update a link from http://comm.nsdl.org/pipermail/fedora-users/2006-December/002330.html to http://article.gmane.org/gmane.comp.cms.fedora-commons.user/2330/ on January 19th, 2011.

The text was modified to remove a link to http://defxws2006.cvt.dk/fedoragsearch/ on January 19th, 2011.

Keeping track of configuration changes to servers is a tough job made tougher when some of the sysadmins work from home. Questions of who did what when and why can be exacerbated by the lack of physical proximity — in other words, I can’t simply yell over the cubical wall to the colleague down the hall to ask him about the new package installed on the server. Besides, that oral history tradition is difficult to maintain and harder to sustain as the number of machines grows. This essay describes a practice for maintaining a Gentoo Linux distribution using GLCU, Subversion, and Trac that is lightweight (doesn’t impose a large burden on the sysadmin staff), effective (although it is lightweight it better documents and makes accessible the state of our systems over the oral history tradition), and cheap (no operating budget dollars were harmed in the creation of this process — only staff time overhead).

Create an All-Encompassing Configurations Directory

The RCS will want to act on a single directory tree, but in most cases our configuration files are spread out over the file system. Most are in /etc, but others exist elsewhere. (The portage “world” file, a record of everything installed on your system, for instance, is in /var/lib/portage.) What we do is create a directory called /server-rcs that will be managed by the RCS, and in that directory is copies or links to all of the configuration files on the system.

Putting /etc (or any other directory) Under Version Control

/server-rcs # ln ../etc .
ln: `../etc': hard link not allowed for directory

What we need to do instead is a trick using the ‘mount‘ command to bind one portion of the file system to another part. From the mount MAN page:

Since Linux 2.4.0 it is possible to remount part of the file hierarchy somewhere else. The call is

mount --bind olddir newdir

After this call the same contents is accessible in two places. One can also remount a single file (on a single file).

So we can bind the entire /etc directory into our RCS space with this command:

mount --bind /etc /server-rcs/etc

Better yet, we put this in our /etc/fstab file (also adding the /var/spool/cron directory as well):

/etc                            /system-rcs/etc                         none    bind/var/spool/cron/crontabs        /system-rcs/var-spool-cron-crontabs     none    bind

Since the /etc directory (and other directories) already exist, we’re going to have to play some games to get them into the repository. For the trick do to this with Subversion, see the FAQ entry on in-place imports.

Handling Individual Files Under Version Control

We could use the mount ‘bind’ trick to put individual files into the /server-rcs space, but that seems overly complicated. Our servers are generally configured with few filesystems, so in many cases the files we need to track in the RCS are within the same filesystem and we can use hard links to put them into the /server-rcs directory. Another alternative is to write a cron job to copy configuration files into the /server-rcs directory, but then realize that this kind of revision control is one way — if we restore a previous version of a file from the RCS, we need to manually copy it back to the original location.

(On the other hand, using the mount ‘bind’ method is a form of self-documenting the otherwise invisible hard links to files in the same filesystem. For that reason, it might be worth considering that option.)

Special Case: /var/lib/portage/world

What we do instead is patch into a hook of the ‘emerge’ command that will save a sorted copy of the world file into /server-rcs. This patch goes into /etc/portage/profile/profile.bashrc:

if [ "${EBUILD_PHASE}" == "setup" ]
then
        sort /var/lib/portage/world > /server-rcs/misc/var-lib-portage-world
fi

Every time ‘emerge’ goes through the ‘setup’ mode when installing a package, it will run this sort command. Note that there is no file locking going on here, so there is a remote chance that the /server-rcs version (but not the /var/lib/portage version) could get corrupted. Such a problem is minor, though, and easily fixed.

Importing into Subversion

svn add --force /server-rcs
svn checkin --message "Importing the configuration files for the server" /server-rcs https://svn.repository.url/svn/configurations/server

Because of the in-place import problem for pre-existing directories (described earlier), we likely had to create some of the repository directory structure already. (In this example, we would have executed a svn mkdir https://svn.repository.url/svn/configurations/server/etc command already to “prime the pump” for adding /etc to the repository.) In line #1, the –force option makes the ‘svn add’ command continue the recursive directory parse to add files and directories to the RCS structure even if some component of those paths were already in the RCS structure. Line #2 checks in our completed /server-rcs directory.

Daily Usage

svn status /server-rcs
svn checkin /server-rcs -m "Free-text description of why you made the changes."

The first line will show you the files modified since the last check-in &mdash hopefully only the files you intended to modify, although this is a good point to check to make sure an inadvertent change didn’t happen. The second line will copy changes to the /server-rcs directory into the RCS along with the free-text note describing why you made the change.

Isn’t this great? It is sort of self-documenting. Not only to you have your brief description of what you did but you also have the exact changes made to the configuration files. If a change doesn’t work out, you have easy access to past configurations that allow you to revert back to a previous state. (Note, though, that we’re not saving actual applications in the RCS — you may have to recompile and install older versions of applications to get back to the previous state.)

Portage Updates with GLCU

See the project on SourceForge for all of the details on installing, configuring and running GLCU. We make one tweak to the GLCU configuration to prompt the sysadmin to complete all of the housekeeping chores: running dispatch-conf to merge changes to configuration files and revdep-rebuild to make sure all of the applications using updated linked libraries are properly recompiled. To do this, add a line to /etc/conf.d/glcu:

updatetc: dispatch-conf && revdep-rebuild -X -pv

A typical update for us looks like:

# glcu /tmp/glcuUpdate-23112****************************************>> Welcome to glcu's easy update featurePrebuilt packages:------------------(  1 ) [binary     U ] app-editors/nano-2.0.1 [1.3.12-r1] USE="ncurses nls spell unicode  -debug -justify -minimal -slang"(  2 ) [binary     U ] media-libs/libsdl-1.2.11 [1.2.8-r1] USE="X esd* -aalib -alsa -arts  -dga -directfb -fbcon -ggi -libcaca -nas -noaudio -noflagstrip -nojoystick -novideo  -opengl -oss -svga -xinerama -xv (-pic%)" Do you want to install the prebuilt package(s) [Y/n]   (or you can either install only specified package number(s) #,     or NOT install package with -# and use i# for injecting)> y[...pre-compiled packages are installed...]>>> Auto-cleaning packages...>>> No outdated packages were found on your system. * GNU info directory index is up-to-date.* IMPORTANT: 1 config files in /etc need updating.* Type emerge --help config to learn how to update config files.glsa's:  ['200612-03'](  1 ) 200612-03 [N] GnuPG: Multiple vulnerabilities ( app-crypt/gnupg ) Do you want to fix all glsa's now? [Y/n]    (or you can either install only specified glsa number(s) #,     or NOT install glsa with -# and use i# for injecting)> y[...packages related to the GLSA are downloaded, compiled and installed...] Do you want to run dispatch-conf && revdep-rebuild -X -pv now? [Y/n] > y[...dispatch-conf and revdep-rebuild are run...]

With the system nicely updated, we can check in all of the changes to the RCS with a note about what we did:

svn ci -m "After running 'glcu' to update app-editors/nano, media-libs/libsdl, and GLSA for app-crypt/gnupg"

Tracking Configuration Changesets and Trouble Tickets with Trac

Trac is an open source wiki and issue tracking system for software development projects. Its stated mission is to “help developers write great software while staying out of the way.” In this case we’ll be using it to help sysadmins manage complex systems while staying out of the way. Trac is a web-based tool that “allow wiki markup in issue descriptions and commit messages, creating links and seamless references between bugs, tasks, changesets, files and wiki pages. A timeline shows all … events in order, making the acquisition of an overview of the [state of the system] and tracking progress very easy.”

Trac is synchronized with our Subversion source code repository, so the timeline of changes (demo) shows each check in to the Subversion RCS (demo), which can be tied to an issue ticket (demo) for a problem or task that is requested, worked on, then closed via simple wiki-like markup. One can also browse through the stored changes (demo) and look at a graphical difference between any two revisions of a file (demo) but also review the log of check in messages (demo) associated with that file over time.

Conclusion

The text was modified to remove a link to Demonstration of Trac Issue Tickets on December 30th, 2010.

The text was modified to remove a link to http://gentoo-wiki.com/MAN_emerge_1 on January 19th, 2011.

The text was modified to remove a link to http://gentoo-wiki.com/MAN_mount_8 on January 19th, 2011.

The text was modified to remove a link to http://gentoo-wiki.com/MAN_emerge#lbAN on January 19th, 2011.

The text was modified to remove a link to http://trac.edgewall.org/changeset?new=trunk%2Fhtdocs%2Fcss%2Ftrac.css%404501&old=trunk%2Fhtdocs%2Fcss%2Ftrac.css%404390 on January 19th, 2011.

On Friday, the Colorado Alliance of Research Libraries announced the creation of a consortium-wide digital repository project similar to that of the Ohio Digital Resource Commons.

Colorado Alliance Digital Repository Project Approved

The Board of Directors of the Colorado Alliance of Research Libraries has approved initial funding for a consortium-wide digital repository project at its October 19, 2006 meeting.

The Board of Directors of the Colorado Alliance of Research Libraries has approved initial funding for a consortium-wide digital repository project at its October 19, 2006 meeting. The project will use the Fedora open source software which was selected after a long evaluation process by the Institutional Repository Implementation Team, chaired by John Culshaw from the University of Colorado at Boulder.

The Alliance Digital Repository (ADR) project allows the participating member libraries to develop a shared technical and development infrastructure to store, preserve and disseminate a whole variety of digital objects including images, text, audio, video, learning objects, data sets or any other kind of material. The project will make use of open source tools developed by others in order to fast track functionality. As new software is developed as part of the project it will also be made available to the open source community.

via Peter Suber’s Open Access News

Welcome to the party!

We’re beginning a new phase of our digital library development at OhioLINK and an oversimplification of one of the consequences of this new phase is that we will be developing more software from scratch rather than adapting stuff that we find out there on the net. (Another consequence of this new phase is our interest in applying the Service-Oriented Architecture paradigm to library applications.) In previous phases, we were somewhat at the mercy of whatever development framework was used in the application we were adopting. As we start this new development where we control more of our own destiny, we wanted to take a step back and look at the available frameworks to support our development efforts. The options we identified at the start were plain Java servlets, Apache Struts, Spring Framework, and EJB3 with JBoss SEAM.

Our analysis is admittedly by proxy rather than through direct experience. If we had more time, we would start our new phase in each one of these (with the associated learning curve for some of them) and pick the one that worked out the best. We have neither the luxury of time nor of excess developer talent, so we looked at what others were saying, created some sample applications, and discussed our gut reactions to each option. In the end, we decided to start with EJB3/SEAM.

Summary of Options

Over the years, the Java community has developed frameworks to support the creation of applications in the Java programming language. These frameworks consist of a combination of code libraries, programming rules, and best practices that have evolved based on the research and experience of the developer community. Just as almost no one designs, codes, debugs, and documents their own low-level file I/O libraries any more, these frameworks provide a level of abstraction over the raw programming language that enables developers to create better code faster.

Within the web application arena, arguably the earliest successful framework was Apache Struts. It was one of the first frameworks to promote a Model-View-Controller (MVC) architecture: the Model represents the business or database code, the View represents the page design code, and the Controller represents the navigational code. By creating these three layers, Struts promoted a practice of design and development away from commingling database code, page design code, and control flow code in the same JavaServer Pages files. In practice it was found that unless these three concerns are separated, larger applications become difficult to maintain.

The latest revolution in design architectures is Dependency Injection (DI). Sometimes also referred to as Inversion of Control (or IoC), it is a programming design pattern and architectural model in which the responsibility for object creation and object linking is removed from the code of the objects themselves. As the term “Dependency Injection” may imply, it is the framework that instantiates the component objects and binds them to each other through the use of setters and/or constructors rather than the components linking themselves together. In this pattern, the objects themselves become loosely coupled, allowing for a more dynamic and testable integration of the component objects. The creation and binding of objects is defined in an XML configuration file (Spring Framework) or via Java Annotations (EJB3).

Two primary frameworks have emerged for the Dependency Injection paradigm: Spring Framework and EJB3. The Spring Framework is the grandparent of subsequent IoC efforts (including EJB3). A de facto standard, it is widely used in the Java programming community with a large number of tools and documentation. EJB3 is a formal specification adopted by participants in the Java Community Process for standardizing Java-related developments. Coupled with the JBoss SEAM framework, the EJB3/SEAM combination is roughly on par with raw technical capabilities of the Spring Framework. SEAM removes some of the complexity of raw EJB3 by providing and preconfiguring popular choices for views (JSF) and models (Hibernate) as well as integrated AJAX-based Remoting and jPBM. Both promote the same MVC architecture as Struts.

The distinguishing characteristics are that Spring acts as an open integration tool at the expense of a complicated XML-based configuration process whereas EJB3/SEAM is simpler in its configuration and more restrictive in what other tools can be easily brought into the framework. In other words, Spring has a wide array of choices for the various framework components and for the most part the developer must manage the integration; EJB3/SEAM lessens the complexity of the framework by limiting the choices available for the various components to only the most popular options. Note that there is overlap between Spring and EJB3; the Pitchfork project allows for JSR-250 (Common Annotations) dependency injection and EJB 3.0 style interception as an add-on to the Spring framework. One should also note that a full EJB3 implementation is a kind of superset to SEAM, and if the needs of our applications go beyond that of SEAM it is possible to reduce our dependency on the SEAM framework by beginning to integrate other choices (and managing that integration ourselves).

OhioLINK staff would have a learning curve associated with both Spring and EJB3/SEAM (and Apache Struts as well, although it has been discounted as an option as a mostly dead-end architecture at this point). On the question of whether the learning curve is greater than the effort of doing things the “plain Java” way, one can point to the large number of developers who have made the leap to these frameworks and are arguably more productive for doing so. Since we are beginning the development of a new code base, it seems to be the ideal time to start up that learning curve with the creation and deployment of a new service. The learning curve might be easier for Spring than for EJB3/SEAM due to the wide variety of materials already produced for Spring, although the corporate backers of EJB3/SEAM seem to be filling this gap at a steady pace. The learning curve for EJB3/SEAM may be shallower, though, because SEAM simplifies the configuration of the framework itself.

At our development team meeting yesterday, we decided that OhioLINK will adopt a Dependency Injection (DI) paradigm over use of Apache Struts and plain Java servlets because of the anticipated large productivity gains after the learning curve. Of the two DI/IoP frameworks widely available now, we decided to adopt EJB3 coupled with JBoss SEAM. The standards-driven nature of EJB3 reduces the risk of adopting a technology that may not be supported in the medium-term. It is expected that JBoss SEAM will flatten the learning curve to make EJB3 more readily understood in the short term while providing a migration path to a broader EJB3 implementation (beyond the capabilities of SEAM) if needed in the future. The Spring Framework, to its credit, also makes it possible to envelop EJB3 objects as part of a Spring-based application, which could smooth the transition to that framework should it become necessary. We also understand that there is some risk of “vendor lock-in” by relying on JBoss SEAM — in our limited experience, applications seemed to deploy better under JBoss Application Server as opposed to a stock Apache Tomcat 5.5 installation. On the other hand (if the “standards” nature of EJB3 is to be trusted), it should be possible to move our code to other application servers, leaving SEAM behind, with minimal changes. We may also be able to further flatten the learning curve by buying a support contract with JBoss in the short- to medium-term.

Details about the Candidates

Enterprise Java Beans 3 (EJB3)

The EJB 3.0 framework is a standard framework defined by the Java Community Process (JCP) and supported by all major J2EE vendors. [The architecture of the Spring framework is based upon the Dependency Injection (DI) design pattern.] Open source and commercial implementations of EJB 3.0 specifications are already available from JBoss and Oracle. EJB 3.0 makes heavy use of Java annotations.

• Based on standards work (the Java Community Process)
• “Configuration by default” using Java Annotations backed by more complicated XML configuration files, if needed
• A more compact, rigidly-defined framework stack; easier configuration, but fewer choices
• Brand new (recently ratified); fewer tools, books, tutorials available

JBoss SEAM

JBoss Seam is a powerful new application framework to build next generation Web 2.0 applications by unifying and integrating popular service oriented architecture (SOA) technologies like AJAX, Java Server Faces (JSF), Enterprise Java Beans (EJB3), Java Portlets and Business Process Management (BPM) and workflow.

Seam has been designed from the ground up to eliminate complexity at the architecture and the API level. It enables developers to assemble complex web applications with simple annotated Plain Old Java Objects (POJOs), componentized UI widgets and very little XML. The simplicity of Seam 1.0 will enable easy integration with the JBoss Enterprise Service Bus (ESB) and Java Business Integration (JBI) in the future.

• Based on EJB3
• Integrates together some of the most widely adopted components such as JavaServer Faces and Hibernate
• Can reportedly be used outside of JBoss Application Server (direct experience show some problems with this)

Spring Framework

The Spring framework is a popular but non-standard open source framework. It is primarily developed by and controlled by Interface21 Inc. The architecture of the Spring framework is based upon the Dependency Injection (DI) design pattern. Spring can work standalone or with existing application servers and makes heavy use of XML configuration files.

• Explicit configuration via XML file
• More flexible than EJB3/SEAM in swapping in and out various modules; yet the decisions made early on limit the ability to swap in and out later.
• Several years old; widely available tools, books, tutorials
• De facto standard of an open source community with benevolent control by a corporate entity.

Apache Struts

Apache Struts is a free open-source framework for creating Java web applications.

• (Summary opinion) Generally seen as a deprecated framework; few new projects start with Struts.
• Weak integration of new techniques such as AJAX.

Plain Java Servlet (No Framework)

• Must build the entire application support layer (object data storage, presentation interfaces, helper functions, etc. from scratch).
• No ready-made integration of new techniques such as AJAX.

Background Information

• Make the Right Decision with Our Side-by-Side Comparison of Spring and EJB 3.0
• Make the Right Decision with Our Side-by-Side Comparison of Spring and EJB 3.0, Part 2
• Discover Seam and Sew Up Your Java Projects Faster than Ever
• Dependency injection – Wikipedia, the free encyclopedia
• Pitchfork project frequently asked questions
• Trying out JBoss’ Seam (Brian Leonard’s Blog)
• SEAM Getting Started Guide
• JBOSS-based SEAM demonstration site

The text was modified to remove a link to http://seam.demo.jboss.com/ on January 13th, 2011.

The text was modified to update a link from http://interface21.com/pitchfork/pitchfork-faq.html to http://www.springsource.com/products/pitchfork/pitchfork-faq on January 19th, 2011.

The Fedora Outreach and Communications team is conducting a survey of the high-level sense of passion and commitment inherent in the Fedora community. I’ve posted some answers back to the FEDORA wiki on behalf of OhioLINK, and am also including the responses here as it fits into the “Why FEDORA?” series of blog postings. (If you are reading this through a RSS news reader, I think you’ll have to actually come to the DLTJ website and scroll down to the bottom of this post to see the table of contents of the series.) On with the responses!

How did you hear about Fedora?

I first remember hearing about FEDORA at a Coalition for Networked Information meeting in 2003. I only really remember it in passing because what was being presented was so radical that I didn’t appreciate what was being described.

I next encountered FEDORA during a conference call with the Internet2 Shibboleth core developers in mid-2004. The topic was enabling cross-repository access management — a topic that is still a challenge today (although the Shibboleth team is working on it). But that time I really started to catch on with what the FEDORA team was doing, and started paying closer attention.

Why did you chose Fedora?

When I arrived as project manager to the Ohio Digital Resource Commons (DRC) project in January 2005, OhioLINK was on the path to expand their existing Documentum installation to include a hosted institutional repository service. The Ohio DRC Steering Committee reviewed and accepted a proposal to use FEDORA as the foundation of this new hosted institutional repository service primarily because OhioLINK would be working with peers to develop the service (rather than working in isolation as likely would have happened with a Documentum-based solution).

Were there economic advantages to your project/org. in selecting Fedora?

The open source, free-to-license nature of FEDORA was definitely an advantage. It allowed us to turn grant funding that would have been used to pay for additional Documentum modules and licenses into to salary for temporary-hire programmers. In that way we felt that we had a better control over our destiny by creating the application code ourselves rather than relying on consultants.

What is Fedora’s unique role in your production system?

OhioLINK is beginning to look at the Service Oriented Architecture (SOAs) software design paradigm, and FEDORA fits right into that model as the content repository for all of our digital objects. If anything, FEDORA’s nature as a best-of-breed content repository — and nothing else — encourages us to think along the likes of sooner than we might otherwise have done.

Is there one specific Fedora attribute that enables your project/organization to accomplish your overall goals.

The fact that FEDORA is completely agnostic to what is contained in a datastream — be it audio, video, image, dataset, PDF, Dublin Core, MODS, EAD, FGDC, TEI, etc. — means that we can truly pursue a goal of managing all of our content in one place. The robustness of the content repository functions allows us to consider more interesting questions such as how this different content is ultimately presented to the end user.

Do you see yourself as an active member of the Fedora community? Why?

Yes. FEDORA represents the ability to take long-term control over the destiny of our digital objects. If, for some reason, the existing core developers at Cornell and UVa disappeared, a vibrant user community (OhioLINK included) can pick up the task of maintaining the software for the collective good. And if no one but OhioLINK is left in a “FEDORA community” our job of migrating out of it, should we desire to do so, is eased by the fact that we have the full view of the source code to help us move content and services to a new platform.

What would inspire you to become more involved?

It would take the existence of more hours in the day, I’m afraid!

What should be the mission of an ongoing Fedora organization?

A FEDORA community should first and foremost inspire communication among users of the FEDORA software. Almost all of us are working with extremely limited resources, and it weakens our collective effort if there is duplicated work underway. This communication should include not only developers but also users of the software.

The text was modified to remove a link to http://www.fedora.info/wiki/index.php/The_Fedora_Users_Interview_Survey on January 13th, 2011.

The text was modified to remove a link to http://www.fedora.info/wiki/index.php/Fedora_Outreach_User_Group on January 13th, 2011.