The main Open Repositories conference concluded this morning with a keynote by Clifford Lynch and the separate user group meetings began. I tried to transcribe Cliff’s great address as best I could from my notes; hopefully I’m not misrepresenting what he said in any significant ways. He has some thought-provoking comments about the positioning of repositories in institutions and the policy questions that come from that. For an even more abbreviated summary, check out this Storify archive of tweets during his keynote. Then I attended the DSpace track of user group programming, and below there are summaries of plans for DSpace version 1.8 and the new DSpace Curation Services.

Repositories: Major Progress and Open Questions

Instead, the growth of repositories has highlighted critical policy discussions of what the missions of institutions of higher education are supposed to be. Questions such as the responsibility to curate knowledge they create, curate the evidence on which inquiry is based, to disseminate knowledge. These weren’t on the table 10-15 years ago. Now they are central issues for discussion in the leadership of universities. Tied to institutional repositories is the question of open access. The scope of issues goes beyond just open access, though. It reaches into the kind of questions that are now getting traction like getting access to research data and an institution’s role of stewardship and disseminate of research data. Also the creation of open educational resources and institution’s responsibility to disseminate such resources. These questions wouldn’t have emerged without the effort to build out institutional repositories. As soon as you start talking about these questions they demand investments in infrastructure of institutional repositories. So we should take satisfaction in the role that the efforts of deploying institutional repositories have played in advancing these critical policy discussions. These questions have gone unanswered far too long.

That said, there is a danger of confusion of mechanism with policy. We went through a bad period around 2004 when institutional repositories were deployed without knowing what would be put in them. Institutional repositories are services to support policies, not ends to themselves. We have mostly gotten past this and can have the discussion of whether institutionally based repositories are the appropriate tool and when should we build discipline-specific repositories or other kinds of platforms that are not institutionally focused.

He also noted that the question of institutional assets and the balance of faculty control with institutional responsibility is being talked about, if only quietly. The piece of video that Clifford referred to — a talk by Derick Law on how universities have failed in their stewardship responsibilities of research — may be this video from the The Blue Ribbon Task Force on Sustainable Digital Preservation and Access’s National Conversation on the Economic Sustainability of Digital Information (skip to “chapter 2″ of the video) held April 1, 2010 in Washington DC.

Not only have institutional repositories acted as focal point for policy, they have also been a focal point for collaborations. Library and IT collaborations were happening long before institutional repositories surfaced. Institutional repositories, though, have been a great place to bring other people into that conversation, including faculty leaders to start engaging them in questions about dissemination of their work. Also chief research officers; in 1995 if you were a university librarian doing leadership work constructing digital resources to change scholarly communication, would have talked to CIO but may not know who your chief research officer was at that point. That set of conversations, which are now critical when talking about data curation, got their start with institutional repositories and related policies.

Another place for conversation has been those in the university administrations concerned with building public support for the institution. By giving the public a deeper understanding of what the institution contributes to culture, industry, health and science, and connecting faculty to this effort. This goes beyond the press release by opening a public window into the work of the institutions. This is particularly important today with questions of public support for institutions.

That said, there are a number of open questions and places where we are dealing with works-in-progress. Cliff then went into an incomplete and, from his perspective, perhaps idiosyncratic, list of these issues.

Repositories are one of the threads that are leading us nationally and internationally into a complete rethinking of the practice of name authority. While it is a librarian, old fashion concept, but it is converging with “identity management” from IT. He offered an abbreviated and exaggerated example: librarians did name authority for authors of stuff in general in 19th century. In 20th century there was too much stuff, particularly stuff in journals and magazines became overwhelming. So libraries backed off and focused only on books and stuff that went into catalogs; the rest they turned over to indexing and abstracting services. We made a few weird choices like an authority file should be as simple as possible to disambiguate authors rather than be as full as possible, so we had the development of things along side name authority files like the national dictionaries of literary biographies.

For scientific journal literature, publishers followed practices about how obscure author names could be (e.g. just last name and first initial). Huge amounts of ambiguity of “telegraphic author names” results in a horribly dirty corpus of data. A variety of folks are realizing that we need to disambiguate authorship by assigning author identifiers and somehow go back and cleanup the mess in the existing bibliographic data of scholarly literature, especially journal literature. Institutions taking more responsibility for the work of their community, and having to do local name authority all over again. We have the challenge of how to reconnect this activity to national and international files. We also have a set of challenges on whether we want to connect this to biographical resources. It brings up issues of privacy, when do people do things of record, and how much else should come along with building a public biography resource. We also see a vast parallel investment of institutional identity management. Institutions haven’t quite figured out that people don’t necessarily publish with the same name that is recorded in the enrollment or employment systems that the institution manages, and that it would be a good idea to tie those literary names to identity files that the institution manages.

We’re not confident of the kind of ecological positioning institutional repositories among a pretty complicated array of information systems found at a typical large university. Those systems include digital library platforms, course management systems, lecture capture systems, facilities for archiving the digital records of the institution, and platforms intended to directly support active research by faculty. All are evolving at their own rate. It is unclear where the institutional repositories fit, and what are the boundaries around them.

Here is one example. What is the difference between an institutional repository and a digital library/collection? You’d get very different answers from different people. One might be who does the curation, how it is sourced, and how it is scoped. The decisions are largely intellectual. Making this confusing is that you’ll see the same platform for institutional repositories and digital library platforms. We are seeing a convergence of the underpinning platforms.

Another one: learning management systems (LMS). These are virtually universal among institution in the same timeframe that institutional repositories have been deployed. We’ve done a terrible job at thinking about what happens to the stuff in them when the course is over. We can’t decide if it is scholarly material, institutional records, or something else. They are tangled up between learning materials and all of the stuff that populates a specific performance of a course such as quizzes and answers, discussion lists, and student course projects. We don’t have taxonomies and policies here and a working distinction between institutional repositories and learning management systems. It is an unusual institution that has as systematic export from the LMS to an IR.

Lecture capture systems becoming quite commonplace; students are demanding them in much the same way that the LMS was demanded. A lecture capture system may be more universally helpful than an LMS. Lectures being captured for a wide range of reasons, but not knowing why means it is difficult to know whether to keep them and how to integrate them into the institution’s resources.

Another example: the extent to which institutional repositories should sit in the stream of active work. As faculty are building datasets and doing computation with them, when is it time for something to go into an institutional repository. How volatile can content be in the repository? How should repositories be connected or considered as robust working storage? He suspects that many institutional repositories are not provisioned with high-performance storage and network connections, and would become a bottleneck in the research process. The answers would be different for big data sets and small data sets, and we are starting to see datasets that are too big to backup or two big to replicate.

Another issue is that of virtual organizations, the kind of collaborative efforts that span institutions and nations. They often allow relatively low overhead to mobilize researchers to work on a problem, and are becoming commonplace in sciences and social sciences and starting to pop up in the humanities. We have a problem for the rules-of-the-road between virtual organizations and institution-based repositories. It is easy to spin up an institutional repository for a virtual organization, but what happens to it when the virtual organization shuts down. Some of these organizations are intentionally transient; how do we assign responsibility for a world of virtual organizations and map them into institutional organizations for long-term stewardship.

Software is starting to concern people. So much scholarship is tied up now in complicated software systems that we are starting to see a number of phenomena. One is data that is difficult to reuse or understand without the software. Another is the is difficulty surrounding reproducibility — taking results and realizing they are dependent on an enormous stack of software and we don’t have a clear way to talk about the provenance of a result that is based on the stack of software versions that would allow for high-confidence in reproduction of results. We’re doing to have to deal with software. We are also entering an era of deliberate obsolescence of software; for instance, any Apple product that is older than a few years is going to the dustbin and it hasn’t been fully announced or realized so that people can deal with it.

Another place that has been under-exploited is the question of retiring faculty and repositories. Taking inventory of someone’s scholarly collections and migrating it to an institutional framework in an orderly fashion.

How we reinterpret institutional repositories going beyond universities. For example there is something that looks a bit like an institutional repository but has some different things about it that belongs in public libraries or historic societies or similar. This dimension bears exploration.

To conclude his comments he talked about a last open issue. When we talk about good stewardship and preservation of digital materials, there are a couple of ideas that have emerged as we tried to learn from our past stewardship of print scholarly literature. One of these principles is that geographic replication is a good thing; we’re starting to see this in a sense that most repositories are based on some geographically redundant storage system or we’ll see a steady migration towards this in the next few years. A second one is organizational redundancy. If you look at the print work, it wasn’t just that the scholarly record wasn’t in a number of independent locations but also that control was replicated among institutions that were making independent decisions about adding materials to their library collection. Clearly they coordinated to a point, but they also have institutional independence. We don’t know how to do this with institutional repositories. This is also emerging in special collections as they become digital. Because they didn’t start life as published materials in many replicated versions, we need other mechanisms to have curatorial responsibility distributed. This is linked to the notion that it is usually not helpful to talk about preservation in terms like “eternity” or “perpetuity” or life-of-the-republic. It is probably better in most cases to think about preservation in one chunk at a time; an institution making a 20-year or 50-year commitment with a well-structured process at the end. That process includes whether an institution should renew the commitment and if not other interested parties could come in and take responsibility with a well-ordered hand-off. This ties into policies and strategies for curatorial replication across institutions and ways that institutional repositories will need to work together. It may be less critical today, but will become increasingly critical.

In conclusion, Cliff said that he hoped left the attendees with a sense that repositories are not things that stand on their own. That they in fact are mechanism that advance policy in a very complex ecology of systems. In fact, we don’t have our policy act together on many systems adjacent to the repository that leads to issues of appropriate scope and interfaces with those systems. Where repositories will evolve to in the future as we understand the role of big data is also of interest.

DSpace 1.8

SWORD is a lightweight protocol for depositing items between repositories; it is a profile of the Atom Publishing Protocol. At the current release, DSpace has be able to accept items; the planned work for 1.8 will make it possible to send items. Some possible use cases: publishing from a closed repository to an open repository, sending from the repository to a publisher, from the repository to a subject-specific service (such as arXiv), or vice versa. The functionality was copied from the Swordapp demo. It supports SWORD v1 and only the DSpace XMLUI. A question was ask about whether the SWORD copy process is restricted to just the repository manager? The answer was that it should be configurable. On the one hand it can be open because it is up to the receiving end to determine whether or not to accept it. On the other hand, a repository administrator might want to prevent items being exported out of a collection.

MIT has rewritten the Creative Commons licensing selection steps. It uses the Creative Commons web services (as XML) rather than HTML iframes, which allows better integration with DSpace. As an aside, the Creative Commons and license steps have been split into two discrete steps allowing different headings in the progress bar.

The DSpace Community Advisory Team prioritized issues to be addressed by the developers, and for this release they include JIRA issue DS-638 for virus checking during submission. The solution invokes the existing Curation Task and requires Clam AV antivirus software to be installed. It is switched off by default and is configured in submission-curation.cfg. Two other issues that were addressed are DS-587 (Add the capability to indicate a withdrawn reason to an Item) and DS-164 (Deposit interface), which was completed as the Google Summer of Code Submission Enhancement project.

Thanks to Bojan Suzic in his Google Summer of Code project, DSpace has had a REST API. The code has been publicly available and repositories have been making use of it, so the committers group want to get it into a finished state and include it in 1.8. There is also work on an alternative approach to a REST API.

DSpace and DuraCloud was also covered; it was much the same that I reported on earlier this week, so I’m not repeating it here.

From the geek perspective, the new release will see increasing modularization of the codebase and more use of Spring and the DSpace Services Framework. The monolithic dspace.cfg will be split up into separate pieces; some pieces would move into Spring config while other pieces could go into the database. It will have a simplified installation process, and several components that were talked about elsewhere at the meeting: WebMVC UI, configurable workflow, and more curation tasks.

Introduction to DSpace Curation Services

Some built-in tasks are to validate metadata against input forms (halts on task failure), count bitstreams by format type, virus scan (uses external virus detection service), on ingest (the desired use case), and the replication suite of tasks for DuraCloud. Other tasks: link checker and 11 others (from Stuart Lewis and Kim Shepherd), format id with DROID (in development), validate/add/replace metadata, status report on workflow items, filter media in workflow (proposed), and checksum validation (proposed).

What does this mean for different users? As a repository or collection manager, it means new functionality — GUI access without GUI development: curation, preservation, validation, reporting. As a developer: rapid development, and deployment of functionality without rebuilding or redeploying the DSpace instance.

The recommended Java development environment for tasks is with a package outside of dspace-api. Make a POM with dependency on dspace-api, especially /curate. Required features of the task are a constructor with no arguments to support loading as a plugin and that it implements the CurationTask interface or extends the AbstractCurationTask class. Deploy it as a JAR and configure (similar to a DSpace plugin)

There are some Java annotations for Curation Task code that are important to know about. Setting @Distributive means that the task is responsible for handling any contained DSpace objects as appropriate. Otherwise the default is to have the task executed across all contained objects (subcommunities, collections, or items). Setting @Suspendable means the task interrupts processing when first FAIL status is returned. Setting @Mutative means the task makes changes to target objects.

Invoking tasks can be done several ways: from the web application (XMLUI), the command line, from workflow, from other code, or from a queue (deferred operation). In the case of the workflow, one can target the action of the task at anywhere in the workflow steps (e.g. before step 1, step 2, step 3 or at item installation). Actions (reject or approve) are based on tasks results, and notifications are sent by e-mail.

A mechanism for discovering and sharing tasks doesn’t exist yet. What is needed is a community repository of tasks. For each task what is needed is: a descriptive listing, documentation, reviews/ratings, link to source code management system, and link to binaries applicable to specific versions.

With dynamic loading with scripting languages in JSR-223, it is theoretically possible to create Curation Tasks in Groovy, JRuby, Jython, although the only one Bill has been able to get to work so far has been Groovy. Scripting code needs a high level of interoperability with Java, and must implement the CurationTask interface. Configuration is a little bit different: one needs a taskcatalog with descriptors for language, name of script, and how the constructor is called. Bill demonstrated some sample scripts.

In his conclusion, Bill said that the new Curation Services: increases functionality for content in a managed framework; has multiple ways of running tasks for different types of users and scenarios; makes it possible to add new code without a rebuild; simplifies extending DSpace functionality; and with scripting lowers the bar even more.

Today was the second day of the Open Repositories conference, and the big highlight of the day for me was the panel discussion on using Fedora as a storage and service layer for DSpace. This seems like such a natural fit, but with two pieces of complex software the devil is in the details. Below that summary is some brief paragraphs about some of the 24×7 lightning talks.

Fedora inside DSpace

Here are some of the questions and responses from the panel. I hope I’m representing everyone’s views as intended; comments and clarifications are welcome.

Will adding DSpace on Fedora make DSpace even more complex? Matt: It is an opportunity to revisit the design assumptions “and clean up your work.” This is an example of where the transition will create the opportunity to tidy up the complexity and make DSpace simpler while gaining the flexibility of Fedora. Mark: The complexity of DSpace isn’t in its content model, it is in trying to use the existing content model to do things DSpace wasn’t designed to do. For instance, Islandora has more complex atomistic content models, particularly with science data, than DSpace’s content model. Bradley: Acknowledged that there is a risk here, but “if it becomes more complex we are doing it wrong.”

There is concern from the DSpace because it may have to change to accommodate Fedora, but Fedora will not need to be changed. Bradley: New ideas are sources of concern. It is difficult to categorize DSpace developers as a whole. Any time you move a major application on top of another component you may find the underlying APIs need to change.

At what level do we align DSpace and Fedora? Do we really need an intermediary format (AIP)? Bradley: If all we do is find a way to graft the existing DSpace workflows onto a Fedora that is very specific to DSpace and can’t be used with other Fedora tools, then we haven’t moved very far. The end goal is to find those places where DSpace is not formally specified enough and get it formally specified. Mark: Islandora hooks Drupal into Fedora. 80%-90% of the time we work with the Drupal-Fedora API — a simple PHP wrapper around Fedora API. It transforms calls into appropriate actions into Drupal. Another option comes from an Italy project that created a synchronization of Fedora objects and Drupal Nodes; it copies information from Fedora into the Drupal RDBMS. Other applications like Omeka have a Fedora plugin. DiscoveryGarden has also looked at things like WordPress with Fedora underneath. As repository services become more intelligent about microservices it would take even less time to make these integrations.

What benefit does Fedora receive? Richard: For the Fedora community, DSpace alignment would provide a rich IR content model for Fedora. Ryan: DSpace was designed as an IR and nothing else; it has that at its core. The problems that people have with DSpace are when people try to make it do something outside that vision. Having a Fedora repository and have a DSpace interface for those IR use cases and something like Hydra or Islandora for people using those use cases. Matt: There has never been a large cohesion of IR workflow in Fedora; having this workflow satisfy the IR use case. Bradley: DSpace with Fedora inside is a repackaging with a slightly different set of existing components. DSpace across its lifetime has tried to become more modular; integration with Fedora will make this clearer. Mark: Would agree that one of the main things DSpace brings to Fedora is the workflow tool and also the back-end data transformation workflows. But he has also never been a fan of the DSpace workflow because the staunch requirement to fill out a lot of metadata is a mistake. Working with science data, researchers want to ingest 100K microscope images without metadata then go back and add metadata with time. Ryan: (Agreeing) Some of the requirements of the native installation of DSpace is difficult to work with in other use cases. Started with configuring the workflow as much as could be done with config file, but then created a new workflow process that still used many of the underlying tools.

The way I think of the motivation is that DSpace on Fedora will have the same easy setup with access to the underlying APIs for customization. Bradley: Yes — that is an aspirational goal. The practical realities mean that we will have to take steps there one at a time. And given the time scale the question comes whether we will get there before we decide to do something else. One of them — sort of unsaid — is to take a look at EPeople and see how that would migrate. Ryan: Unlocking the data is one thing, but unlocking the underlying datastreams as a API. DSpace storage API is opaque. You can rebuild everything from the underlying storage. [Ryan also calls out an old DLTJ post: A key advantage of DSpace with Fedora inside. "Why Fedora? Because You Don’t Need Fedora".]

Lightning Talks

Mark Phillips talked about a PREMIS Event Service. He needed a way to log events that occur during the life of digital objects (virus checks, ingest, fixity check, replication) — when they occur and their outcomes. So he built a microservice based on AtomPub. Each repository component sends outcomes of an event to this service, a central event collector. Uses the PREMIS Event and Agent Modules. Events metadata include: event_identifier, event_type, event_timestamp, event_outcome, outcome_detail, agent_identifier, object_identifier, and event_detail. Agent metadata includes details about the software, human or organization triggering the event. An AtomPub feed for each object returns all events for that object. The system includes a basic search interface to see all events of a particular type and enables a feed to be set up on those searches. There is the ability to harvest all events via OAI-PMH and Atom. It is built with Django and Python and will be release on the MetaArchive Google Code page.

Peter Sefton talked about The Fascinator & Fedora Commons: A Toolkit Tour. Fascinator is a java-based platform targeting repository solutions. It is open source (GPL), a plugin-based platform, and highly customizable. It was first used as an aggregator of data from various sources into a discovery service. They tried doing the same thing on the desktop computer (something a researchers could put on a personal computer, index data, group/describe it). He thinks of the process as a conveyor belt: harvest digital objects, transform them, store them, index them and find them. Harvest: draw into your digital ecosystem files, databases, online resources. Transform: be ready to present and share. Real web stuff — not PDFs. Video and image previews. Multiple renditions for a multi-platform world. Storage: Store originals and their friends; basic filesystem storage or use Fedora Commons. Index: Apache SOLR index. Find: Faceted search interface. Web previews (turn Word into HTML and PDF for preview, same for video transcoding). Easily customizable UI (Jython and Velocity). It is used by REDDBOX, Mint (described at a session yesterday), and a library of university policies (policies sent from Microsoft Sharepoint and transformed into HTML and PDF).

Rich Rogers talked about Publishing Large, Data-Rich Collections on the Web with Exhibit3. We collect and we like to share what we collect. Nowadays we live on the web, so how can we share out collections there? Exhibit is all I need to publish my collection to the web: no backend database, no server application; it will even convert a spreadsheet into usable data on the web. Originally created by the SIMILE Project at MIT, Exhibit is an entire data publication platform with ‘list’ and ‘tabular. views. There is also a rich library of additional views. If temporal data, scrollable interactive timeline. If geospatial data, interactive mapping displays. Numerical data, scatter or time plots. It is installable by HTML configuration and uses a browser-resident RDF database in JSON.

Geri Ingram and Carol Godby talked about Sustaining Collaboration Among Open Access Repositories focused on the WorldCat Digital Collection Gateway. WorldCat broadens exposure to digital object collections; end users click through to the hosting repository server from WorldCat.org. Metadata from OAI-PMH-compliant repositories are regularly harvested to WorldCat through the Gateway; digital objects remain on local repository server. The gateway includes a translation service that allows repository managers to create mappings from Dublin Core (or selected other metadata schemas) to MARC.

This week I am attending the Open Repositories conference in Austin, Texas, and yesterday was the second preconference day (and the first day I was in Austin). Coming in as I did I only had time to attend two preconference sessions: one on the integration — or maybe “invasion” of the Spring Framework — into DSpace and one on the introduction of the DuraCloud service and code.

DSpace and the Spring Framework

I came into the session more familiar with the Spring Framework than with the DSpace code, so I found the session to be a good introduction to some of the DSpace concepts even though I wasn’t the target audience. (I imagine the target audience was someone familiar with the DSpace code wanting to learn about the Spring Framework.) Thanks, Mark, for putting up the web tutorial and walking through it during the preconference session.

DuraCloud Introduction

The session showed the web-based administration interface and the supporting tools for integrating a DSpace repository and a Fedora repository into DuraCloud. Attendees were also given access to a command-line Java application that could be used to upload content into a DuraCloud instance, although sadly it wasn’t demonstrated during the preconference session. (Perhaps I’ll try it out on the sly later with the DuraCloud credentials they gave us at the session…) In addition to the functinality being built into DSpace and Fedora there will be REST-based code libraries for Java, PHP and Python — meaning that any developer could write code to make use of DuraCloud with any repository platform. The whole DuraCloud application is going to be released into open source under the Apache 2.0 license as part of the efforts to create a community using the code and encourage others to write new services for DuraCloud. This is something I’m going to keep watching; I’ve already signed up for a preview account for when the beta is released later this month.

Thanks also to DuraSpace for sponsoring the evening reception after the preconfernce session at the University of Texas Club.

My place of work, OhioLINK, is part of a larger group called the Educational Technology Division of the University System of Ohio. In that capacity, we’re seeking two senior repository software developers to work in our downtown Columbus, OH, office.

The position description can be a little tricky to get to — the Ohio State University jobs website does not allow deep linking into job descriptions — so I’m reproducing the entire description here:

Position Information

Number of Positions Available 1     Both current Ohio State employees and the general public may apply for this unclassified professional position.       University Title Senior Systems Manager-Not Sap   Working Title Sr. Repository Developer   Department Office of Research-OARnet   Department Location Columbus   Requisition Number 347544 [Also requisition number 347545]  Summary of Duties Supports software development operations for Ohio Academic Resources Network (OARnet), in collaboration with the Chancellor for the Ohio Board of Regents (OBR) and the University System of Ohio (Education Technology Division), in accordance with university policies, goals, and objectives; participates in regular operation of SCRUM-based software development team; identifies project development requirements in conjunction with stakeholders, including Product Owner, community representatives and SCRUM team lead; develops technical solutions to meet the business objectives of product requirements in accordance with established OARnet/OBR software development standards; divides technical solutions into component-level features; assigns, monitors, and reviews component-level feature development, testing and integration tasks performed by team members; serves as technical SME for application development environment; performs investigation and tracking of industry trends and exploration of advanced technologies; serves as an expert consultant within and outside OARnet, and significant participation in advising and planning committees and task forces; designs, plans, and coordinates development/construction of systems; serves as a mentor to other development associates.   Additional Information Successful completion of a background check required.   Required Qualifications Bachelor’s Degree in computer & information science or an equivalent combination of education and experience; extensive (5 years) Java development experience involving DSpace/Manakin, Cocoon, XML/XSLT and HTML/CSS site creation; considerable experience (3 years) with JBoss Application Server; considerable experience (3 years) with Linux/Unix, Perl, shell scripting; experience (1 year) with Log4j, JUnit, Maven and Apache Commons.   Desired Qualifications Master’s of Library Science degree; experience with digital archive projects including metadata schema creation.   Target Salary $76,000 – $84,000 Annually   Job Category Information Technology (IT)   Job Appointment (FTE%) 100%   Full/Part Time Full Time   Temporary or Regular Regular   Posting Start Date 10-09-2009   Posting End Date 10-25-2009   Does this position accept online applications? Yes   Faculty Application Instructions

OARnet is a constituent of the University System of Ohio Educational Technology Division, and OARnet’s administrative agent is Ohio State University. To apply for the position, go to Ohio State University’s Job Site, select “Search Postings” and use either requisition number 347544 or 347545.

This is a textual representation of a lightning talk done on Feb 26th at Code4Lib 2008. When the video of the talk is up (thanks, Noel!) I’ll link it here, too. The video is now available, and that article includes an update on progress since the this article was posted.

OhioLINK has a collection of JPEG2000 images as an access format that were generated for use in our DLXS-based content system. We are in the process of migrating those collections to DSpace and were looking for a mechanism to leverage the existing JPEG2000 files and not have to generate new derivatives. We are also considering the use of JPEG2000 as a preservation format, and would find it attractive to use the same image format for both access copies and preservation copies. We looked at Zoomify, but to perform its scaling function it generates JPEG tiles at several resolutions and storing those tiles can triple or quadruple disk space requirements. Or, one could use the ‘enterprise’ version of Zoomify and its proprietary PFF format or the equally proprietary MrSID format. We didn’t want to be locked into either of these scenarios. Our solution is to create a web application that mimics the directory-of-JPEG-tiles solution, but to dynamically generate the tiles our of a JPEG2000 master.

The free version of Zoomify reads JPEG tiles out of a directory structure that looks like this:

The shim mimics that directory structure. It parses the URL of the request and dynamically creates the appropriate JPEG tile (or metadata file) out of the JPEG2000 image.

The Code

The source code is available from the OhioLINK DRC source code repository (Subversion access). We plan to integrate it into DSpace 1.5 as part of the Ohio Digital Resource Commons, and I may create a Fedora disseminator to serve up the tiles as well.

Thanks go out to Keith Gilbertson and John Davison on the OhioLINK staff for their help in making this work as well as Stu Hicks and François d’Erneville for being a sounding board for these ideas.

The text was modified to remove a link to https://drc-dev.ohiolink.edu/browser/j2kTileRenderer/trunk on January 13th, 2011.

The text was modified to update a link from http://code4lib/conference/2008 to http://code4lib.org/conference/2008 on January 28th, 2011.

The text was modified to update a link from http://www.kakadusoftware.com/Purchasing.html to http://www.kakadusoftware.com/index.php?option=com_virtuemart&Itemid=19&vmcchk=1&Itemid=19 on January 28th, 2011.

Richard Rodgers presented this talk based on the work of he and MacKenzie Smith in the Digital Library Research Group at MIT. The original abstract of the presentation was:

Many questions are raised as previously unreachable digital content is found in and among new repositories–is each repository an island or a separately searchable resource? SIMILE (Semantic Interoperability of Metadata and Information in Unlike Environments) has developed an extensive ‘tool chain’ for gathering and manipulating data assets. Richard Rodgers and MacKenzie Smith, MIT, will demonstrate how tools developed by the SIMILE project can be used as powerful instruments for the federation, discovery, exploration, and curation of metadata.

The mission of the SIMILE suite of projects is to build tools for data interoperability. Dealing with heterogeneous metadata in repository design and use is a complex challenge, and the position that SIMILE takes is that no matter what single metadata scheme you select at the start of a repository project, one runs into trouble as subsequent collections come in with other semantically-rich collection-specific metadata schemes. This puts the repository designer between a rock (semantic reduction and loss because metadata crosswalks are “lossy”) and a hard place (one has serious scalability problems — does one construct separate queries for each metadata schema — if all of the uniqueness of the metadata coming to the repository is embraced.

SIMILE uses RDF and other semantic web technologies contributing to the solution of heterogeneous metadata problem. Statements about documents are inherently more mixable than the documents themselves, and RDF is a more mixable language than trying to harmonize metadata. RDF represents data as a graph, not as a table (RDBMS) or tree (XML). The tools created by SIMILE fall into four categories:

• Convert: RDFizers (for converting structured data to RDF, such as MARC into RDF), Babel
• Visualize: Gadget (a data graph viewer for XML; it constructs all of the XPATHS in a document and projects them along with frequency of occurrences as a way to look at XML documents from a structural level), Welkin (same as Gadget except for RDF)
• Browse: Longwell (see below), Piggy Bank (Firefox plugin; RDFizes an HTML page by using JavaScript to scrape metadata from websites and putting it into your personal repository), Semantic Bank (a way to publish RDF and create communities of RDF content)
• Lightweight UI: Timeline, exhibit widgets (highly interactive faceted browse displays that divide the processing between client and server through the use of AJAX)

Richard went into detail on Longwell, a faceted browser web application. Using a RDF triple-store backend (Sesame), Longwell presents data in a configurable, extensible user interface. One of the interesting technologies it uses is the W3C-defined Fresnel Display Vocabulary. There is not anything equivalent to CSS in the RDF world to a layout styling language. The W3C thought they could spur development of RDF tools if there was a way of expressing a display vocabulary in RDF, hence the Fresnel Display Vocabulary. Longwell has been embedded into DSpace as an optional advanced search engine called “DWell”.

Update at 20070129T1646 — also see Dorothea Salo summary:

Source: Caveat Lector » SIMILE
Address : http://cavlec.yarinareth.net/archives/2007/01/28/simile/
Date Visited: Mon Jan 29 2007 16:43:35 GMT-0500 (EST)

This is a summary of a presentation by James L. Hilton, Vice President and CIO of University of Virginia, at the opening keynote session of Open Repositories 2007. I tried to capture the esessence of his presentation, and omissions, contradictions, and inaccuracies in this summary are likely mine and not that of the presenter.

Setting the stage

This is a moment in which institutions may be willing to invest in open source development in a systematic way (as opposed to what could currently be characterized as an ad hoc fashion) driven by these factors:

• Fear. Prior to Oracle’s hostile take-over of PeopleSoft, the conventional wisdom of universities was that they needed to buy their core enterprise applications rather than build them. In doing so, they sought the comfort of buying the security of a leading platform. Oracle’s actions diminished that comfort level. Blackboard acquisition of WebCT and lawsuit against a competitor does not help either.
• Disillusionment and ERP fatigue. What was largely thought to be an outsourced project was found to be an endless upgrade cycle. Organizations need to build entire support units to handle the upgrades for large ERP systems rather than supporting the needs of the users.
• Incredulity — we’re supposed to do what? The application of technology typically has a disruptive impact (cannot predict the end), the stakes are incredibly high (higher education and/or research could be lost in a decade), it tends to be expensive, and the most common survival strategy is to seed many expensive experiments in the hopes that one will be in the right place at the time the transition needs to happen. The massive investment anticipated for technology to support academic computing (libraries, high-performance clusters, etc) will pale in comparison to the investment in administrative computing.
• Rising tide of collaboration. This is a realization that the only way to succeed is through collaboration. To paraphrase Hilton, “In the new order it will be picking the right collaborative partners where the new competitive advantage will come from.”

Distinctions

Hilton offered these definitions and contrasts as a way to frame the rest of his discussion. First was Open or “free” software. Free as in beer, or free as in “adopt a puppy.” The software comes with the ability to do with as you want with the code, not just the ability to use the code. They he defined the term License as a contract — what ever you agree to you are bound to; you cannot use copyright law to protect you. The rules and conditions that are applied to the software do matter.

Lastly, he talked about Copyleft or “viral” licensing. There are different interpretations of “open” in open source. “Copyleft” has come to mean that code should be freely available to be used and modified, and it should never by locked up. GPL is an example. This is often called “viral” because if you include software with this license in any other work that is released, the additional software must be released under the same license. This is seen by some as valuable because it prevents open source from being encircled by proprietary code. Copyleft is contrasted with an “open/open” license — you can do whatever you want to do with a code under this license. An “open/open” license places no restrictions on what users do with code in derivative software packages.

Case Study — Michigan’s Sakai Sojourn

Hilton briefly described why UMich went down the Sakai path in 2001-2002:

• Legacy system with no positive trajectory forward. It could never be released into open source; all of the development would have to be carried on UMich’s shoulders forever.
• Saw market consolidation in CMS. This was mostly evident in the commercial sector with Blackboard and WebCT being the dominant choices. They had concerns about the cost of licenses in this environment down the road.
• Saw the potential of tapping the institution’s core competencies and starting a virtuous cycle of development, teaching and research. Or, put another way, they didn’t want core competencies in teaching and research held hostage to a commercial development cycle.
• Strategic desire to blur the distinction between the laboratory/classroom and between knowledge creation/digestion. They realized that the functions of a research support tool and a course support tool were pretty much the same under different skins, and they sought to blur that distinction even more.
• NRC report and the need for collaboration. UMich was willing to fund the project two years internally but knew after that need to find collaborative partners by the fifth year in order to be declared a success.
• A moment of time opportunity that synchronized the development process of several partners with funding provided by the Mellon Foundation.

There were also specific goals for the Sakai project. The new system had to replicate the functionality of existing course and research collaboration environments. They also wanted experience in finding partners willing to collaborate. Hilton said, “Sakai was/is at least as interest from a collaboration perspective as it is from the technology perspective.” Bringing together disparate organizations with different beliefs on how things should be done is a challenge. Additionally, they wanted to get better as an institution at discerning open source winners; it shouldn’t be like a lottery. Lastly, they wanted to implement software parts that were not built at UMich. Each partner institutions committed to implementing the same thing even if wasn’t built at that institution. This is tough to do, but they knew they needed to do it for their own good in the long run.

What happened? Not only did the original partners show up, but the community came, too. Even more interesting was that the community was formed with dues-paying members — even in a world where the software is free. It became a vibrant community, too, with a conference every six months. Sakai was released under an open-open license model, and corporate partners showed up as well (selling support services, or hosting services, or hardware for the software). The software did grow up and left its home; a separate foundation now holds the intellectual property of the code (originally partners assigned copyright to UMich). They also positioned Sakai to be a creditable threat to the commercial entities in order to force them to the standards table.

Takeaway lessons that generalize to open source development

First, the benefits of open source development.

• destiny control (but only when you really need to drive). having the control is not always a good thing. Is it worth the effort? Is the project core to the institution’s mission? (Does it directly support scholarship and teaching?)
• builds community and camaraderie (in the case of Sakai, both locally at UMich and internationally)
• unbundles software ownership and its support. inspires more competition in the implementation and support space.
• community source provides institutions an opportunity to leverage links between open source, open access and culture of the academy/wider world (a.k.a. put up or shut up)

Then, the challenges of open source development.

• Guaranteeing clean code (IP) is hard (read as “impossible”). A certain amount of faith about the code they get and there needs to be consideration for mitigating risks.
• Figuring out who is authorized to license institutionally-owned code is challenging and then you have to convince them to give it away. No one in the institution typically has been appointed or given the authority to release code. One of the things that the sakai licensing discussions highlighted was institutional differences in requirements and aesthetics.
• Patent quagmire always looming. How do you know your software is not infringing? How do you make sure you don’t inadvertently give away all institution patents? Be careful when looking at licenses from an institutional perspective versus an individual perspective.
• There is also the inevitable lawsuit risk. Or, as your counsel might say to you, “Let me get this straight, we can get sued but there’s no one we can sue.”

Then, some discoveries that they made along the way.

• An open source project not a silver bullet. The commitment to build rather than buy must align with institutional priorities and competencies; it is not right for every project/application.
• Licensing does matter; it is a contract: whatever you stick in its rules is what sticks. There are probably have too many open source license options and some sort of standardization is needed. Also keep in mind that if you release something under an open/open license, you can’t include any copyleft components.
• Communities don’t just happen, they require: specific shared purpose (when visions vary, or when they change, collaborations struggle); and governance (e.g., separate board with dedicated developers sitting between institutions). Cooperation (“I won’t hurt you if you don’t hurt me”) is not collaboration.
• Open (community) source requires real project discipline. “It is as spontaneous as a shuttle launch.” Along the way one needs to learn to balance pragmatics and ideals. One also needs to learn to trust your partners. “It really requires learning to let go.” Letting go, and having the community make the decisions, may be the quickest path to efficiency.

Reflection on open/community source for repositories

Repositories are at the center of everything at the institution. It connects with the library, with the presses/scholarly publishing operation, with classroom teaching, with the laboratory, and with the world. It is a core piece of of infrastructure for the university of the 21st century. As institutions, we need to make sustaining investments in our repositories.

Hilton sees three different approaches to “community” in the existing projects:

• dspace: community of user/developers. The come together to talk about what they want to do, write code, and support each other. Clearly there are enthusiastic users as developers.
• eprints: appears as like a vendor talking with customers wanting the community help shape the direction.
• fedora: in transition from a combination of the previous two models moving towards a Sakia-like model. it will require institutions to make commitments to it.

In the end, Hilton asked some thought-provoking questions. Is now the time for institutional investment in open/community source? Will a coherent community (or communities) emerge in ways that are sustainable? — is there a shared vision?

The text was modified to update a link from http://www.virginia.edu/vpcio/bio.html to http://www.virginia.edu/vpcio/biography.html on January 19th, 2011.

Open Repositories 2007 is coming up next year, and it looks to be an interesting meeting. The first day is open user group meetings for DSpace, Fedora, and Eprints, followed by general conference sessions that cover issues that cut across all of the open repository systems. This year, the user groups will partition their programs into Plenary, Technical Issues, and Management Issues and the partitions will be staggered so that IT managers can attend all plenary sessions, technical staff can attend all technical sessions, etc.

The call for participation for the general conference has gone out. Its Program Committee is seeking submissions in the form of an extended abstract of no more than 500 words by October 2, 2006. The contributions must be written in English and should be double spaced. The Program Committee will select relevant submissions. Selected speakers will receive an email by November 6, 2006 with guidelines for their presentation. Presentations will be limited to 20 minutes, plus 10 minutes for questions.

This looks to be a really good meeting. You can track it on HitchHikr at http://hitchhikr.com/index.php?conf_id=75.

The text was modified to remove a link to http://openrepositories.org/call on October 29th, 2010.