The snow is falling here in central Ohio, so I’m eager to leave here and head to warm Dallas for ALA Midwinter 2012. I’m looking forward to catching up with colleagues; making new acquaintances; learning the latest thinking on RDA, linked data, and standards activity; and talking about free/open source software in libraries. On the latter point, I encourage you to come see me give an introduction to the newly announced FOSS4LIB site, answer questions, and take feedback on Saturday morning (10:30 to 11:30) or Sunday morning (10:30 to 11:30). (Or, if you are not coming to Midwinter, sign up for one of the free webinar sessions later in January and February.)

ALA is using a new iteration of its scheduler this year, and it keeps getting better and better. This one even allows you to embed your selected schedule as an <iframe> on an arbitrary page. So here is my schedule:

You can follow me on Twitter where I’ll be tweeting about #alamw12. A Twitter mention or direct message is also the best way to get ahold of me while in Dallas.

Safe travels if you are headed to Midwinter, and I hope to run into you there.

The main Open Repositories conference concluded this morning with a keynote by Clifford Lynch and the separate user group meetings began. I tried to transcribe Cliff’s great address as best I could from my notes; hopefully I’m not misrepresenting what he said in any significant ways. He has some thought-provoking comments about the positioning of repositories in institutions and the policy questions that come from that. For an even more abbreviated summary, check out this Storify archive of tweets during his keynote. Then I attended the DSpace track of user group programming, and below there are summaries of plans for DSpace version 1.8 and the new DSpace Curation Services.

Repositories: Major Progress and Open Questions

Instead, the growth of repositories has highlighted critical policy discussions of what the missions of institutions of higher education are supposed to be. Questions such as the responsibility to curate knowledge they create, curate the evidence on which inquiry is based, to disseminate knowledge. These weren’t on the table 10-15 years ago. Now they are central issues for discussion in the leadership of universities. Tied to institutional repositories is the question of open access. The scope of issues goes beyond just open access, though. It reaches into the kind of questions that are now getting traction like getting access to research data and an institution’s role of stewardship and disseminate of research data. Also the creation of open educational resources and institution’s responsibility to disseminate such resources. These questions wouldn’t have emerged without the effort to build out institutional repositories. As soon as you start talking about these questions they demand investments in infrastructure of institutional repositories. So we should take satisfaction in the role that the efforts of deploying institutional repositories have played in advancing these critical policy discussions. These questions have gone unanswered far too long.

That said, there is a danger of confusion of mechanism with policy. We went through a bad period around 2004 when institutional repositories were deployed without knowing what would be put in them. Institutional repositories are services to support policies, not ends to themselves. We have mostly gotten past this and can have the discussion of whether institutionally based repositories are the appropriate tool and when should we build discipline-specific repositories or other kinds of platforms that are not institutionally focused.

He also noted that the question of institutional assets and the balance of faculty control with institutional responsibility is being talked about, if only quietly. The piece of video that Clifford referred to — a talk by Derick Law on how universities have failed in their stewardship responsibilities of research — may be this video from the The Blue Ribbon Task Force on Sustainable Digital Preservation and Access’s National Conversation on the Economic Sustainability of Digital Information (skip to “chapter 2″ of the video) held April 1, 2010 in Washington DC.

Not only have institutional repositories acted as focal point for policy, they have also been a focal point for collaborations. Library and IT collaborations were happening long before institutional repositories surfaced. Institutional repositories, though, have been a great place to bring other people into that conversation, including faculty leaders to start engaging them in questions about dissemination of their work. Also chief research officers; in 1995 if you were a university librarian doing leadership work constructing digital resources to change scholarly communication, would have talked to CIO but may not know who your chief research officer was at that point. That set of conversations, which are now critical when talking about data curation, got their start with institutional repositories and related policies.

Another place for conversation has been those in the university administrations concerned with building public support for the institution. By giving the public a deeper understanding of what the institution contributes to culture, industry, health and science, and connecting faculty to this effort. This goes beyond the press release by opening a public window into the work of the institutions. This is particularly important today with questions of public support for institutions.

That said, there are a number of open questions and places where we are dealing with works-in-progress. Cliff then went into an incomplete and, from his perspective, perhaps idiosyncratic, list of these issues.

Repositories are one of the threads that are leading us nationally and internationally into a complete rethinking of the practice of name authority. While it is a librarian, old fashion concept, but it is converging with “identity management” from IT. He offered an abbreviated and exaggerated example: librarians did name authority for authors of stuff in general in 19th century. In 20th century there was too much stuff, particularly stuff in journals and magazines became overwhelming. So libraries backed off and focused only on books and stuff that went into catalogs; the rest they turned over to indexing and abstracting services. We made a few weird choices like an authority file should be as simple as possible to disambiguate authors rather than be as full as possible, so we had the development of things along side name authority files like the national dictionaries of literary biographies.

For scientific journal literature, publishers followed practices about how obscure author names could be (e.g. just last name and first initial). Huge amounts of ambiguity of “telegraphic author names” results in a horribly dirty corpus of data. A variety of folks are realizing that we need to disambiguate authorship by assigning author identifiers and somehow go back and cleanup the mess in the existing bibliographic data of scholarly literature, especially journal literature. Institutions taking more responsibility for the work of their community, and having to do local name authority all over again. We have the challenge of how to reconnect this activity to national and international files. We also have a set of challenges on whether we want to connect this to biographical resources. It brings up issues of privacy, when do people do things of record, and how much else should come along with building a public biography resource. We also see a vast parallel investment of institutional identity management. Institutions haven’t quite figured out that people don’t necessarily publish with the same name that is recorded in the enrollment or employment systems that the institution manages, and that it would be a good idea to tie those literary names to identity files that the institution manages.

We’re not confident of the kind of ecological positioning institutional repositories among a pretty complicated array of information systems found at a typical large university. Those systems include digital library platforms, course management systems, lecture capture systems, facilities for archiving the digital records of the institution, and platforms intended to directly support active research by faculty. All are evolving at their own rate. It is unclear where the institutional repositories fit, and what are the boundaries around them.

Here is one example. What is the difference between an institutional repository and a digital library/collection? You’d get very different answers from different people. One might be who does the curation, how it is sourced, and how it is scoped. The decisions are largely intellectual. Making this confusing is that you’ll see the same platform for institutional repositories and digital library platforms. We are seeing a convergence of the underpinning platforms.

Another one: learning management systems (LMS). These are virtually universal among institution in the same timeframe that institutional repositories have been deployed. We’ve done a terrible job at thinking about what happens to the stuff in them when the course is over. We can’t decide if it is scholarly material, institutional records, or something else. They are tangled up between learning materials and all of the stuff that populates a specific performance of a course such as quizzes and answers, discussion lists, and student course projects. We don’t have taxonomies and policies here and a working distinction between institutional repositories and learning management systems. It is an unusual institution that has as systematic export from the LMS to an IR.

Lecture capture systems becoming quite commonplace; students are demanding them in much the same way that the LMS was demanded. A lecture capture system may be more universally helpful than an LMS. Lectures being captured for a wide range of reasons, but not knowing why means it is difficult to know whether to keep them and how to integrate them into the institution’s resources.

Another example: the extent to which institutional repositories should sit in the stream of active work. As faculty are building datasets and doing computation with them, when is it time for something to go into an institutional repository. How volatile can content be in the repository? How should repositories be connected or considered as robust working storage? He suspects that many institutional repositories are not provisioned with high-performance storage and network connections, and would become a bottleneck in the research process. The answers would be different for big data sets and small data sets, and we are starting to see datasets that are too big to backup or two big to replicate.

Another issue is that of virtual organizations, the kind of collaborative efforts that span institutions and nations. They often allow relatively low overhead to mobilize researchers to work on a problem, and are becoming commonplace in sciences and social sciences and starting to pop up in the humanities. We have a problem for the rules-of-the-road between virtual organizations and institution-based repositories. It is easy to spin up an institutional repository for a virtual organization, but what happens to it when the virtual organization shuts down. Some of these organizations are intentionally transient; how do we assign responsibility for a world of virtual organizations and map them into institutional organizations for long-term stewardship.

Software is starting to concern people. So much scholarship is tied up now in complicated software systems that we are starting to see a number of phenomena. One is data that is difficult to reuse or understand without the software. Another is the is difficulty surrounding reproducibility — taking results and realizing they are dependent on an enormous stack of software and we don’t have a clear way to talk about the provenance of a result that is based on the stack of software versions that would allow for high-confidence in reproduction of results. We’re doing to have to deal with software. We are also entering an era of deliberate obsolescence of software; for instance, any Apple product that is older than a few years is going to the dustbin and it hasn’t been fully announced or realized so that people can deal with it.

Another place that has been under-exploited is the question of retiring faculty and repositories. Taking inventory of someone’s scholarly collections and migrating it to an institutional framework in an orderly fashion.

How we reinterpret institutional repositories going beyond universities. For example there is something that looks a bit like an institutional repository but has some different things about it that belongs in public libraries or historic societies or similar. This dimension bears exploration.

To conclude his comments he talked about a last open issue. When we talk about good stewardship and preservation of digital materials, there are a couple of ideas that have emerged as we tried to learn from our past stewardship of print scholarly literature. One of these principles is that geographic replication is a good thing; we’re starting to see this in a sense that most repositories are based on some geographically redundant storage system or we’ll see a steady migration towards this in the next few years. A second one is organizational redundancy. If you look at the print work, it wasn’t just that the scholarly record wasn’t in a number of independent locations but also that control was replicated among institutions that were making independent decisions about adding materials to their library collection. Clearly they coordinated to a point, but they also have institutional independence. We don’t know how to do this with institutional repositories. This is also emerging in special collections as they become digital. Because they didn’t start life as published materials in many replicated versions, we need other mechanisms to have curatorial responsibility distributed. This is linked to the notion that it is usually not helpful to talk about preservation in terms like “eternity” or “perpetuity” or life-of-the-republic. It is probably better in most cases to think about preservation in one chunk at a time; an institution making a 20-year or 50-year commitment with a well-structured process at the end. That process includes whether an institution should renew the commitment and if not other interested parties could come in and take responsibility with a well-ordered hand-off. This ties into policies and strategies for curatorial replication across institutions and ways that institutional repositories will need to work together. It may be less critical today, but will become increasingly critical.

In conclusion, Cliff said that he hoped left the attendees with a sense that repositories are not things that stand on their own. That they in fact are mechanism that advance policy in a very complex ecology of systems. In fact, we don’t have our policy act together on many systems adjacent to the repository that leads to issues of appropriate scope and interfaces with those systems. Where repositories will evolve to in the future as we understand the role of big data is also of interest.

DSpace 1.8

SWORD is a lightweight protocol for depositing items between repositories; it is a profile of the Atom Publishing Protocol. At the current release, DSpace has be able to accept items; the planned work for 1.8 will make it possible to send items. Some possible use cases: publishing from a closed repository to an open repository, sending from the repository to a publisher, from the repository to a subject-specific service (such as arXiv), or vice versa. The functionality was copied from the Swordapp demo. It supports SWORD v1 and only the DSpace XMLUI. A question was ask about whether the SWORD copy process is restricted to just the repository manager? The answer was that it should be configurable. On the one hand it can be open because it is up to the receiving end to determine whether or not to accept it. On the other hand, a repository administrator might want to prevent items being exported out of a collection.

MIT has rewritten the Creative Commons licensing selection steps. It uses the Creative Commons web services (as XML) rather than HTML iframes, which allows better integration with DSpace. As an aside, the Creative Commons and license steps have been split into two discrete steps allowing different headings in the progress bar.

The DSpace Community Advisory Team prioritized issues to be addressed by the developers, and for this release they include JIRA issue DS-638 for virus checking during submission. The solution invokes the existing Curation Task and requires Clam AV antivirus software to be installed. It is switched off by default and is configured in submission-curation.cfg. Two other issues that were addressed are DS-587 (Add the capability to indicate a withdrawn reason to an Item) and DS-164 (Deposit interface), which was completed as the Google Summer of Code Submission Enhancement project.

Thanks to Bojan Suzic in his Google Summer of Code project, DSpace has had a REST API. The code has been publicly available and repositories have been making use of it, so the committers group want to get it into a finished state and include it in 1.8. There is also work on an alternative approach to a REST API.

DSpace and DuraCloud was also covered; it was much the same that I reported on earlier this week, so I’m not repeating it here.

From the geek perspective, the new release will see increasing modularization of the codebase and more use of Spring and the DSpace Services Framework. The monolithic dspace.cfg will be split up into separate pieces; some pieces would move into Spring config while other pieces could go into the database. It will have a simplified installation process, and several components that were talked about elsewhere at the meeting: WebMVC UI, configurable workflow, and more curation tasks.

Introduction to DSpace Curation Services

Some built-in tasks are to validate metadata against input forms (halts on task failure), count bitstreams by format type, virus scan (uses external virus detection service), on ingest (the desired use case), and the replication suite of tasks for DuraCloud. Other tasks: link checker and 11 others (from Stuart Lewis and Kim Shepherd), format id with DROID (in development), validate/add/replace metadata, status report on workflow items, filter media in workflow (proposed), and checksum validation (proposed).

What does this mean for different users? As a repository or collection manager, it means new functionality — GUI access without GUI development: curation, preservation, validation, reporting. As a developer: rapid development, and deployment of functionality without rebuilding or redeploying the DSpace instance.

The recommended Java development environment for tasks is with a package outside of dspace-api. Make a POM with dependency on dspace-api, especially /curate. Required features of the task are a constructor with no arguments to support loading as a plugin and that it implements the CurationTask interface or extends the AbstractCurationTask class. Deploy it as a JAR and configure (similar to a DSpace plugin)

There are some Java annotations for Curation Task code that are important to know about. Setting @Distributive means that the task is responsible for handling any contained DSpace objects as appropriate. Otherwise the default is to have the task executed across all contained objects (subcommunities, collections, or items). Setting @Suspendable means the task interrupts processing when first FAIL status is returned. Setting @Mutative means the task makes changes to target objects.

Invoking tasks can be done several ways: from the web application (XMLUI), the command line, from workflow, from other code, or from a queue (deferred operation). In the case of the workflow, one can target the action of the task at anywhere in the workflow steps (e.g. before step 1, step 2, step 3 or at item installation). Actions (reject or approve) are based on tasks results, and notifications are sent by e-mail.

A mechanism for discovering and sharing tasks doesn’t exist yet. What is needed is a community repository of tasks. For each task what is needed is: a descriptive listing, documentation, reviews/ratings, link to source code management system, and link to binaries applicable to specific versions.

With dynamic loading with scripting languages in JSR-223, it is theoretically possible to create Curation Tasks in Groovy, JRuby, Jython, although the only one Bill has been able to get to work so far has been Groovy. Scripting code needs a high level of interoperability with Java, and must implement the CurationTask interface. Configuration is a little bit different: one needs a taskcatalog with descriptors for language, name of script, and how the constructor is called. Bill demonstrated some sample scripts.

In his conclusion, Bill said that the new Curation Services: increases functionality for content in a managed framework; has multiple ways of running tasks for different types of users and scenarios; makes it possible to add new code without a rebuild; simplifies extending DSpace functionality; and with scripting lowers the bar even more.

Today was the second day of the Open Repositories conference, and the big highlight of the day for me was the panel discussion on using Fedora as a storage and service layer for DSpace. This seems like such a natural fit, but with two pieces of complex software the devil is in the details. Below that summary is some brief paragraphs about some of the 24×7 lightning talks.

Fedora inside DSpace

Here are some of the questions and responses from the panel. I hope I’m representing everyone’s views as intended; comments and clarifications are welcome.

Will adding DSpace on Fedora make DSpace even more complex? Matt: It is an opportunity to revisit the design assumptions “and clean up your work.” This is an example of where the transition will create the opportunity to tidy up the complexity and make DSpace simpler while gaining the flexibility of Fedora. Mark: The complexity of DSpace isn’t in its content model, it is in trying to use the existing content model to do things DSpace wasn’t designed to do. For instance, Islandora has more complex atomistic content models, particularly with science data, than DSpace’s content model. Bradley: Acknowledged that there is a risk here, but “if it becomes more complex we are doing it wrong.”

There is concern from the DSpace because it may have to change to accommodate Fedora, but Fedora will not need to be changed. Bradley: New ideas are sources of concern. It is difficult to categorize DSpace developers as a whole. Any time you move a major application on top of another component you may find the underlying APIs need to change.

At what level do we align DSpace and Fedora? Do we really need an intermediary format (AIP)? Bradley: If all we do is find a way to graft the existing DSpace workflows onto a Fedora that is very specific to DSpace and can’t be used with other Fedora tools, then we haven’t moved very far. The end goal is to find those places where DSpace is not formally specified enough and get it formally specified. Mark: Islandora hooks Drupal into Fedora. 80%-90% of the time we work with the Drupal-Fedora API — a simple PHP wrapper around Fedora API. It transforms calls into appropriate actions into Drupal. Another option comes from an Italy project that created a synchronization of Fedora objects and Drupal Nodes; it copies information from Fedora into the Drupal RDBMS. Other applications like Omeka have a Fedora plugin. DiscoveryGarden has also looked at things like WordPress with Fedora underneath. As repository services become more intelligent about microservices it would take even less time to make these integrations.

What benefit does Fedora receive? Richard: For the Fedora community, DSpace alignment would provide a rich IR content model for Fedora. Ryan: DSpace was designed as an IR and nothing else; it has that at its core. The problems that people have with DSpace are when people try to make it do something outside that vision. Having a Fedora repository and have a DSpace interface for those IR use cases and something like Hydra or Islandora for people using those use cases. Matt: There has never been a large cohesion of IR workflow in Fedora; having this workflow satisfy the IR use case. Bradley: DSpace with Fedora inside is a repackaging with a slightly different set of existing components. DSpace across its lifetime has tried to become more modular; integration with Fedora will make this clearer. Mark: Would agree that one of the main things DSpace brings to Fedora is the workflow tool and also the back-end data transformation workflows. But he has also never been a fan of the DSpace workflow because the staunch requirement to fill out a lot of metadata is a mistake. Working with science data, researchers want to ingest 100K microscope images without metadata then go back and add metadata with time. Ryan: (Agreeing) Some of the requirements of the native installation of DSpace is difficult to work with in other use cases. Started with configuring the workflow as much as could be done with config file, but then created a new workflow process that still used many of the underlying tools.

The way I think of the motivation is that DSpace on Fedora will have the same easy setup with access to the underlying APIs for customization. Bradley: Yes — that is an aspirational goal. The practical realities mean that we will have to take steps there one at a time. And given the time scale the question comes whether we will get there before we decide to do something else. One of them — sort of unsaid — is to take a look at EPeople and see how that would migrate. Ryan: Unlocking the data is one thing, but unlocking the underlying datastreams as a API. DSpace storage API is opaque. You can rebuild everything from the underlying storage. [Ryan also calls out an old DLTJ post: A key advantage of DSpace with Fedora inside. "Why Fedora? Because You Don’t Need Fedora".]

Lightning Talks

Mark Phillips talked about a PREMIS Event Service. He needed a way to log events that occur during the life of digital objects (virus checks, ingest, fixity check, replication) — when they occur and their outcomes. So he built a microservice based on AtomPub. Each repository component sends outcomes of an event to this service, a central event collector. Uses the PREMIS Event and Agent Modules. Events metadata include: event_identifier, event_type, event_timestamp, event_outcome, outcome_detail, agent_identifier, object_identifier, and event_detail. Agent metadata includes details about the software, human or organization triggering the event. An AtomPub feed for each object returns all events for that object. The system includes a basic search interface to see all events of a particular type and enables a feed to be set up on those searches. There is the ability to harvest all events via OAI-PMH and Atom. It is built with Django and Python and will be release on the MetaArchive Google Code page.

Peter Sefton talked about The Fascinator & Fedora Commons: A Toolkit Tour. Fascinator is a java-based platform targeting repository solutions. It is open source (GPL), a plugin-based platform, and highly customizable. It was first used as an aggregator of data from various sources into a discovery service. They tried doing the same thing on the desktop computer (something a researchers could put on a personal computer, index data, group/describe it). He thinks of the process as a conveyor belt: harvest digital objects, transform them, store them, index them and find them. Harvest: draw into your digital ecosystem files, databases, online resources. Transform: be ready to present and share. Real web stuff — not PDFs. Video and image previews. Multiple renditions for a multi-platform world. Storage: Store originals and their friends; basic filesystem storage or use Fedora Commons. Index: Apache SOLR index. Find: Faceted search interface. Web previews (turn Word into HTML and PDF for preview, same for video transcoding). Easily customizable UI (Jython and Velocity). It is used by REDDBOX, Mint (described at a session yesterday), and a library of university policies (policies sent from Microsoft Sharepoint and transformed into HTML and PDF).

Rich Rogers talked about Publishing Large, Data-Rich Collections on the Web with Exhibit3. We collect and we like to share what we collect. Nowadays we live on the web, so how can we share out collections there? Exhibit is all I need to publish my collection to the web: no backend database, no server application; it will even convert a spreadsheet into usable data on the web. Originally created by the SIMILE Project at MIT, Exhibit is an entire data publication platform with ‘list’ and ‘tabular. views. There is also a rich library of additional views. If temporal data, scrollable interactive timeline. If geospatial data, interactive mapping displays. Numerical data, scatter or time plots. It is installable by HTML configuration and uses a browser-resident RDF database in JSON.

Geri Ingram and Carol Godby talked about Sustaining Collaboration Among Open Access Repositories focused on the WorldCat Digital Collection Gateway. WorldCat broadens exposure to digital object collections; end users click through to the hosting repository server from WorldCat.org. Metadata from OAI-PMH-compliant repositories are regularly harvested to WorldCat through the Gateway; digital objects remain on local repository server. The gateway includes a translation service that allows repository managers to create mappings from Dublin Core (or selected other metadata schemas) to MARC.

Today was the first main conference day of the Open Repositories conference in Austin, Texas. There are 300 developers here from 20 countries and 30 states. I have lots of notes from the sessions, and I’ve tried to make sense of some of them below before I lose track of the entire context.

The meeting opened with the a keynote by Jim Jagielski, president of the Apache Software Foundation. He gave a presentation on what it means to be open source project with a focus on how Apache creates a community of developers and users around its projects.

Slides 50 and 51 of "Open Source: It's Not Just for IT Anymore"

• Give Me Credit (“You can use, modify and redistribute my code in your product but give me credit”): Apache License, Berkeley License, and MIT License
• Give Me Fixes (“You can use, modify and redistribute my code in your product but give me the source for any fixes you make to it.”): Mozilla Public License, Eclipse Public License, and GNU Library or “Lesser” General Public License
• Give Me Everything (“You can use, modify and redistribute my code in your product but give me your entire product’s source code.”): GPL

He also explained why community and code are at peer levels in Apache. One is not possible without the other; it takes an engaged community to create great code and great code can only be created by a healthy community. He also described how the primary communications tool for projects is not new-fangled technologies like wikis and conference calls and IRC. The official record of a project is its e-mail lists. This enables the broadest possible inclusion of participants across many time zones and the list archives enable people to look into the history of decisions. If discussions take place in other forums or tools, the summary is always brought back to the e-mai list.

Jim’s concluding thoughts were a great summary of the presentation, and I’ve inserted them in on the right.

I missed the first concurrent session of the day due to work conflict, so the first session I went to was the after lunch 24×7 presentations. That is no more than 24 slides in no more than seven minutes. I like this format because it forces the presenters to be concise, and if the topic is not one that interests you it isn’t long until the next topic comes up. The short presentations are also great for generating discussion points with the speakers during breaks and the reception. Two of these in particular struck a cord with me.

The first was “Technology Trends Influencing Repository Design” by Brad McLean of DuraSpace. His list of four trends were:

1. Design for mobile, not just PCs. The model of a mobile app — local computation and page rendering backed by web services for retrieving data — is having several impacts on design: a reinforcement of the need for lightweight web services and UIs; accounting for how screen size has shrunk again; and having a strategy for multi-platform apps will become critical.
2. More programming language(s) than you need/want. Java, Python, Ruby, Scala, LISP, Groovy, JavaScript and the list goes on. This proliferation of languages has forced looser coupling between components (e.g. a JavaScript based script can consume data from and write data to a Java-based servlet engine). The implications he listed for this are that it is even clearer that true integration challenges are in the data modeling and policy domains; harder to draw neat boxes around required skill sets; and that you might lose control of your user experience (and it might be a good thing).
3. Servers and clusters. Clusters are not just for high-performance computing and search engines anymore. Techniques like map/reduce are available to all. He said that Ebay was the last major internet company to deploy its infrastructure on “big iron” but he didn’t attribute that statement to a source. (Seems kind of hard to believe…) The implications are that we should look to replicated and distributed SOLR indexing (hopefully stealing a page from “noSQL” handbook); keep an eye on Map/Reduce-based triple stores (interesting idea!); and repository storage will be spanning multple systems.
4. What is a filesystem. Brad noted that with filesystems what was once hidden from the end user (think the large systems of the 1960s, 1970s and 1980s) became visible (the familiar desktop file folder structure) and is now becoming hidden again (as with mobile device apps). Applications are now storing opaque objects again; how do we effectively ingest them into our repositories?

I also attended the two sessions on identifiers in the afternoon (Peter Sefton’s “A Hint of Mint: Linked Authority Control Service” and Richard Rodgers’s “ORCID: Open Research and Contributor ID — An Open Registry of Scholarly IDs”), but the time is late and tomorrow’s events will come soon enough. Given eough time and energy, I’ll try to summarize those sessions later.

This week I am attending the Open Repositories conference in Austin, Texas, and yesterday was the second preconference day (and the first day I was in Austin). Coming in as I did I only had time to attend two preconference sessions: one on the integration — or maybe “invasion” of the Spring Framework — into DSpace and one on the introduction of the DuraCloud service and code.

DSpace and the Spring Framework

I came into the session more familiar with the Spring Framework than with the DSpace code, so I found the session to be a good introduction to some of the DSpace concepts even though I wasn’t the target audience. (I imagine the target audience was someone familiar with the DSpace code wanting to learn about the Spring Framework.) Thanks, Mark, for putting up the web tutorial and walking through it during the preconference session.

DuraCloud Introduction

The session showed the web-based administration interface and the supporting tools for integrating a DSpace repository and a Fedora repository into DuraCloud. Attendees were also given access to a command-line Java application that could be used to upload content into a DuraCloud instance, although sadly it wasn’t demonstrated during the preconference session. (Perhaps I’ll try it out on the sly later with the DuraCloud credentials they gave us at the session…) In addition to the functinality being built into DSpace and Fedora there will be REST-based code libraries for Java, PHP and Python — meaning that any developer could write code to make use of DuraCloud with any repository platform. The whole DuraCloud application is going to be released into open source under the Apache 2.0 license as part of the efforts to create a community using the code and encourage others to write new services for DuraCloud. This is something I’m going to keep watching; I’ve already signed up for a preview account for when the beta is released later this month.

Thanks also to DuraSpace for sponsoring the evening reception after the preconfernce session at the University of Texas Club.

One of the highlights of the Code4Lib annual meeting is the “lightning talk” rounds. A lightning talk is a fast-paced 5 minute talk on a topic of the presenter’s choosing. They are usually scheduled on an ad-hoc, first-come-first-served basis on the day of the event. They are an opportunity to provide a platform for someone who is just getting started with public speaking, who wants to ask a question or invite people to help with a project, or for someone to boast about something he or she did or tell a short cautionary story. These things are all interesting and worth talking about, but there might not be enough to say about them to fill up a full session timeslot.

“Virtual Lightning Talks” replicates this conference activity online in a virtual meeting environment. Each one-hour block consists of 10 six-minute sessions (one minute for the presenter to take control of the virtual meeting environment and test audio followed by a five minute presentation). Presenters show their work by sharing their entire desktop; the presentation can consist of slides, web browser, command-line shell, or any other application that can be shown on the desktop.

The first round will be on April 4th at 1:30pm Eastern U.S. Daylight Time. The Virtual Lightning Talks page on the Code4Lib wiki has more information and space to sign up to be a presenter or attendee.

Screenshot of Sample 'Web Presence for Small Libraries'

They created a sample library called Loremville, TN public library to demonstrate key aspects of the service. I did not ask them how long that particular example will be around, so you may follow that link at a later date and not find it.

This “Library Website in a Box” is a concept that has been around for many years, and the latest trigger to try something was a resolution from the last OCLC Members Council to make scaled-down versions of OCLC services for “small and rural libraries.” The Innovation Lab group conducted some research about the existing state of public library web presences by sorting the IMLS-reported data from 2008 by number of volumes held and number of library staff. They looked at the lowest quartile (roughly 20,000 volumes or less and staff size in the low single digits) and found that there was generally no web presence for these libraries. In the second quartile there were instances of library websites, but they did the library no credit (outdated, poorly constructed, incomplete information — as was said at the meeting, these libraries had a presence on the web but probably shouldn’t). Some had automation systems supplied by large groups, but others didn’t have evidence of an automation system. So the project charter was to find an easy and inexpensive way for a library in these quartiles to create a desktop and mobile device web presence.

One of the unique aspects of the project development was to first set the approximate lower ($5/month) and upper ($40/month) price boundaries and find a way to provide the highest level of service possible at those price points. The Innovation Lab team tried techniques such as a farm of WordPress sites but found they couldn’t make the revenue-versus-cost equation to work. In the end they constructed a custom database-driven content system in PHP. Institutional data is initially pulled from sources such as the WorldCat Registry, and there will be some process for a library to “claim” its site. There might also be a way for a library to create a site for itself if no registry data yet exists.

There are four levels of site authorization: public (unauthenticated) viewing, a registered patron, a staff member, and an administrator. Content from the pages are edited by the administrator at the subscribing library with a WYSIWYG¹ editor. There are content boxes for the library’s location, staff/volunteers listing, events calendar and news, hours and phone number, policies, and service information. The library’s address is fed into a Google Maps service to display a map of the area surrounding the library.

The dynamic parts of each library’s website could have a list of books from various sources like the New York Times and Oprah’s book group. The service also envisions offering a “default digital collection” using public domain works in text, PDF, and mobile reading device formats from sources such as Project Gutenberg and the Internet Archive.

The inventory and circulation module is simple and straight-forward. Each item has only eight full-text fields with the intention that the description will likely be done by a volunteer without professional library training. Cataloging can be done by typing in the information or scanning the ISBN with an app on a mobile device; item information is pulled from WorldCat if found. The cataloging application does not attach holdings to WorldCat, but the OCLC number is kept and might be used to facilitate offloading MARC records in cases where a library outgrows this simple circulation module to a more functional integrated library system. The circulation functions are check-in, check-out, renew, place hold and cancel hold. There are no financial functions in the system.

At these price levels, the system needs to be highly automated and self-supporting. The cost to OCLC of one call to a customer support phone number could easily run through all the revenue OCLC would receive from the subscribing library in a year. A widely adopted implementation at the targeted price points means that OCLC could dedicate one or two technical staff to support and upgrade the system in addition to the hardware and support service amortization.

One unresolved issue is domain names — what is the URL that will be used for each library’s site. OCLC is investigating options such as partnering with a domain registrar company (someone like GoDaddy), becoming a domain registrar themselves, or putting all the sites under one domain. The economics of each of these options will be a factor. The sometimes cumbersome aspects of migrating domain names from one service to another may make that activity cost-prohibitive as well.

Mike Teets noted that this was at a “project” stage, not a “product” stage. My paraphrasing of what this distinction means is that the technology to create a product is largely done, but the decisions on the final formative pieces of the technology and the surrounding support infrastructure — is not yet done (and might never be done). There isn’t even a formal name for it yet; it is being called “A Web Presence for Small Libraries.” One desired additional feature is to add e-mail boxes for library staff/functions to the site. Those in the room, including me but more importantly others who are more closely aligned with the target “small and rural” public library population, were pretty excited about it and wanting to talk further about if what we saw could be made a reality.

Personal Impressions

Like others in the room, I came away impressed by the project demonstration. It is definitely fits the bill as a basic library website and even a starter inventory and circulation management system. I see that libraries could start with something like this for the cost of a couple of books added to the collection (roughly estimated at $60/year). At this price level, OCLC thinks they could sustain the costs of operations plus have some left over for investment in incremental improvements. I also think that because a library would pay a minimal fee for it, they would feel a tangible sense of ownership over the site and would keep them up-to-date. From this a library could “graduate” to another service — their own Drupal or WordPress site, to a shared ILS or Webscale Management Services.

The content areas seemed the most appropriate for the target audience. The web page design is modern, and I could see options for future enhancements as time and revenue permit such as providing limited options to personalize the template (change colors, adding picture — or adding links to pictures that might be stored on services such as Flickr).

One attendee at the session suggested that rather than OCLC prepopulating a digital content library of all public domain content that the set be limited to those that are the most downloaded so as not to overwhelm the user with a lot of unused (and/or unusable) digital content. That sounds like a good suggestion to me.

OCLC Staff are looking for feedback on this project. They say that the system is “production ready” with all the software controls and data recovery features of OCLC behind it. What they think is missing is community support to have local engagement with the targeted libraries to show them what is possible. That is an area where OCLC needs help. (I can only imagine the shocked silence of a volunteer at a small library to get a call from OCLC — if they even knew what OCLC was — with an offer to create a website for the library. “Only $5/month…sign up now and we’ll throw in a second one for free!”) There is an e-mail address — innovation@oclc.org — that goes to all the OCLC Innovation Lab members, and a WebJunction group for public discussion.

About the Innovation Lab

For this project, the Innovation Lab sought out cooperation from OCLC staff to build prototypes and components of the service in their spare time. Each Monday the self-selected group would get together to show what had been built and discuss ways to move the project forward in the following week. In this way they rapidly iterated over ideas to come up with what was ultimately proposed.

The text was modified to update a link from http://expreimental.worldcat.org/ to http://experimental.worldcat.org/ on January 12th, 2011.

The text was modified to update a link from http://www.webjunction.org/923 to http://www.webjunction.org/923 on January 12th, 2011.

The text was modified to remove a link to http://www.clicweb.org/e_discover/e_discovermarc.html on June 9th, 2011.

Footnotes

1. What-You-See-Is-What-You-Get, meaning as you edit the document with styles like bolding, underlining, and bulleted lists, what you see in the editing interface is exactly how it will appear in the viewing interface.

A last-minute change to my plans for ALA Midwinter came on Tuesday when I was sought out to fill in for a speaker than canceled at the ALCTS Digital Preservation Interest Group meeting. Options for outsourcing storage and services for preserving digital content has been a recent interest, so I volunteered to combine two earlier DLTJ blog posts with some new information and present it to the group for feedback. The reaction was great, and here is the promised slide deck, links to further information, and some thoughts from the audience response.

Slide Deck and References

To examine the similarities and differences in costs, let’s use the OhioLINK Satellite Image collection as a prototypical example. It consists of about 2 terabytes (2TB) of high-quality images in TIFF format, with about 7.5GB of data going into the repository each month. In the interest of exploring everything that S3 can do, there is an assumption that approximately 4GB of data will be transferred out of the archive each month; OCLC’s Digital Archive does not have a end-user dissemination component.

The point of showing this scenario is to show the widest range of costs — from a storage-only solution like Amazon S3 to a soup-to-nuts service like OCLC Digital Archive. A word about the redacted costs. Some of the numbers for OCLC’s Digital Archive response (from 2008) came from a confidential quote, so the numbers were removed from the public table. For the numbers that are publicly listed, the values come from Barbara Quint’s article.

The articles and blog posts I referenced in the course of the presentation were:

Iglesias, Edward and Wittawat Meesangnil (2010). Using Amazon S3 in Digital Preservation in a mid sized academic library: A case study of CCSU ERIS digital archive system. The Code4Lib Journal, issue 12, retrieved 5-Jan-2011 from http://journal.code4lib.org/articles/4468

Murray, Peter (2008). Long-term Preservation Storage: OCLC Digital Archive versus Amazon S3. Disruptive Library Technology Jester. Retrieved 5-Jan-2011 from http://dltj.org/article/oclc-digital-archive-vs-amazon-s3/

Murray, Peter (2009). Can We Outsource the Preservation of Digital Bits?. Disruptive Library Technology Jester. Retrieved 5-Jan-2011 from http://dltj.org/article/outsource-digital-bits/

Quint, Barbara (2008). OCLC Introduces High-Priced Digital Archive Service. Information Today. Retrieved 5-Jan-2011 from http://newsbreaks.infotoday.com/nbReader.asp?ArticleId=49018

• Amazon S3. Retrieved 5-Jan-2011 from http://aws.amazon.com/s3/
• Chronopolis. Retrieved 5-Jan-2011 from http://chronopolis.sdsc.edu/
• DAITSS. Retrieved 5-Jan-2011 from http://daitss.fcla.edu/
• DuraCloud. Retrieved 5-Jan-2011 from https://wiki.duraspace.org/display/duracloud/DuraCloud
• Iron Mountain. Retreived 5-Jan-2011 from http://www.ironmountain.com/news/2009/impr02232009.asp
• OCLC Digital Archive. Retrieve 5-Jan-2011 from http://www.oclc.org/us/en/digitalarchive/
• Private LOCKSS Networks. Retrieved 5-Jan-2011 from http://lockss.stanford.edu/lockss/Private_LOCKSS_Networks

Some Thoughts

The end-of-year holidays are behind us and (in the northern parts of the northern hemisphere) the cold days of winter in front of us. What better time to bag it all and head to the warm(er) temperatures of San Diego, California for the ALA Midwinter meeting. I mean — come’on — do you really want to dive into all of that work that piled up over the past week or so? (You say that even more work will pile up if you attend the meeting? Bah, humbug!) If you are going, I wholeheartedly endorse the new ALA Connect-based meeting planner. It is at times frustratingly slow, but chock full of ways to slice-and-dice meeting events that were not possible in the earlier version. (I’m going to put in a suggested enhancement that the iCal file export includes URLs to the meeting listing online; that would be immensely helpful.)

Here is my schedule of events, with links into meeting descriptions on ALA Connect where appropriate. Also don’t forget to check out the What’s Happening (229KB, PDF) document for the inside scoop from the ALA staff to the ALA Council members. If you are interested in getting together with me, let me know and we’ll find a time.

Friday, January 7th

I’m arriving late, but normally I would try to get to the RMG Integrated Library System vendor panel. If the past year is any guide, I think it is going to be another wild year for ILS vendors. I do expect to pop into the LITA Happy Hour at the Uber Lounge of the Se San Diego Hotel.

Saturday, January 8th

After time for lunch, some exhibits and a private meeting I’ll have a choice to make. On the one hand is the regular SPARC/ACRL forum on the topic of Open Access and the changing state of scholarly publishing. The lineup is great: Caroline Sutton, President of the Open Access Scholarly Publishers Association (OASPA);
Wim van der Stelt, Executive Vice President of Corporate Strategy for Springer; and Catriona McCallum, Senior Editor for PLoS Biology and Consulting Editor for PLoS ONE. That sounds like a lively discussion.

On the other hand is the ad hoc Google Books Task Force meeting. The topic is certainly of interest, and news of any movement by the court on the settlement has been too quiet. But I don’t see an agenda for the meeting and I don’t know what will be covered. Fortunately, these two meetings are near each other in the convention center. Maybe I’ll start here and move to the SPARC/ACRL Forum.

Sunday, January 9th

Then I have another tough choice. I could go to the LITA Emerging Technologies Interest Group meeting. There will be a follow-up discussion to the panel from ALA Annual (which unfortunately I missed due to a family illness). Or I could go to the LITA Cloud Computing / Virtualization Interest Group meeting. I can’t find an agenda for this meeting either, but the topic is of interest at work.

Rounding out the day will be the public launch of an experimental service “providing a low cost and easy-to-use Web site service for small and rural public libraries” by the OCLC Innovation Lab.

Monday, January 10th

Tuesday, January 11th

The text was modified to remove a link to https://www3.oclc.org/app/ala_registration/ on January 28th, 2011.

The text was modified to remove a link to http://www.sesandiego.com/ on June 9th, 2011.

At the American Library Association conference this weekend, I’ll be part of a panel presentation from the LITA Emerging Technologies Interest Group with the title “What Is Your Library Doing about Emerging Technologies?” The presentation will be on Saturday, June 26 from 1:30pm to 3:30pm in room 103B of the Washington Convention Center. The publicity blurb:

A new job title of “Emerging Technology Librarian” seems to reflect an awareness among today’s libraries that there is a need for a librarians whose main role is to explore, evaluate, promote, and implement various emerging technologies. 19 librarians in different fields of librarianship at academic, school, and public libraries will discuss the topic of emerging technologies at libraries, their evaluation, implementation, adoption, and management challenges.

My panel group met by conference call this afternoon to discuss the topic, and I came away feeling great about the synergy of this group. The panel style is the panelists responding to a question from the moderator and reactions from each other with time for questions from the audience. No canned presentations!

Thanks to Bohyun Kim from Florida International University for setting up the panel discussion.