Last month OCLC announced a new service offering for long-term storage of libraries' digital collections. Called Digital Archive™, it provides "a secure storage environment for you to easily manage and monitor the health of your master files and digital originals." Barbara Quint has an article in Information Today called "OCLC Introduces High-Priced Digital Archive Service" in which she makes a comparison to Amazon's Simple Storage Service (or "S3″) from primarily a cost perspective: "The price for S3 storage at Amazon Web Services is 15 cents a gigabyte a month or $1.80 a year, in comparison to OCLC's $7.50 a gig." Barbara also goes into some of the technical differences, but I think it might be worthwhile to go a little more into depth on them.

OCLC's Digital Archive

According to the service overview, Digital Archive is a content hosting service that provides:

• Systems management
• Physical security
• Data security
• Data backups
• Disaster recovery
• ISO 9001 certification
• Manifest verification
• Virus check
• Format verification
• Fixity check

It is targeted towards the preservation of digital masters. There is a document on the Digital Archive website called Our commitment that describes other aspects of a digital preservation program: "OCLC is actively developing processes for full preservation of digital assets to ensure complete renderability, regardless of technology changes. This preservation system will likely involve a combination of migration and emulation." But it is not clear whether these services, beyond "bit preservation" activities, is part of the Digital Archive service or will be part of an add-on service to be developed later.

This "Digital Archive" is a revamping of an older product from OCLC, also called "Digital Archive" but one that included a web harvesting tools component. The service and support documentation on the OCLC website still refers to the former version of Digital Archive, so there is little information about how the service works beyond what one can infer from the sales information.

Amazon's S3

Amazon describes S3 as "a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites." Files are transfered across the internet to Amazon's services and stored in multiple data centers. Files can be retrieved using standard HTTP mechanisms (the same protocol that powers the web) and are protected by an optional access control mechanism. S3 does have a Service Level Agreement (SLA) that offers guarantees on performance.

SLA seems to extend only to availability of the service, not to a long term commitment to keeping track of files on the service.

AWS [Amazon Web Services, LLC] will use commercially reasonable efforts to make Amazon S3 available with a Monthly Uptime Percentage (defined below) of at least 99.9% during any monthly billing cycle (the "Service Commitment"). In the event Amazon S3 does not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.

There is no mention specifically in the S3 SLA about permanence of file storage. In leu of that, one seems to be covered by the overarching Amazon Web Services Customer Agreement, which has several points of interest from a preservation use perspective:

3.3. Termination or Suspension by Us Other Than for Cause.
3.3.2. Paid Services…. We may suspend your right and license to use any or all Paid Services (and any associated Amazon Properties)…, or terminate this Agreement in its entirety (and, accordingly, cease providing all Services to you), for any reason or for no reason, at our discretion at any time by providing you sixty (60) days' advance notice in accordance with the notice provisions set forth in Section 15 below.

So if they desire to terminate a library's use of the service (assuming there was no specific cause — such as a violation of the terms of use — to do so), they have to give 60 days notice. That's when the "Data Preservation in the Event of Suspension or Termination" clause kicks in:

3.7.2. In the Event of Termination Other Than for Cause. In the event of any termination by us of any Service or any set of Services, or termination of this Agreement in its entirety, other than a for cause termination under Section 3.4.1, (i) we will not take any action to intentionally erase any of your data stored on the Services for a period of thirty (30) days after the effective date of termination; and (ii) your post termination retrieval of data stored on the Services will be conditioned on your payment of Service data storage charges for the period following termination, payment in full of any other amounts due us, and your compliance with terms and conditions we may establish with respect to such data retrieval.

The customer agreement then goes on to say:

3.8. Post-Termination Assistance.Following the suspension or termination of your right to use the Services by us or by you for any reason other than a for cause termination (i.e., a termination under Section 3.2 or under Section 3.3), you shall be entitled to take advantage of any post-termination assistance we may generally make available with respect to the Services, such as data retrieval arrangements we may elect to make available. We may also endeavor to provide you unique post-suspension or post-termination assistance, but we shall be under no obligation to do so. Your right to take advantage of any such assistance, whether generally made available with respect to the Services or made available uniquely to you, shall be conditioned upon your acceptance of and compliance with any fees and terms we specify for such assistance.

Perhaps the most troubling aspect, from a preservation point-of-view, deals with data security and backups. Specifically, Amazon says that data security and backups are the responsibility of the customer. The Amazon Web Services Customer Agreement says (emphasis added):

7.2. Security. We strive to keep Your Content secure, but cannot guarantee that we will be successful at doing so, given the nature of the Internet. Accordingly, without limitation to Section 4.3 above and Section 11.5 below, you acknowledge that you bear sole responsibility for adequate security, protection and backup of Your Content. We strongly encourage you, where available and appropriate, to use encryption technology to protect Your Content from unauthorized access and to routinely archive Your Content. We will have no liability to you for any unauthorized access or use, corruption, deletion, destruction or loss of any of Your Content.

That kind of security and data backup is something you'd want in a preservation service. Since activities against S3 storage is limited only by a knowing a private "key"¹ (as opposed to limiting to particular IP addresses or not allowing deletes/modifications from the web at all), it is a real possibility that the archive can be harmed if the private key is disclosed. Furthermore, S3 does not have a backup/restore service for retrieving files that were accidentally or maliciously deleted.

Feature Comparison

It is useful to compare Amazon's S3 on a point-by-point basis OCLC's Digital Archive service to try to put some meaning behind the cost numbers.

This is a useful comparison because it would indicate what one would have to layer on top of S3 to reach the level of service provided by Digital Archive. For instance, it would be possible to create an application that would perform the manifest and format verifications as well as the periodic virus and fixity checks against the files in S3. It would even be possible to run that application in Amazon's Elastic Compute Cloud (EC2) — a "virtual computing environment" that allows developers to easily create and deploy software on the internet. Since data transferred between Amazon EC2 and Amazon S3 is free of charge, there wouldn't be the S3 cost of periodically downloading the data to perform the virus and fixity checks.

One advantage to note about the S3 solution is that it can perform as a "light archive" — meaning the data is available to users in addition to being part of the content repository. In contrast to the OCLC Digital Archive service — a "dark archive" — access to the data is highly or completely restricted. Still, the lack of automated backups and a robust data security infrastructure in the S3 infrastructure are notable from a preservation data service perspective.

Cost Comparison

To examine the similarities and differences in costs, let's use the OhioLINK Satellite Image collection as a prototypical example. It consists of about 2 terabytes (2TB) of high-quality images in TIFF format, with about 7.5GB of data going into the repository each month. In the interest of exploring everything that S3 can do, there is an assumption that approximately 4GB of data will be transfered out of the archive each month; OCLC's Digital Archive does not have a end-user dissemination component.

For this prototypical example, S3 would cost $3,800 in the first year and roughly $3,615 per year after that, with the added benefit that end-users could access the content without using our infrastructure. There are costs associated with the OCLC Digital Archive service that had to be redacted from the public version of this table due to a confidentiality clause, but the costs that are assumed for ongoing storage based on Barbara Quint's article are comparable to the quote I got from OCLC and represent a large portion of the total yearly costs.

By way of comparison, we are planning the purchase of 50TB of storage this summer for roughly $250K; that is about $5,000/TB. Amortize the cost of the hardware over five years and assume 150% of the purchase price represents maintenance, personnel support, and other factors, and we get $2,500/TB/year. This doesn't include software costs, so it is comparable to S3 in the functions table above; software would have to be written to verify the manifest and file formats on ingest as well as the monthly fixity and virus scanning. It also represents only one copy of the data; it does not include the duplication across data centers that both Digital Archive and S3 provide.

Conclusions

OCLC's Digital Archive product goes pretty far down the path of a preservation-worthy archive of digital files. The value-added services, in addition to simply storing and retrieving files, make it as close to a one-stop shop as I've seen so far. Whether outsourced digital preservation services makes sense — particularly at this price point — remains to be seen, especially since is hard to make a comparison since I'm betting that most of us aren't (yet) doing all of the ongoing activities with digital preservation masters that Digital Archive is doing.

Amazon's S3 is an inexpensive, network-oriented file hosting service, and as such it doesn't have many of the features built into it that we would want to see in a preservation archive service. Beyond raw file service, one would need to add layers of software and human activities to perform the functions that Digital Archive provides now.

Looking at OCLC's Digital Archive and Amazon S3 is almost an apples-to-oranges comparison, both in price and in functionality. Comparing functionality first, S3 is missing critical components of a preservation storage system — namely, rigorous access control and a content backup/restore facility. Comparing costs, though, S3 is dramatically cheaper…and has the benefit of serving up large files to end-users using Amazon's distributed infrastructure.

It is possible to level the functionality playing field a bit by taking responsibility for the ongoing maintenance of files in the S3 archive — those things that Digital Archive offers as value-added services over raw file storage. An EC2 virtual machine running in Amazon's infrastructure can perform the virus and fixity scanning. And with good key maintenance (as with passwords, regularly changing the private key and securing it appropriately), S3 could conceivably offsite copies of content stored offline (e.g. burned to preservation quality optical media). Again, in this scenario one has to take responsibility for refreshing the offline media and occasionally running comparisons against the S3 offsite copy.

Footnotes

1. S3 uses secret keys — a 40-character password — to verify the identify of the client making the request. If the private key becomes known, anyone on the internet can perform operations actions as the content owner.

Thanks, Noel, and everyone else who made the video editions of Code4Lib 2008 presentations possible. I just had a chance to notice that the video from my JPEG2000 to Zoomify Shim lightning talk was online:

Some updates since the post and the presentation were first done. The code that exists in the source code repository now was refactored to use JJ2000 as part of the Sun ImageIO package. We were seeing non-threadsafe problems with Kakadu and thought that using the multithreaded ImageIO package would help. Unfortunately, even with extensive caching, it did not. My next task is to bring Kakadu back into the picture using the threadsafe JNI implementation that is part of the ImageIO-ext project to see if that helps.

Unfortunately, time ran out before this needed to go into initial production with the OhioLINK DRC roll-out, so it isn't in production. The scheme shows promise, though, so I'm going to keep working with it…

Blake Carver (of LISNews and LISHost fame) announced two new projects yesterday: LISWire and LISEvents. In the same spirit that I would categorize open source, open access, and open knowledge, these services level the playing field for the publication of library-oriented press releases and announcements of events.

LISWire - The Librarian's News Wire

For the publication of press releases and other newsy information, LISWire is a service targeted towards the librarian community. Looking for announcements related to public libraries? Try http://liswire.com/taxonomy/term/24/. How about items about open source in academic libraries? http://liswire.com/taxonomy/term/23,27 is the URL for you. Interested in both announcements from both for-profit and not-for-profit organizations? Then look at http://liswire.com/taxonomy/term/35+36. Best of all, these URLs work in your browser and your RSS reader; you see the same content either way. I think we need some work on the taxonomies; I'm going to send my suggestions to Blake and Robin Blum (his collaborator) and suggest you do, too.

LISEvents - The Librarian's Events

Built on the same concept as LISWire is a draft of a site for posting information about events. This is definitely still a work in progress, but the same capabilities and possibilities are showing through. I think it would be useful to add a taxonomy for location so that it would be possible to see events in, say, Ohio and/or the midwest region; I can easily see myself subscribing to such a feed to keep up with what is going on in the area. And let's not forget an "online region" for webinars and similar events.

Congratulations Blake and Robin, and thanks for trying to build these services for our community. If they take off, they will be a real, tangible benefit for us.

In reading a background paper for the American Social History Online portal, I was reacquainted with a paper by Muriel Foulonneau, Thomas Habing and Tim Cole from UIUC called "Automated Capture of Thumbnails and Thumbshots for Use by Metadata Aggregation Services."¹ This is the abstract:

The practice of including thumbnails in short record displays, increasingly common in local implementations, is being adopted by metadata aggregation service providers as well. In addition, thumbnails and Web thumbshots have begun appearing as part of Web search results. This article reports on a project at the University of Illinois at Urbana-Champaign (UIUC) to make more comprehensible heterogeneous resources available on the UIUC CIC metadata portal by incorporating thumbnails and thumbshots of image and Webpage resources in the context of the OAI Protocol for Metadata Harvesting. In addition to thumbnails provided by partner data providers, UIUC has developed an automated process to generate thumbnails and thumbshots from the Webpages resources pointed to by the metadata records.

The paper cites dissatisfaction with results from metadata portals that consist exclusively of textual descriptions of the objects. It also cites studies that show the addition of thumbnail images to the results display improves user satisfaction. With that in mind, UIUC wrote Thumbgrabber — a Windows application written in Visual Basic that uses Internet Explorer to find images in websites and/or take image snapshots of web pages as they have been rendered. In the UIUC context, the application is fed URLs from records harvested via OAI-PMH, although it would seem like it would be able to process any arbitrary list of URLs.

This is a useful tool to keep in mind as we think more about aggregating the metadata records into vertical (subject-specific) portals and repurpose metadata records in other ways.

Footnotes

1. Foulonneau, M., Habing, T.G., Cole, T.W. (2006). Automated Capture of Thumbnails and Thumbshots for Use by Metadata Aggregation Services. D-Lib Magazine, 12(1) DOI: 10.1045/january2006-foulonneau

There has been an increasing focus on the cost of textbooks in the mainstream media this year, and I don't think it is the case that I'm just becoming more sensitized to it. Take for example the editorial from the Washington Post on February 7th. The second paragraph succinctly describes the issues being debated most often:

There are several reasons that textbooks are so costly. For one, even though there have been no major advances in fields such as calculus and elementary physics in decades or even centuries, publishers still churn out new editions of textbooks on these subjects every three or four years. The changes are typically superficial, but they prevent students from being able to purchase used, older editions. Publishers also frequently bundle unwanted additional materials such as CD-ROMs and study guides with textbooks. Professors rarely assign these extra materials, which drive up costs, and students often cannot sell the books back to bookstores once the shrink-wrap has been removed. Publishers can get away with these shenanigans because there's a fundamental disconnect in the textbook marketplace: The people paying for the books (the students) are not the ones choosing them (the teachers).

The editorial goes on to suggest that what is needed is regulation "to ensure that both students and professors are able to make informed choices" — regulation of the kind proposed in House bill H.R. 4137 (covered on DLTJ earlier).

The New York Times has a posting on their editorial blog on April 10th that follows up on the federal legislation. It says that the House and Senate versions are currently being reconciled in a conference committee, and goes on to summaries the key points of the House-proposed legislation.

The public radio program Marketplace had a feature on April 15th that covered textbooks. It also had a statement that I hadn't heard in other stories:

[Executive director of Higher Education for the American Association of Publishers Bruce] Hildebrand says it costs a lot to produce multi-color glossy pages and the other materials teachers want, like lecture notes, PowerPoint slides and exam questions. In effect, students are paying higher prices to make life easier for their professors. They usually don't even know what the prices are.

While I've heard the claim before that professor's don't know the true costs of textbooks and have a hard time finding that information, this is the first time I've heard that "students are paying higher prices to make life easier for their professors." I'm not necessarily buying that. In fact, there was a comment from a listener who teaches biology that was broadcast on yesterday's show refuting that. The listener says "That is completely ridiculous. Even if I were to use only the slides that they provide, it would still take me hours per lecture to write a lecture the first time."

Then there was news on April 16th about the Public Interest Research Group announcement that they had reach 1,000 signatures on an online petition to make textbooks affordable including the use of open textbooks, when available. The trade periodical Inside Higher Ed was one that published an article about the PIRG's accomplishment. AAP's Bruce Hildebrand praised the effort (that would ironically reduce traditional publisher revenue), saying "any faculty member or group that is willing to make that level of commitment to provide a free textbook, I applaud them." He then goes on to caution that content creators are assuming the role of publisher, along with all of the duties to keep the content updated and integrate it into instructional systems. This article also lists several examples of how Open Educational Resources is gaining ground, including a description of a management school professor who stopped using traditional textbooks 10 years ago.

Lots of activity here; it is definitely worth keeping an eye on.

This is a stub for a page where I'll put my schedule. I'm creating the stub so there is something in the tag page and Atom feed for DLTJ at Annual as listed on the "Who's Blogging Annual?" page on the ALA Wiki. At the very list, I know I'll be at the JPEG2000 program. More posted here later.

This morning I got an invitation to join ResearcherID, a new author profile service from Thomson Scientific. The service sounds nice enough — who doesn't want to take steps to avoid confusion between authors? — and if you have access to other Thomson products (like ISI Web of Knowledge or Web of Science) it may be even nicer. I'm all for the establishment of unique identifiers so we can start to do some interesting things with co-citation analysis and mining the web of connections in journal articles, but I'm not signing up. At least not yet.

"A Unique Identifier"

The e-mail invitation starts with this:

A unique identifier

When you register, you are assigned a unique ID number that expressly associates you with your published works, regardless of any possible nomenclature confusion or institutional affiliation changes. These unique identifiers allow everyone who accesses ResearcherID.com to easily find a specific author's work, avoiding the common problem of author misidentification.

With the service, I can assert that particular citations are papers that I have written. (Interesting sidebar — I wonder if the ResearcherID service attempts any sort of automated and/or manual verification of the claim that I was an author/co-author of a paper.) I do this either by uploading a list of citations (in Thomson's RIS format only) or selecting them from ISI Web of Knowledge or Web of Science (if I'm at a subscribing institution). Then I have a nicely formatted web page with all of my papers. Granted, it isn't the only way I can go about doing this, but the Thomson service does seem to make a point about distinguishing itself by assigning me a "ResearcherID" in the process.

There seems to be something more going on here, though. Why go through the effort of assigning me an alpha-numeric string and publicizing it if it isn't going to be used somewhere else?

But what about the other end of the use case?

The End User License Agreement (EULA) are pretty standard stuff, but buried in the middle is this:

5.3. Except as described in this Agreement, You may not use, copy, adapt, translate, modify, sell, distribute or otherwise create derivative databases, services or works of or based on the ResearcherID Service (including the Researcher Registry) or the materials accessible in the ResearcherID Service.

That would seem to effectively eliminate the ResearcherID(tm) as a useful identifier in other systems. Presumably without paying Thomson for the right to use the identifier. And without knowing the cost of using the identifier in other systems (there doesn't seem to be any information published on ways to use the identifier elsewhere, even though an earlier section of the EULA describes such usage for "Sponsored Services"), I'm not ready to sign onto (and therefore effectively promote) such a service.

The full End User License Agreement

You can't get to the EULA without receiving an invitation and putting your personal information into the website. It is only after you get the invitation and go through the first step of entering in your demographic information that you get to see the EULA. (The "Privacy Policy" and "Terms of Use" links on the ResearchID website do not contain this information. So, for the benefit of others in critiquing this post, I've copied the entire EULA below.

ResearcherID Terms of Use and Privacy Policy

8.8. Each Party intends that TS's Affiliates and Third Party Sponsors shall be third party beneficiaries of this Agreement and, thus, entitled to enforce this Agreement as if an original party hereto. There shall be no other third party beneficiaries.

I found this meme via Karen Schneider's entry. Although I wasn't explicitly tagged, I thought it was interesting enough to add an entry to the meme's Flikr pool.

With all due respect to Karen — and I agree that a love of reading is important — but it is a sense of wonder that encourages a love of reading and all sorts of other critical character traits. This is a picture of my daughter when she was about three years old. She is on the back deck of our Connecticut house watching a caterpillar crawl up our gate. She loves to read (and now three years later is reading scores of books on horses and dolphins from the elementary school library), and as her father I hope the same sense of curiosity will sustain her love for reading, arts, sciences, and life.

Since I wasn't tagged, I'm not inflicting the meme on anyone else.

It was only a few months ago that I was teasing Dan Chudnov for joining Twitter. Now I've gone and done it myself. I don't expect to be using it much, but after observing the "Falls Church, VA" incident yesterday, I thought it would be an useful tool to have at-the-ready. Here's the story of what inspired it.

Someone on the Code4Lib IRC channel (was it ‘lbjay'?) asked if anyone knew about an explosion in the Falls Church, VA, area after reading a report about it on Twitter. I ran a search in TweetScan for "Falls Church, VA" and was able to watch the event unfold as the "DC emergency tweet network" fired up. Eventually it was determined that it was indeed an earthquake event, but the discussion of the event via Twitter was enough to catch the attention of at least one media blogger.

It reminded me a great deal of the 1994 Northridge earthquake in California. Many of the landlines were down or jammed with too many people calling, but the internet stayed up and an IRC channel was set up so that reports of the earthquake effects to be broadcast from the region. If the same thing to happen today, Twitter — through the internet or through mobile devices — would likely be the tool used to track the event.

Now, back to Twitter, here are the parts that I can't figure out. Almost immediately after I registered for the service and signed in for the first time, I was automatically following betseymerkel, someone who appears to be working with open source software in a Cleveland-area library. I don't remember doing anything to cause me to start following her, although I suppose it is possible I made a stray click somewhere. And through the first 24 hours with the account, four people are following me. I didn't tell anyone else about my activities — the only two tweets I've posted dealt with setting up the account. I don't think I know any of these people (betseymerkel is one of them), so I don't get why they would spontaneously start following me. Thoughts?

Oh, and you can start following me, if you want. I'll probably follow colleagues during library conferences, but then use something like TwitterSnooze to turn off the chatter in-between events.

Next Day Follow-up

Another related story — the Chronicle of Higher Education Wired Campus Blog reports on the use of Twitter during a lockdown at the University of Richmond on Tuesday. Jim Groom, an instructor at the University of Mary Washington, posted a blog entry about how he and others found information and comfort in the Twitter posts passing between rooms of the building and with the outside world. A commenter to the Chronicle's Wired Campus Blog entry notes, "ASU has an emergency text service, but it's not as fast as Twitter (when Twitter isn't down)." Which brings to mind dangers of relying on a free-to-use service as a primary — or even simply expected — mode of communication during times of emergencies.

It was announced on December 13th last year with much discussion here on DLTJ and elsewhere on the blogosphere. It would seem uncharacteristic of Google to announce something like that and keep the world waiting for months to see at least a beta of the concept. Is there some sort of technical problem with the concept? Legal or business model problem? Or was it just a trial balloon for a service yet to be developed and Google wanted some free market research from the community?

Anybody know?

As a result of discussions coming from the OAI-ORE open meeting in Baltimore in the first week of March, the document editors released a new version of the ORE alpha specifications (labeled "0.3″) earlier this month to coincide with the open meeting at Southampton, UK. In a message to the technical committee, Herbert Van de Sompel summarized the changes as:

• Data Model: changes in the relationship between Resource Map and Aggregation; introduction of a solution regarding the reference in context; revised approach regarding nesting aggregations.
• Atom serialization: Significant revision of the mapping from the ORE Model to Atom.
• Introduction of an ORE Primer (formerly called the data model overview)
• Introduction of a document describing implementation issues regarding HTTP URIs for Aggregations and Resource Maps.
• RDF serialization document: This document was first made available in alpha 0.2. It deals with RDF/XML but also RDFa that can, for example be used, in splash pages.

Comments are still welcome, and the OAI-ORE discussion list has been quite active the past few weeks. The document editors are aiming for a beta release of the specification in mid-May.

Last month the ILS Discovery Interface Task Force of the DLF called a meeting of library system vendors (including one commercial support organization for open source ILS software) to talk about the state of computer-to-computer interfaces in-to and out-of the ILS. The meeting comes as the work of the task force is winding down. An outcome of the meeting, the "Berkeley Accord," was posted last week to Peter Brantley's blog. The accord has three basic parts: automated interfaces for offloading records from the ILS, a mechanism for determining the availability of an item, and a scheme for creating persistent links to records.

Taken as a whole, these three items are arguably the most sought-after functionality by software developers seeking to extend the functionality of traditional library catalogs. The three enable all sorts of other things to happen with data stored in the ILS. Already, there has been a great deal of discussion cross-posted to the DLF ILS-DI mailing list and the Jangle-Discuss group about the form and format of the mechanism for determining the availability of an item. (Think that is easy? It comes down to the "maybe" case and the various definitions of "maybe.")

All in Favor, Say "Aye"

The statement was signed by Talis, Ex Libris, LibLime, BiblioCommons, SirsiDynix, Polaris Library Systems, VTLS, California Digital Library, OCLC, and AquaBrowser. It would be interesting to go through and calculate the percentage of reach that the signees have in the library marketplace, but for academic libraries I'm guessing it is pretty high. There is one notable exception, but I'll get to that in a moment.

All Opposed, Say "Nay"

Fortunately, none of the representatives disagreed. To disagree with the stated goals would call into question the vendor's seriousness about open data systems and the right of the library to make use of its own data. It would probably also put them at a severe commercial disadvantage.

Abstentions? The Representative from Emeryville?

As some have noted, all of the participating vendors agreed to the principles in the accord with the exception of Innovative Interfaces, which abstained. On Friday, Betsy Graham of Innovative posted a reply on the company blog. (Good to see your post, Betsy! I hope to see you around the blogosphere in the future.) She says, in part:

Innovative considers its commitment of resources carefully. We abstained for what I believe to be a good reason: that without having worked out the details, we simply were not sure what we would be committing to. The participants at this meeting had greatly varying needs and expectations for what the ILS-DI would mean to them. Some, while endorsing, even went so far as to endorse "not just what is on the table". Not to be too flip, but OCLC has endorsed this document. Does this then mean that we'll all soon be harvesting WorldCat at our whim? Maybe not.

I'm finding it difficult to reconcile Peter's preface to the Berkeley Accord and Betsy's posting. It seems that the statement was drafted at the meeting out of the parts of the DLF ILS Discovery Interface draft that could be readily agreed upon. When I prompted Peter in a comment on his blog posting asking about the effort to include information other than bibliographic MARC data (e.g. item holdings, serials checkin, and order information) in the harvesting, he said "it was made optional to achieve the larger good of obtaining a baseline agreement."

The accord is good step, and I respect the efforts of the representatives in trying to come to a baseline that everyone could support. So why wouldn't Innovative sign onto the same baseline? Admittedly, the details are yet to be worked out, but the details are mostly about how one is going to sling the bits of data around. It is the principles that are important. I'd be willing to bet that Innovative would be among the first to support the consuming of data described in the accord in advancing its ILS-agnostic Encore discovery layer product. And if others want to support more of the DLF ILS Discovery Interface draft than the three baseline operations described in the accord, then I hope they shout from the highest mountaintop that they plan to do so; but to do so does seem to be above and beyond what is called for in the wording (and my interpretation) of the accord.

Also, without intending to be overly confrontational (particularly since I wasn't at the DLF meeting with the ILS vendors), I'll point out that OCLC's response could have been from the perspective of their European ILS operations and not from WorldCat itself. Although the discussions about what is coming in WorldCat Grid would start to fulfill the three functions described in the accord from the perspective of a union catalog.

Disclaimer

It should go without saying, but probably doesn't hurt to explicitly mention in closing, that the views expressed here are my own and not necessarily that of my employer.

My place of work is looking to acquire educational videos in a digital form with an eye towards long-term preservation. At this point we receive a physical form (preferably DVD, but sometimes VHS) and digitize it to a very lossy access format (RealMedia, in this case). With this change, we would get a preservation-worthy digital copy from the producer/distributor and forego the physical version.

There is quite a lot written on preserving video, but I wanted to distill the requirements down into statements that vendors could reasonably provide today. I think these are pretty sound requirements, but I'm looking for feedback. In particular, I'm not quite sure how to handle the transfer of closed caption text from the publisher/distributor; suggestions are welcome.

[Jester's note: I just realized that an earlier version of this posting went out to the net about two hours after this "final" version. Sorry about publishing the work-in-progress early; I must have hit the wrong button in the new version of WordPress…]

File Formats

Some of the clearest guidance on file formats comes from this short excerpt from the Moving Image section of the U.K. Arts and Humanities Data Service Preservation Handbook:

Guidance on the preservation of digital video should, by necessity, change over time. […] The MPEG-2 and MPEG-4 formats are better suited to high-quality digital video. MPEG-2 is better known for its use as a format for DVD-Video, which encourages confidence when considering the likelihood that the format will be readable in the long-term. The format has an average transfer rate of 2-5 megabits per second, but there may be disk space restraints and the software tools necessary to convert and store this format are costly. MPEG-4 has a lower transfer rate of 1-2 megabits per second and is intended for streaming video. Other codecs, such as QuickTime, Windows Media, Real Video and Open DIVX, are useful for specific purposes, but not suitable for preservation. ¹

The Library of Congress Sustainability of Digital Formats site has an entry for MPEG-2 (also known as H.262) and an entry for MPEG-4 (more completely, MPEG-4 file format version #2) that give the nitty-gritty details for the file formats.

The preservation master copies we want to store has a frame size of 720 pixels by 480 pixels. (That size is for NTSC format videos, common in USA, Canada and Japan. Master copies of PAL-format videos, common in Australia, New Zealand, the United Kingdom and most of Europe, is 720 x 576.) This is the standard resolution used in MPEG-2-compressed commercially distributed DVD movies.² These frame sizes are appropriate for analog video signals. ("As defined by ITU-R Recommendation BT.601, more commonly know by the abbreviations Rec. 601 or BT.601 or its former name, CCIR 601. [It is] a standard published by the CCIR (now ITU-R) for encoding interlaced analogue video signals in digital form."³ ) The audio is 48KHz stereo at 224 kb/s or better.

Unresolved Issues

• Standards for transferring the closed captioning text.

Resources Consulted

Footnotes

1. Knight, G., & McHugh, J. (2005). Preservation Handbook: Moving Image. p. 3.
2. Audio/Video Capture and Management (2002).
3. "Rec. 601″ (2008).

Unlike previous upgrades, this left some functionality broken — notably some of the links in the second block under the "about" heading to the left (if you are reading this from http://dltj.org/ itself). But, you know what? — it's Friday afternoon and all of the important bits are working. I think. So if you see anything odd, please let me know.

cforms contact form by delicious:days

Earlier this year, Microsoft announced that it was giving $3 million in "funding, software, technological expertise, training and support services" to the Library of Congress to build on-site and online exhibits of LC historical collections. Others have commented on this. From a Jester's point of view, I've got problems with this on two fronts: Microsoft using LC in a cheap marketing ploy and LC's use of a new technology that impedes access for no good technical reason.

Library of Congress as shill for Microsoft

Interestingly, neither the Microsoft press release nor the Library of Congress press release mention the dollar figure. The first mention of it appears to be from Government Computer News in an issue dated 21-Jan-2008. That article says, "Microsoft will provide an initial grant of technology, services and funding worth more than $3 million" — I can't find mention of how much of it is in the form of cash and how much of it is in the form of in-kind licenses and/or equipment.

This excerpt from the undated page "Microsoft signs cooperative agreement with U.S. Library of Congress" is the beginnning of my uneasiness. Emphasis and links added:

Scale: The Library of Congress receives upwards of 2 million visitors per year. Also, LOC.GOV is one of the top sites for search engines for international and U.S. historical searches and receives millions of hits and unique users per month. Children and teachers across the country will learn by using this site, the materials created here will be approved curriculum in all 50 states.

Scope of influence: This initiative will influence library technology worldwide. Libraries large and small from around the country and the world look to the Library of Congress for technical guidance and are certain to take note of what tools are being used in the NVE ["New Visitor Experience"] and on MyLoC.gov [not yet operational]. Because of its scale and breadth, it will have influence not only in the U.S., but also for scalable web sites in general. The Library of Congress will engage top educators to create educational content which will meet strict guidelines mandated by state departments of education for inclusion in public schools.

Partners: Schematic: User Experience. Portal Solutions: infrastructure implementation. [What the heck does this mean?]

The Technology: For the on-site visitor, Kiosks will be built using Windows Vista and Windows Presentation Foundation (WPF) using touch-screen technology for an immersive experience. Visitors will be given a "passport" (optionally mapped to a Windows Live ID), which can be used to digitally keep track of what exhibits each visitor has seen.

The publicly available web site will offer the ability to visit the library in a virtual environment which will complement the physical visitor's experience. Because the Library of Congress is so vast, tourists will be able to "complete" their visit on line using the web site, which will have kept track of their visit when they were present at the library using the "passport" technology.

Clearly Microsoft is using this $3 million "gift" (again, we're not sure how much of it is real cash and how much is in-kind software licenses and suck) to push the adoption of Silverlight. I've got a problem with a public institution like the Library of Congress being used to push a commercial advantage. If Microsoft paid YouTube $3M to make videos available in Silverlight rather than Flash, it would be a different story. That Microsoft convinced HardRock (not a cultural heritage institution) to put up content in Silverlight doesn't bother me. If Silverlight had been adopted by the Library of Congress based on its own merits, it would be another story entirely. (Based on the information released to date, it doesn't appear that this was the case.) That leads into the next issue I have with the LC/Microsoft deal.

Silverlight-exclusive Content Impedes Access

Libraries (and cultural heritage institutions in general) exist to provide wide access to content to the public. That Silverlight is used internally to the New Visitor Experience kiosk environment is not a big deal. As soon as it leaks out into the open web, though, it is. (There are already Silverlight versions of Library of Congress online exhibitions.) For some, it is because Silverlight isn't supported for our computers. (For instance, when I visit the previously mentioned Hard Rock Memorabilia site, I'm told "The Silverlight Plugin does not work on pre-Intel Macs. Sorry.") For others, it is because the act of installing a plug-in is a barrier.

Silverlight does not have a wide scope of adoption; it is not installed already on the vast majority of machines on the net. Many access the internet in places that don't allow for plug-ins to be installed. (We hear about this at OhioLINK with regards to the educational videos available through RealMedia at OhioLINK — and RealMedia has been out for a decade! We're actively investigating a shift to Flash-based players, by the way.) Content that is available exclusively in Silverlight is effectively not available to those that cannot — for technical or know-how reasons — install the plug-in.

A justification could be offered if Silverlight represented a big-enough shift in capability to justify the added effort to install the plug-in. I don't see any evidence that it is. For instance, I'm not convinced that Deep Zoom in the upcoming version of Silverlight is really all that interesting. It would appear to use JPEG tiles to get information from the server to the applet in the browser — the same fundamental technique used by the Zoomify Flash applet and AJAX techniques like Google Maps. The Silverlight framework seems to give a clean, one-step way to implement the creation of the tiles on the server, but that can be replicated in other ways. (See for instance, my own efforts to create a shim between JPEG2000 and Zoomify.) Besides, from a content accessibility perspective, why would we make the programmer's life easier if it makes the viewer's life harder.

Final Thoughts

The LC/Microsoft would appear to be a good deal for Microsoft: for a token sum of money, probably primarily in the form of in-kind software licenses and support, they get a big platform for the exposure of Silverlight. For the Library of Congress: the deal is okay for now, but when the gift ends, to what extent will the money for software licenses and support have to be diverted from other operating budget items. For the users of the New Visitor Experience: probably a wash — visitors get a slick experience that could be replicated in any number of technologies and techniques. For users on the open web: a bad deal because their cultural heritage content has been put behind significant, if not insurmountable, barriers.