Mind-expanding topics this week. The threads start with a potentially morbid, but definitely intriguing, topic: what is to become of our personal digital legacies? If that isn’t enough to blow your mind, the next topic is an accounting of the amount of information processed in 2008. Still hanging in there? Then think about what could become of the book if we take advantage of its digital nature. You might not have much room to think big thoughts after those threads, but if you do the last one explores what could become of how our machines talk to each other.

My apologies for missing last week’s DLTJ Thursday Threads. I compose these entries on Wednesday evening and last week I snuggled with one of the children as he went to bed and I didn’t get back up again. That’s 31 straight weeks of Thursday Threads without interruption; not bad for an experiment many months ago. I haven’t mentioned this recently, so let me say it now: thank you for all of the positive feedback and for your interest in this series. It has been fun and useful to me to look back on the highlights of the week and put them in some context, and judging by the rising subscription count some readers find it useful too.

Feel free to send this to others you think might be interested in the topics. If you find these threads interesting and useful, you might want to add the Thursday Threads RSS Feed to your feed reader or subscribe to e-mail delivery using the form to the right. If you would like a more raw and immediate version of these types of stories, watch my FriendFeed stream (or subscribe to its feed in your feed reader). Comments and tips, as always, are welcome.

Personal Digital Legacies

Here it is. I’m dead, and this is my last post to my blog. In advance, I asked that once my body finally shut down from the punishments of my cancer, then my family and friends publish this prepared message I wrote—the first part of the process of turning this from an active website to an archive.

- The last post, Penmachine, Derek K. Miller

Ed Summers pointed to this blog post in a tweet in which Ed also said: “apart from being incredibly moving [this post] makes me wonder (again) what archiving services exist for depositing online work.” Also earlier this week was an article forwarded to me from Ron Murray in New Scientist with the title Digital legacy: Respecting the digital dead. The article covers the efforts of the British Library in working with personal digital archives, offering an overview of the techniques that border on the field of digital forensics to preserve the digital legacies of donated personal archives. And those two item follow a book I recently read called Your Digital Afterlife by Evan Carroll and John Romano (which itself I found by way of an NPR news story).

It has me wondering and considering (if not yet acting on) my digital legacy. Some of it is highly personal, like letting my spouse know how to access all of the digital bill paying sites that I use for the family’s finances. Other parts are fairly public, like what to do with this blog and my social media accounts. And it all brings me back around to Ed’s thought: is there a role for libraries in this space? Is some of what is represented in personal digital legacies the kind of highly-local content that libraries should be preserving? Or, put another way, how would our profession respond to the preservation desires of a patron who has used the Google tools shown in this 90 second video from Google (via the Google Operating System blog, Unofficial news and tips about Google).

Accounting of Information Processed

Comparisons of Digital Capacity Measurements

Three years ago, the world’s 27 million business servers processed 9.57 zettabytes, or 9,570,000,000,000,000,000,000 bytes of information.

Researchers at the School of International Relations and Pacific Studies and the San Diego Supercomputer Center at the University of California, San Diego, estimate that the total is equivalent to a 5.6-billion-mile-high stack of books stretching from Earth to Neptune and back to Earth, repeated about 20 times.

- World’s servers process 9.57ZB of data a year, by Lucas Mearian, NetworkWorld

Those numbers come from a news release of a report from the University of California at San Diego that measures the amount of information swirling around the world’s computers. And it includes this handy table of comparisons of digital capacity measurements that attempts to put it into perspective. (Although trying to imaging the number of home computer hard drives in the state of Minnesota still boggles the mind.) I also find it to be a nice reality check. After all, my field’s contribution to that number can’t be too big, can it? So our problems really aren’t that big at all…

The Digital Book

Presentation by Mike Matas at TED (4 minutes)

The idea behind Push Pop Press [is] a digital creation tool designed to blow up the concept of the book. Frictionless self-publishing is a fertile new space, but this particular startup got a little help from former vice president Al Gore, whose exacting demands on an app version of his book Our Choice: A Plan to Solve the Climate Crisis gave this would-be company its first real boost.

Developed by former Apple employees Mike Matas and Kimon Tsinteris, Push Pop Press will be a publishing platform for authors, publishers and artists to turn their books into interactive iPad or iPhone apps — no programming skills required.

- Gore, Ex-Apple Engineers Team Up to Blow Up the Book, by Brian X. Chen, Wired.com Gadget Lab

I won’t call what these gentlemen show an “ebook” — something that brings to my mind static words on a digital page. No, this is a digital book — something wholly new to the process of communicating ideas from author to reader. To see why this is different, watch the four minute TED Talk video.

New Ways of Internetworking

Imagine a web where our browsers connected directly to each other to do voice, video, media sharing and run applications, using P2P and real-time APIs, rather than going through centralized servers that controlled traffic and permissions. That’s a potent idea and if implemented properly could future-proof a part of the web from authoritarian crack-downs, disruptions by disasters and more. It could also establish a permanent lawless zone of connected devices with no central place to stop anyone from doing anything in particular.

It just so happens that something like that may now be under development in the most official of venues. The World Wide Web Consortium (W3C) announced today the formation of a new Web Real-Time Communications Working Group to define client-side APIs to enable Real-Time Communications in Web browsers, without the need for server-side implementation. The Group is chaired by engineers from Google and Ericsson.

- This Could be Big: Decentralized Web Standard Under Development by W3C, by Marshall Kirkpatrick, ReadWriteWeb

A team of researchers at Rutgers University have launched the latest of a group of wireless network initiatives aiming to create a more open alternative to the Internet. MondoNet aims to enable a mesh network that lets a hybrid collection of new and existing Wi-Fi, WiMax and other wireless devices connect to each other without going through a central carrier.

A draft proposal for MondoNet describes its premise as well as how it will gather the best of existing technologies for mobile ad-hoc wireless mesh networks (MANETs). The project’s goal to create a system that provides both greater freedom and privacy for individual users than today’s Web.

- Rutgers team proposes Net alternative, Rick Merritt, EE Times

I’m including these as forward-looking points of interest. I don’t know if either will amount to anything substantial, but I do think it is interesting that researchers are looking at the next evolutionary steps in networking. Either of these proposals would be dramatic changes in flows of information between network users. I like how both seem to be building in a fundamental pillar of privacy into the design.

About two years ago I wrote a blog post wondering if we could outsource the preservation of digital bits. What prompted that blog post was an announcement from Iron Mountain of a Cloud-Based File Archiving service. Since then there have been a number of other services that have sprung up that are more attuned to the needs of cultural heritage communities (DuraCloud and Chronopolis come to mind), but I have wondered if the commercial sector had a way to do this cheaply and efficiently. The answer to that question is “maybe not” as Iron Mountain has told Gartner Group (PDF archive) that it is closing its Virtual File Store services and its Archive Service Platform.

The Gartner analysis goes on to say: “Virtual File Store customers that stay with Iron Mountain will be transferred to a higher-value offering, File System Archiving (FSA) in 2012. The new offering will be a hybrid that leverages policy-based archiving on site and in the cloud with indexing and classification capabilities.” The Register has more details and speculation about what happened. As always, the full story might be more interesting that what the news reports are saying. In any case — just to close this loop — if you were thinking of trying this particular option, think no further.

The text was modified to remove a link to http://www.ironmountain.com/digital-archiving/inactive-data-file-archiving.html on June 9th, 2011.

This week Amazon takes center stage of DLTJ Thursday Threads with a report of their new Kindle Singles program for medium-form digital content and a screen-reader-aware version of the Kindle reader application for PCs. After that is a look at how scholarly discourse is changing — radically! — with the availability and use of near-real-time feedback loops. And we close out with a peek at shaky ground in the world of ISBN identifiers.

As a sidenote to last week’s comment about this blog migrating to Amazon’s service…there are still a few hiccups. For instance, last week’s edition of DLTJ Thursday Threads wasn’t published via the RSS feed until late in the day and it wasn’t until Friday that the e-mail subscribers received it. I think those issues are ironed out now, but if you notice any other problems please let me know.

Kindle Singles — Compelling Ideas Expressed at Their Natural Length — Now Available in the Kindle Store

Before the advent of digital reading, writers often had to choose between making their work short enough for a magazine article or long enough to deliver the “heft” required for book marketing and distribution. Three months ago, Amazon made a call to serious writers, thinkers, scientists, business leaders, historians, politicians and publishers to join Kindle in making a new kind of content available to readers—Kindle Singles. Typically between 5,000 and 30,000 words, each Kindle Single is intended to allow a single killer idea — well researched, well argued and well illustrated — to be expressed at its natural length. Today, Amazon is introducing the first set of Kindle Singles to the Kindle Store. …

The new Kindle Singles section of the Kindle Store is now available at www.amazon.com/kindlesingles. Available to both Kindle device and app users, and priced between $0.99 and $4.99, the first set of Kindle Singles include original reporting, essays, memoirs and fiction. Amazon plans to frequently launch many more Kindle Singles over time.

Is there room for commercial content between “short enough for a magazine article” and a full-fledged book? Amazon seems to think so with this announcement of the Kindle Singles program. Among the first are three works from TEDTalk speakers priced at $2.99 each. The content is only available in digital form and only in the proprietary Kindle format. This may be a problem for a library trying to acquire this content for its collection (although this is just a subset of the more general issue of acquiring content saddled in proprietary formats with restrictive digital rights management). What makes this problem more acute, though, is that Amazon is seeking high quality content for the Kindle Singles channel (“Singles will be a highly curated group of content they feel is valuable to their readers” according to the Kindle Expert website). That might make the content more desirable by patrons and more likely to be considered preservation-worthy. (You can read about one author’s perspective on publishing in the Kindle Singles program.)

Kindle for PC with Accessibility Plugin

Kindle for PC with Accessibility Plugin is a free application for your Windows PC. It provides the following accessibility features:

• Text-to-speech reading with adjustable voice settings
• Voice-guided menu navigation
• Large font sizes
• High contrast reading mode
• Keyboard navigation
• Accessible shortcuts

Because this software is an assistive technology, there are no restrictions on text-to-speech reading. In order to use the text-to-speech feature, an external screen reader program must be installed and running on the Windows PC. Tested screen readers include: JAWS and NVDA. An external screen reader is used to read aloud menus and navigation items, while book text is read by a built-in text-to-speech engine.

Although I’m hard pressed to find the formal announcement, a version of the Kindle for PC with Accessibility Plugin was made available earlier this month. The National Federation of the Blind has a review of the software with some constructive criticism that hopefully Amazon will take to heart. What is interesting is that one can use a screen reading program such as the commercial Jaws for Windows or the open source NonVisual Desktop Access (NVDA) to have the text of the book read aloud “regardless of a publisher’s [text-to-speech] … choice.” If you are serving a population of users with a sight impairment, this may be an option to look at to expand the universe of accessible materials to everything available in the Kindle store.

Peer review: Trial by Twitter

For many researchers, the pace and tone of this online review can be intimidating — and can sometimes feel like an attack. How are authors supposed to respond to critiques coming from all directions? Should they even respond at all? Or should they confine their replies to the conventional, more deliberative realm of conferences and journals? “The speed of communication is ahead of the sheer time needed to think and get in the lab and work,” said Felisa Wolfe-Simon, a postdoctoral fellow at the NASA Astrobiology Institute in Mountain View, California, and the lead author on the arsenic paper. Aptly enough, she circulated that comment as a tweet on Twitter, which is used by many scientists to call attention to longer articles and blog posts.

To bring some order to this chaos, it looks as though a new set of cultural norms will be needed, along with an online infrastructure to support them. The idea of open, online peer review is hardly new. Since Internet usage began to swell in the 1990s, enthusiasts have been arguing that online commenting could and should replace the traditional process of pre-publication peer review that journals carry out to decide whether a paper is worth publishing.

This article in Nature News points out the problem when commentary on scientific studies moves at Twitter speed. The old mechanisms of published peer-reviewed articles followed by commentary in later issue of the same journal in the form of published letters is being challenged by the internet world of blogs and tweets. As the author says, a new form of cultural norms is required as well as mechanisms to track the discourse. [Via Eric Schmell]

eBook Identifier Confusion Shakes Book Industry

Last Thursday, I was fortunate to be at a presentation of the Book Industry Study Group (BISG) about identification of eBooks. BISG hired Michael Cairns, the principal of Information Media Partners, to do a study of the use, issues and practice surrounding assignment of ISBNs in the US book industry. Think of him as a structural engineer hired to inspect the damage to the supply chain’s supporting infrastructure after an earthquake. Cairns conducted 55 separate interviews with a total of 75 industry experts from all facets of the industry. (I was interviewed for my expertise in the use of ISBN in library linking systems).

• BISG eBook ISBN Study Findings Released Michael Cairns’ blog
• Summary of BISG Presentation From BISG, PDF 730 KB

Cairns (@personanondata on Twitter) is an industry veteran- he’s held senior executive positions at Bowker and other companies. His presentation was clear and direct, and he quickly went to the heart of the matter. He found very little support for the policy set forth by the 2005 revision of the ISBN standard regarding when to assign a new ISBN to an ebook.

Eric Hellman writes about his views of the dysfunction surrounding ISBN assignments for ebooks. “What problems?” you might ask — Eric writes has an example of how Barnes and Noble was enhancing some ebooks for their Nook platform. By itself, this activity wouldn’t result in assigning a new ISBN. But because publishers are now exerting more control over setting the prices of ebooks (the so-called “agency model“) the existence of these Nook-enhanced versions needs to cross back-and-forth between the publisher’s and retailer’s electronic systems. The only commonly agreed upon identifier? The ISBN. And this proliferation of ISBN assignments is making trouble for library’s efforts to effectively identify material — which is to say nothing about what it is doing to our efforts to shoehorn these distinctions between various works into the MARC format used by our catalogs. Is that a separate record for that manifestation with a different ISBN?

A last-minute change to my plans for ALA Midwinter came on Tuesday when I was sought out to fill in for a speaker than canceled at the ALCTS Digital Preservation Interest Group meeting. Options for outsourcing storage and services for preserving digital content has been a recent interest, so I volunteered to combine two earlier DLTJ blog posts with some new information and present it to the group for feedback. The reaction was great, and here is the promised slide deck, links to further information, and some thoughts from the audience response.

Slide Deck and References

To examine the similarities and differences in costs, let’s use the OhioLINK Satellite Image collection as a prototypical example. It consists of about 2 terabytes (2TB) of high-quality images in TIFF format, with about 7.5GB of data going into the repository each month. In the interest of exploring everything that S3 can do, there is an assumption that approximately 4GB of data will be transferred out of the archive each month; OCLC’s Digital Archive does not have a end-user dissemination component.

The point of showing this scenario is to show the widest range of costs — from a storage-only solution like Amazon S3 to a soup-to-nuts service like OCLC Digital Archive. A word about the redacted costs. Some of the numbers for OCLC’s Digital Archive response (from 2008) came from a confidential quote, so the numbers were removed from the public table. For the numbers that are publicly listed, the values come from Barbara Quint’s article.

The articles and blog posts I referenced in the course of the presentation were:

Iglesias, Edward and Wittawat Meesangnil (2010). Using Amazon S3 in Digital Preservation in a mid sized academic library: A case study of CCSU ERIS digital archive system. The Code4Lib Journal, issue 12, retrieved 5-Jan-2011 from http://journal.code4lib.org/articles/4468

Murray, Peter (2008). Long-term Preservation Storage: OCLC Digital Archive versus Amazon S3. Disruptive Library Technology Jester. Retrieved 5-Jan-2011 from http://dltj.org/article/oclc-digital-archive-vs-amazon-s3/

Murray, Peter (2009). Can We Outsource the Preservation of Digital Bits?. Disruptive Library Technology Jester. Retrieved 5-Jan-2011 from http://dltj.org/article/outsource-digital-bits/

Quint, Barbara (2008). OCLC Introduces High-Priced Digital Archive Service. Information Today. Retrieved 5-Jan-2011 from http://newsbreaks.infotoday.com/nbReader.asp?ArticleId=49018

• Amazon S3. Retrieved 5-Jan-2011 from http://aws.amazon.com/s3/
• Chronopolis. Retrieved 5-Jan-2011 from http://chronopolis.sdsc.edu/
• DAITSS. Retrieved 5-Jan-2011 from http://daitss.fcla.edu/
• DuraCloud. Retrieved 5-Jan-2011 from https://wiki.duraspace.org/display/duracloud/DuraCloud
• Iron Mountain. Retreived 5-Jan-2011 from http://www.ironmountain.com/news/2009/impr02232009.asp
• OCLC Digital Archive. Retrieve 5-Jan-2011 from http://www.oclc.org/us/en/digitalarchive/
• Private LOCKSS Networks. Retrieved 5-Jan-2011 from http://lockss.stanford.edu/lockss/Private_LOCKSS_Networks

Some Thoughts

With the close of the year approaching, this issue marks the 14th week of DLTJ Thursday Threads. This issue has a publisher’s view of Amazon’s strong-arm tactics in book pricing, research into the possibility that academic authors could game Google Scholar with spam, demonstrations of how Amazon’s Mechanical Turk drives down the cost of enlisting humans to overwhelm anti-spam systems, and a story of multispectral imaging adding information in the process of digital preservation.

As the new year approaches, I wish you the best professionally and personally.

Books After Amazon

What happens when an industry concerned with the production of culture is beholden to a company with the sole goal of underselling competitors? Amazon is indisputably the king of books, but the issue remains, as Charlie Winton, CEO of the independent publisher Counterpoint Press puts it, “what kind of king they’re going to be.” A vital publishing industry must be able take chances with new authors and with books that don’t have obvious mass-market appeal. When mega-retailers have all the power in the industry, consumers benefit from low prices, but the effect on the future of literature—on what books can be published successfully—is far more in doubt.

Onnesha Roychoudhuri publishes this view of Amazon’s marketing practices in the lastest issue of the Boston Review. From the publisher’s pespective, the strong-arm tactics described sound horrible. But the story also points to cracks appearing — at least for the bigger publishers. That may leave smaller, independent publishers in a big squeeze. [Via OCLC Research's Above-the-Fold]

Academic Search Engine Spam and Google Scholar’s Resilience Against it

Abstract: In a previous paper we provided guidelines for scholars on optimizing research articles for academic search engines such as Google Scholar. Feedback in the academic community to these guidelines was diverse. Some were concerned researchers could use our guidelines to manipulate rankings of scientific articles and promote what we call ‘academic search engine spam’. To find out whether these concerns are justified, we conducted several tests on Google Scholar. The results show that academic search engine spam is indeed—and with little effort—possible: We increased rankings of academic articles on Google Scholar by manipulating their citation counts; Google Scholar indexed invisible text we added to some articles, making papers appear for keyword searches the articles were not relevant for; Google Scholar indexed some nonsensical articles we randomly created with the paper generator SciGen; and Google Scholar linked to manipulated versions of research papers that contained a Viagra advertisement. At the end of this paper, we discuss whether academic search engine spam could become a serious threat to Web-based academic search engines.

Joeran Beel and Bela Gipp have this article in the most recent issue of Journal of Electronic Publishing. In addition to being able to game Google Scholar, the authors note that Microsoft Academic Search and CiteSeer (as well as their own academic search engine currently under development — SciPlore) have the same issues. Although it is possible, we don’t know if it is being done — or even if there would be an penalties in the academic community for doing so.

Mechanical Turk: Now with 40.92% spam

At this point, Amazon Mechanical Turk has reached the mainstream. Pretty much everyone knows about the concept. Post small tasks online, pay people cents, and get thousands of micro-tasks completed. Unfortunately, this resulted in some unfortunate trends. Anyone who frequents just a little bit the market will notice the tremendous number of spammy HITs. (HIT = a task posted for completion in the market; stands for Human Intelligence Task). “Test if the ads in my website work”. “Create a Twitter account and follow me”. “Like my YouTube video”. “Download this app”. “Write a positive review on Yelp”. A seemingly endless amount of spam HITs come to the market, mainly with the purpose of spamming “social media” metrics. So, with Dahn Tamir and Priya Kanth (MS student at NYU), we decided to examine how big is the problem. How many spammers join the market? How many spam HITs are there?

This post from Panos Ipeirotis, Associate Professor at the IOMS Department at Stern School of Business of New York University, describes a review of activities posted to Amazon’s Mechanical Turk service. Spam is everywhere, and it appears that the Mechanical Turk is reducing the friction between buyers and workers of spam activity. [Via Ron Murray]

Cutting-Edge Imaging Helps Scholar Reveal 8th-Century Manuscript

With a manuscript like the St. Chad Gospels, multispectral imaging—a series of scans, each based on a single part of the color spectrum—allows his team to create images that have the equivalent of three-dimensional detail, down to revealing the thickness of brush strokes on letters and illustrations. Cockled pages can be virtually flattened out so that all their details can be studied. Studied color band by color band, the chemical composition of ink can be determined.

This article by Jennifer Howard at the Chrnoicle of Higher Education reviews the story of how 8th-century documents in England were digitized by scholars at the University of Kentucky. It caught my eye because of the mention of multispectral imaging; this is something that the JPEG2000 file format can natively store. Digitization at this level doesn’t just provide alternative, online access to documents — it actually adds new information to the process of researching those documents. [Note: the link is behind a publisher paywall. If you would like to see it, send me an e-mail and I'll forward you a short-term link from the Chronicle's website.]

Earlier this month, the JPEG 2000 Implementation Working Group, the Wellcome Trust Library, and the U.K. Digital Preservation Coalition hosted a free one-day seminar called JPEG2000 for the Practitioner. The presentation slides are now linked to the seminar program and is a short report of the event by Christy Henshaw of Wellcome Library. The presentation slides by themselves carry a great deal of depth even without a recording of the audio. In particular I can recommend “What did JPEG 2000 ever do for us?” by Simon Tanner and “JPEG 2000 standardization – a pragmatic viewpoint” by Richard Clark. As brief introductions to where we’ve been with JPEG 2000 and where we could go.

Hat tip to Ron Murray for pointing this out to me.

A colleague forwarded an article from The Register with news of a new service from Iron Mountain for Cloud-Based File Archiving. It is billed as a “storage archiving service designed to help companies reduce costs of storing and managing static data files.” My place of work is facing an increasing need large-scale digital preservation storage with the acquisition of a large collection of music and the conversion of our educational videos from physical DVD preservation to digital preservation. We’re talking terabytes of content that is we need to keep in its archival form — uncompressed, high quality media files (not the lower quality, derivatives for day-to-day access). It doesn’t make sense to keep that on expensive SAN storage, of course, so this article struck me at just the right time to consider alternatives.

Architecture Diagram for Iron Mountain's Virtual File Store service, showing the placement of the Virtual File Store appliance relative to other assets on the data center network. Graphic from product datasheet (http://www.ironmountain.com/resources/vfs/virtual_file_store_datasheet.pdf)

According to the product literature, the service works by putting a black box on your network where one can drop files via CIFS or NFS. The black box transfers the files over the internet to two Iron Mountain data centers. Files can then be retrieved via an on-line on-demand service or by exchanging physical media with Iron Mountain for bulk retrieval needs.

We know, of course, that digital preservation is more than just preserving the digital bits: it is the intellectual exercise of describing the stored information, the effort of maintaining an accurate catalog of that information, and the burden of migrating file formats or emulating platforms to read old file formats. Handling the raw bits is a big deal, too — checksumming to ensure unaltered status, refreshing files to new storage media, and protection from physical disasters. This Iron Mountain solution seems to address this more mechanical portion of digital preservation, and it one that probably can benefit from aggregating the service needs of many customers (and so is ripe for outsourcing).

Is anyone doing something similar with their physical preservation of digital media? Are there other companies that do the same thing? (I know of OCLC’s Digital Archive service — I did a comparison of it with Amazon S3 last year.)

David Lowe, Preservation Librarian at the University of Connecticut, is coordinating a survey of JPEG2000 use for digital imagery. The survey asks questions about the use of the JPEG2000 file format (for archival purposes or for access systems), tools used (both JPEG2000 toolkits and software that embeds JPEG2000 toolkits), and considerations of mathematically lossless versus visually lossless compression settings.

This is his announcement:

I am writing to solicit your help with a survey of library-related digital project staff regarding the implementation of the JPEG 2000 standard for digital images (specifically still images and not motion). We estimate that this task will take approximately 15 minutes of your time. It is available now at: http://www.surveymonkey.com/s.aspx?sm=WXFAJwyRNZZilRWzrnum_2fw_3d_3d

The survey will remain active until October 31, 2008. Afterward, we will post the results via a report uploaded to our institutional repository, digitalcommons.uconn.edu.

Please note that in our report, personal information from the survey will not be revealed, and any comments used will remain unattributed unless the respondent prefers to be credited and indicates that desire in a separate email to me directly at david.lowe@uconn.edu.

Thank you for your help,

David Lowe
Preservation Librarian
UConn Libraries

I encourage you to take the survey as well.

Last month OCLC announced a new service offering for long-term storage of libraries’ digital collections. Called Digital Archive™, it provides “a secure storage environment for you to easily manage and monitor the health of your master files and digital originals.” Barbara Quint has an article in Information Today called “OCLC Introduces High-Priced Digital Archive Service” in which she makes a comparison to Amazon’s Simple Storage Service (or “S3″) from primarily a cost perspective: “The price for S3 storage at Amazon Web Services is 15 cents a gigabyte a month or $1.80 a year, in comparison to OCLC’s $7.50 a gig.” Barbara also goes into some of the technical differences, but I think it might be worthwhile to go a little more into depth on them.

OCLC’s Digital Archive

• Systems management
• Physical security
• Data security
• Data backups
• Disaster recovery
• ISO 9001 certification
• Manifest verification
• Virus check
• Format verification
• Fixity check

It is targeted towards the preservation of digital masters. There is a document on the Digital Archive website called Our commitment that describes other aspects of a digital preservation program: “OCLC is actively developing processes for full preservation of digital assets to ensure complete renderability, regardless of technology changes. This preservation system will likely involve a combination of migration and emulation.” But it is not clear whether these services, beyond “bit preservation” activities, is part of the Digital Archive service or will be part of an add-on service to be developed later.

This “Digital Archive” is a revamping of an older product from OCLC, also called “Digital Archive” but one that included a web harvesting tools component. The service and support documentation on the OCLC website still refers to the former version of Digital Archive, so there is little information about how the service works beyond what one can infer from the sales information.

Amazon’s S3

SLA seems to extend only to availability of the service, not to a long term commitment to keeping track of files on the service.

AWS [Amazon Web Services, LLC] will use commercially reasonable efforts to make Amazon S3 available with a Monthly Uptime Percentage (defined below) of at least 99.9% during any monthly billing cycle (the “Service Commitment”). In the event Amazon S3 does not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.

There is no mention specifically in the S3 SLA about permanence of file storage. In leu of that, one seems to be covered by the overarching Amazon Web Services Customer Agreement, which has several points of interest from a preservation use perspective:

3.3. Termination or Suspension by Us Other Than for Cause.
3.3.2. Paid Services…. We may suspend your right and license to use any or all Paid Services (and any associated Amazon Properties)…, or terminate this Agreement in its entirety (and, accordingly, cease providing all Services to you), for any reason or for no reason, at our discretion at any time by providing you sixty (60) days’ advance notice in accordance with the notice provisions set forth in Section 15 below.

So if they desire to terminate a library’s use of the service (assuming there was no specific cause — such as a violation of the terms of use — to do so), they have to give 60 days notice. That’s when the “Data Preservation in the Event of Suspension or Termination” clause kicks in:

3.7.2. In the Event of Termination Other Than for Cause. In the event of any termination by us of any Service or any set of Services, or termination of this Agreement in its entirety, other than a for cause termination under Section 3.4.1, (i) we will not take any action to intentionally erase any of your data stored on the Services for a period of thirty (30) days after the effective date of termination; and (ii) your post termination retrieval of data stored on the Services will be conditioned on your payment of Service data storage charges for the period following termination, payment in full of any other amounts due us, and your compliance with terms and conditions we may establish with respect to such data retrieval.

The customer agreement then goes on to say:

3.8. Post-Termination Assistance.Following the suspension or termination of your right to use the Services by us or by you for any reason other than a for cause termination (i.e., a termination under Section 3.2 or under Section 3.3), you shall be entitled to take advantage of any post-termination assistance we may generally make available with respect to the Services, such as data retrieval arrangements we may elect to make available. We may also endeavor to provide you unique post-suspension or post-termination assistance, but we shall be under no obligation to do so. Your right to take advantage of any such assistance, whether generally made available with respect to the Services or made available uniquely to you, shall be conditioned upon your acceptance of and compliance with any fees and terms we specify for such assistance.

Perhaps the most troubling aspect, from a preservation point-of-view, deals with data security and backups. Specifically, Amazon says that data security and backups are the responsibility of the customer. The Amazon Web Services Customer Agreement says (emphasis added):

7.2. Security. We strive to keep Your Content secure, but cannot guarantee that we will be successful at doing so, given the nature of the Internet. Accordingly, without limitation to Section 4.3 above and Section 11.5 below, you acknowledge that you bear sole responsibility for adequate security, protection and backup of Your Content. We strongly encourage you, where available and appropriate, to use encryption technology to protect Your Content from unauthorized access and to routinely archive Your Content. We will have no liability to you for any unauthorized access or use, corruption, deletion, destruction or loss of any of Your Content.

That kind of security and data backup is something you’d want in a preservation service. Since activities against S3 storage is limited only by a knowing a private “key”¹ (as opposed to limiting to particular IP addresses or not allowing deletes/modifications from the web at all), it is a real possibility that the archive can be harmed if the private key is disclosed. Furthermore, S3 does not have a backup/restore service for retrieving files that were accidentally or maliciously deleted.

Feature Comparison

This is a useful comparison because it would indicate what one would have to layer on top of S3 to reach the level of service provided by Digital Archive. For instance, it would be possible to create an application that would perform the manifest and format verifications as well as the periodic virus and fixity checks against the files in S3. It would even be possible to run that application in Amazon’s Elastic Compute Cloud (EC2) — a “virtual computing environment” that allows developers to easily create and deploy software on the internet. Since data transferred between Amazon EC2 and Amazon S3 is free of charge, there wouldn’t be the S3 cost of periodically downloading the data to perform the virus and fixity checks.

One advantage to note about the S3 solution is that it can perform as a “light archive” — meaning the data is available to users in addition to being part of the content repository. In contrast to the OCLC Digital Archive service — a “dark archive” — access to the data is highly or completely restricted. Still, the lack of automated backups and a robust data security infrastructure in the S3 infrastructure are notable from a preservation data service perspective.

Cost Comparison

For this prototypical example, S3 would cost $3,800 in the first year and roughly $3,615 per year after that, with the added benefit that end-users could access the content without using our infrastructure. There are costs associated with the OCLC Digital Archive service that had to be redacted from the public version of this table due to a confidentiality clause, but the costs that are assumed for ongoing storage based on Barbara Quint’s article are comparable to the quote I got from OCLC and represent a large portion of the total yearly costs.

By way of comparison, we are planning the purchase of 50TB of storage this summer for roughly $250K; that is about $5,000/TB. Amortize the cost of the hardware over five years and assume 150% of the purchase price represents maintenance, personnel support, and other factors, and we get $2,500/TB/year. This doesn’t include software costs, so it is comparable to S3 in the functions table above; software would have to be written to verify the manifest and file formats on ingest as well as the monthly fixity and virus scanning. It also represents only one copy of the data; it does not include the duplication across data centers that both Digital Archive and S3 provide.

Conclusions

OCLC’s Digital Archive product goes pretty far down the path of a preservation-worthy archive of digital files. The value-added services, in addition to simply storing and retrieving files, make it as close to a one-stop shop as I’ve seen so far. Whether outsourced digital preservation services makes sense — particularly at this price point — remains to be seen, especially since is hard to make a comparison since I’m betting that most of us aren’t (yet) doing all of the ongoing activities with digital preservation masters that Digital Archive is doing.

Amazon’s S3 is an inexpensive, network-oriented file hosting service, and as such it doesn’t have many of the features built into it that we would want to see in a preservation archive service. Beyond raw file service, one would need to add layers of software and human activities to perform the functions that Digital Archive provides now.

Looking at OCLC’s Digital Archive and Amazon S3 is almost an apples-to-oranges comparison, both in price and in functionality. Comparing functionality first, S3 is missing critical components of a preservation storage system — namely, rigorous access control and a content backup/restore facility. Comparing costs, though, S3 is dramatically cheaper…and has the benefit of serving up large files to end-users using Amazon’s distributed infrastructure.

It is possible to level the functionality playing field a bit by taking responsibility for the ongoing maintenance of files in the S3 archive — those things that Digital Archive offers as value-added services over raw file storage. An EC2 virtual machine running in Amazon’s infrastructure can perform the virus and fixity scanning. And with good key maintenance (as with passwords, regularly changing the private key and securing it appropriately), S3 could conceivably offsite copies of content stored offline (e.g. burned to preservation quality optical media). Again, in this scenario one has to take responsibility for refreshing the offline media and occasionally running comparisons against the S3 offsite copy.

Footnotes

1. S3 uses secret keys — a 40-character password — to verify the identify of the client making the request. If the private key becomes known, anyone on the internet can perform operations actions as the content owner.

My place of work is looking to acquire educational videos in a digital form with an eye towards long-term preservation. At this point we receive a physical form (preferably DVD, but sometimes VHS) and digitize it to a very lossy access format (RealMedia, in this case). With this change, we would get a preservation-worthy digital copy from the producer/distributor and forego the physical version.

There is quite a lot written on preserving video, but I wanted to distill the requirements down into statements that vendors could reasonably provide today. I think these are pretty sound requirements, but I’m looking for feedback. In particular, I’m not quite sure how to handle the transfer of closed caption text from the publisher/distributor; suggestions are welcome.

[Jester's note: I just realized that an earlier version of this posting went out to the net about two hours before this "final" version. Sorry about publishing the work-in-progress early; I must have hit the wrong button in the new version of WordPress...]

File Formats

Guidance on the preservation of digital video should, by necessity, change over time. [...] The MPEG-2 and MPEG-4 formats are better suited to high-quality digital video. MPEG-2 is better known for its use as a format for DVD-Video, which encourages confidence when considering the likelihood that the format will be readable in the long-term. The format has an average transfer rate of 2-5 megabits per second, but there may be disk space restraints and the software tools necessary to convert and store this format are costly. MPEG-4 has a lower transfer rate of 1-2 megabits per second and is intended for streaming video. Other codecs, such as QuickTime, Windows Media, Real Video and Open DIVX, are useful for specific purposes, but not suitable for preservation. ¹

The Library of Congress Sustainability of Digital Formats site has an entry for MPEG-2 (also known as H.262) and an entry for MPEG-4 (more completely, MPEG-4 file format version #2) that give the nitty-gritty details for the file formats.

The preservation master copies we want to store has a frame size of 720 pixels by 480 pixels. (That size is for NTSC format videos, common in USA, Canada and Japan. Master copies of PAL-format videos, common in Australia, New Zealand, the United Kingdom and most of Europe, is 720 x 576.) This is the standard resolution used in MPEG-2-compressed commercially distributed DVD movies.² These frame sizes are appropriate for analog video signals. (“As defined by ITU-R Recommendation BT.601, more commonly know by the abbreviations Rec. 601 or BT.601 or its former name, CCIR 601. [It is] a standard published by the CCIR (now ITU-R) for encoding interlaced analogue video signals in digital form.”³ ) The audio is 48KHz stereo at 224 kb/s or better.

Captioning Text

To make matters more complicated, a whole set of different schemes are used for DVDs. (On VHS recordings, closed caption text was encoded in one of the non-visible lines that make up the video signal. Since the DVD format only included visible lines, other schemes were required.) The most popular seems to be the Scenarist Closed Caption (SCC) format. This is a binary file that exists on the DVD along side the video files.

Resources Consulted

The text was modified to update a link from http://ahds.ac.uk/ to http://www.ahds.ac.uk/ on January 28th, 2011.

The text was modified to update a link from http://ahds.ac.uk/preservation/ahds-preservation-documents.htm to http://www.ahds.ac.uk/preservation/ahds-preservation-documents.htm on January 28th, 2011.

The text was modified to update a link from http://ahds.ac.uk/preservation/video-preservation-handbook.pdf to http://www.ahds.ac.uk/preservation/video-preservation-handbook.pdf on January 28th, 2011.

The text was modified to update a link from http://ahds.ac.uk/preservation/video-preservation-handbook.pdf to http://www.ahds.ac.uk/preservation/video-preservation-handbook.pdf on January 28th, 2011.

The text was modified to update a link from http://ncam.wgbh.org/publications/adm/guideline_h.html to http://ncam.wgbh.org/invent_build/web_multimedia/accessible-digital-media-guide/guideline-h-multimedia on January 28th, 2011.

Footnotes

1. Knight, G., & McHugh, J. (2005). Preservation Handbook: Moving Image. p. 3.
2. Audio/Video Capture and Management (2002).
3. “Rec. 601″ (2008).