Comments on: Analysis of CDL’s XTF textIndexer to Replace the Local Files with FEDORA Objects

By: SourceForge.net: xtf-user

SourceForge.net: xtf-user — Wed, 13 Sep 2006 18:48:31 +0000

[...] To the FEDORA and XTF communities — At OhioLINK, we’re aggressively pursing the California Digital Library’s XTF software as a front end for our digital collections in a FEDORA content repository. I’ve written up some observations about what XTF integrated with FEDORA might look like and would welcome your comments and observations. We’d particularly like to know if anyone else is pursuing as similar path. The URLs are: http://dltj.org/2006/08/xtf-fedora-1/ http://dltj.org/2006/08/xtf-fedora-2/ Public comments (in the form of responses on the blog) or private ones (e-mail replies) would be most appreciated. Martin Haye, one of the lead developers of XTF, has been kind enough to offer some replies already and so far this seems like a viable solution. Peter [...]

By: the jester

the jester — Wed, 23 Aug 2006 13:57:43 +0000

[quote comment="2994"]One thought on lazy files: if you have some mechanism for random byte-level access to a data stream in the Fedora object, you could supply that to XTF through the StructuredStore interface. The interface was designed with this in mind, though we haven’t ever used it for that.[/quote]

Ah, that is a bit of a problem — at present, access to objects in FEDORA requires a Web Services call and the calls to retrieve data (through a “disseminator” or directly a datastream) do not support a byte-range request. (There has been some talk about this, so it might get done some day.)

[quote comment="2994"]What you call a “disseminator” is similar to subclasses of IndexSource, such as XMLIndexSource. There is also a PDFIndexSource and HTMLIndexSource that you might be interested in. If you end up creating others for Word or other formats, perhaps you’d consider contributing them back into XTF.[/quote]

Exactly — the point of distinction is where the transformation to XML takes place…as a subclass of IndexSource in XTF or as a function of the repository (the disseminator). In either case, the transformation is happening in a piece of Java code. My inclination, probably borne out by just raw familiarity at the moment, is to do the transformation as a FEDORA disseminator, but the code would certainly be used to write a new IndexSource class. I’ll let you know if we come up with anything new that might be useful to you.

Thank you for your comments! They have been most helpful.

The text was modified to update a link from http://comm.nsdl.org/pipermail/fedora-users/2006-May/001723.html to http://article.gmane.org/gmane.comp.cms.fedora-commons.user/1728/ on February 12th, 2011.

By: Martin Haye

Martin Haye — Tue, 22 Aug 2006 23:04:49 +0000

I agree with your strategy. However you get a set of documents from Fedora (I’m not that familiar with it), you basically want to replicate the functionality of SrcTreeProcessor, which does the work of wrapping the input sources and passing them to XMLTextProcessor for the heavy lifting.

One thought on lazy files: if you have some mechanism for random byte-level access to a data stream in the Fedora object, you could supply that to XTF through the StructuredStore interface. The interface was designed with this in mind, though we haven’t ever used it for that.

What you call a “disseminator” is similar to subclasses of IndexSource, such as XMLIndexSource. There is also a PDFIndexSource and HTMLIndexSource that you might be interested in. If you end up creating others for Word or other formats, perhaps you’d consider contributing them back into XTF.

By: Martin Haye

Martin Haye — Tue, 22 Aug 2006 22:52:49 +0000

One clarification before I get to general comments: processFile() forms the path for the lazy tree, but the file isn’t actually created until the queued documents are processed.