Last week I posted about a Yahoo Pipes construct that turns a Zotero website library into an RSS feed. As Dan Cohen noted in a twitter response to the posting, the Zotero team is planning to add an RSS capability in a future release of the website, so this pipe will ultimately be usurped by that capability, but in the meantime it is a handy tool. It was my first full-scale foray into creating a Yahoo Pipes construct from scratch, so I thought it would be useful to document how it works (in case I need to do something similar again). You might find this useful, too; especially the part about how to put a pubDate element into the RSS feed.

A couple of things to note: The basic structure is to iterate over a list of items. The modules on the right side (above) work to construct a URL to the Zotero Library page while the modules on the left actually get that page, find the list of items (by defining where the list is located in the HTML structure), and massage them into an RSS feed. In the case of this example, the Pipe is iterating over a list of <tr> elements in the table that displays the library items. Strings with elements separated by periods are XPATH-like constructs. For instance, the “Path to Item List” in the upper right module is body.div.1.div.div.1.table.tbody.tr. The corresponding XPATH is /html/body/div[2]/div/div[2]/table/tbody/tr. Note a few things: the leading element (html) is omitted, slashes are replaced by periods, and ordinal numbers are separated from their base elements with periods and are subtracted by 1. Later on there are examples where attribute names (“href“) and references to text nodes (“content“) are also included in these path strings.

Graphic representation of the Yahoo! Pipes construct to turn a Zotero library web page into an RSS feed.

The process starts with the two Private Text Input modules at the top right — one each for the Zotero Username and the Zotero User Number. The defaults are set to my values, and they are marked private — meaning that if someone clones this pipe, these values are not carried along.

Directly below is a URL Builder module. The base URL is http://www.zotero.org, and there are three path elements: the first is a connection from the Zotero Username Private Text Input, the second is a connection from the Zotero User Number Private Text Input, and the third is the literal “items“. This builds a URL that looks like http://www.zotero.org/dltj/683/items and that corresponds to the Zotero user’s library items page.

Starting at the upper left of the diagram, the output of the URL Builder is connected to the URL field of a Fetch Data module. The “Path to Item List” parameter is set to body.div.1.div.div.1.table.tbody.tr and that is a pointer to the portion of the XHTML document that contains the library items. Because the Zotero website is outputting XML (as XHTML), we can use the Fetch Data module and parse the page as if it was an XML document. The Path to Item List is an XPATH-like structure that points into the document structure (see note above). The result of this module is a list of items — the table rows in this case — that are processed by the remaining modules.

The next module down is a Rename module, where the value of the XPATH-like path item.td.0.a.content is copied to the item title element. The XPATH, from the root of the “Path to Item List” in the module above, is td/a; note here the added item at the front and the content at the end. Specifying the zeroth td element isn’t needed, but it brings symmetry with subsequent modules. content corresponds to the text node under the a element when viewing this as an XML document.

What follows is a series of Loop modules that act on different parts of the items in the list. The first builds the link element of the item using the String Builder module. The href in the XML is a relative path, so the String Builder adds the literal “http://www.zotero.org” to the value found in item.td.0.a.href (the href attribute of the anchor element of the first td element). The resulting string is assigned to the link element of each item in the list.

The second Loop module encapsulates a Date Builder module, and this is the inspiration for writing this post. It took a very long time to figure out how to get the pubDate element into each item of the resulting RSS feed. As it turns out, one cannot simple assign the pubDate element like we did the title element above. Instead, one sets the timestamp to the y:published element and Yahoo Pipes takes it from there. And it isn’t enough to assign a text string to that element; it has to be a Date type, constructed using the Date Builder module. The Date Builder module is very flexible in what it accepts, and it creates a canonical timestamp form that can be used by other modules. In this case, the Date Builder module takes as a source input the string value found at item.td.2.a.content. Believe me — it took a long time to figure this out, and it was only done by piecing together various suggestions and examples; there doesn’t seem to be any clear documentation about this.

The third and fourth Loop modules go together. The third takes the value found in item.td.0.a.span.class and applies a String Regex module to it. The value of that class attribute contains the type of item in the Zotero library, and it takes the form of “img-book” or “img-journalArticle” or “img-conferencePaper“. There are two regular expressions defined: the first removes the “img-” prefix from each value and the second replaces all instances of an upper case letter with a space plus the lower case version of the letter. The latter rule turns “journalArticle” into “journal article” (note that there is a space in this field prior to the \L part). The result is assigned to the item.itemType element. This is used in the final Loop module to build a item.description element to create the string: <p>This journal article was saved to my Zotero library.</p>.

That’s all there is to it. Yahoo Pipes applies all of these modules to each of the items in the list retrieved from the Zotero library page and generates the corresponding RSS feed.

Earlier this week, the Zotero team released its 1.5 Beta version. Among the much anticipated features is the ability to synchronize citations among computers and with a display at the Zotero website. And if you set the profile permissions on the website, anyone can view your display of citations as well. As Dan Cohen, Director of the Center for History and New Media where Zotero is being created, points out, it is fun to see what others have added to their Zotero library. You might even want to watch what others have saved to their library. That is where this Zotero-to-RSS mashup comes in.

I’ve set up a Yahoo! Pipe that examines a Zotero user’s citation page and turns it into an RSS feed. There is one prerequisite — the user must have turned on the profile option on the Zotero website to “Publish Entire Library” (see figure 1). If you want to publish your own citations, you’ll need to set this by using the “Settings” link in the upper right corner of any Zotero.org page when you have signed in.

Figure 1. Zotero Privacy Settings

To use the Yahoo! Pipe, you will need two bits of information: the Zotero username and the Zotero user number. These bits of information are available in the URL of a library on the Zotero website. For instance, the URL to my Zotero page is::

The dltj is my Zotero Username and 683 is my Zotero User Number. Similarly, for Dan Cohen’s Zotero page, his Username is dancohen and his User Number is 2. With that information you can go to the Zotero-to-RSS Yahoo! Pipe (http://pipes.yahoo.com/peteratohiolink/zoterolibraryrss), and plug it into the form (see figure 2). Select the “Run Pipe” button and after a few seconds you’ll see the results.

Figure 2. Yahoo! Pipe Configuration

If you see an error “error fetching” warning with a “403 Forbidden” message, that means the user has not opened up their Zotero library for public viewing. But if they (or you!) have, then you can use the “Get as RSS” link as a way to track additions to the library. For instance, I’ve added my Zotero library feed as a “Blog” source to my FriendFeed profile.

George Mason University issued a statement this morning regarding the lawsuit filed against it by the Thomson Scientific division of Reuters. It looks like GMU intends to fight the lawsuit. It has restored the functionality that allowed Zotero to understand the EndNote® citation style file format and announced that it will not renew its site license for Zotero EndNote. [Bah -- got this wrong initially. Thanks for pointing this out, Ryan.]

The Thomson Reuters Corporation has sued the Commonwealth of Virginia over Zotero, a project based at George Mason University’s Center for History and New Media (CHNM). A free and open-source software initiative, Zotero aims to create the world’s best research tool and has already been adopted by hundreds of thousands of users at countless colleges and research universities. CHNM announces that it has re-released the full functionality of Zotero 1.5 Sync Review to its users and the open source community. As part of its formal response to this legal action, Mason will also not renew its site license for EndNote.

As academics themselves, the creators of the Zotero project strive to serve the scholarly community and to respond to its needs in an age of digital research. In line with that simple goal, they maintain that anything created by users of Zotero belongs to those users, and that it should be as easy as possible for Zotero users to move to and from the software as they wish, without friction. CHNM concurs with the journal Nature, which recently editorialized [link added] about this matter: “The virtues of interoperability and easy data-sharing among researchers are worth restating.”

CHNM remains committed to the openness it has promoted since its founding at Mason in 1994 and to the freedoms of users of its websites and software. Its ambitious development cycle and plans for Zotero’s future remain unchanged. CHNM will continue to develop and implement new research technologies in the pursuit of better ways to create and share scholarship. CHNM greatly appreciates the many supportive comments it has received from scholars, librarians, and administrators around the globe.

The statement was also posted in the official Zotero blog and copied into the personal blog of Dan Cohen, director of GMU’s Center for History and New Media. In addition, Zotero project co-director Sean Takats has a post describing some of the new features of the just-released update to the Zotero 1.5 Sync Preview development version.

Response from Thomson Reuters

The Scientific business of Thomson Reuters has initiated a law suit against George Mason University (GMU) because of violations of the terms and conditions of the EndNote® Desktop license agreement.

Thomson Reuters VP, Business Strategy and Development, Dave Kochalko said, “Simply put, we strongly believe that the creators of Zotero have reverse engineered our software code which enables EndNote’s bibliographic formatting capability. These format files only exist as software code; there is no content or information independent of lines of code and these files can only be interpreted by the computer. A key value of EndNote is its ability to format a bibliography within a manuscript and the format files are integral to that capability. We have talented employees who have invested many years in building this resource for the EndNote community.”

Kochalko added, “We are absolutely a proponent of interoperability and easy data sharing provided contracts are not breached and intellectual property is respected. We have worked diligently over the past several months to resolve this matter amicably. Since it has become clear that a resolution is not possible at this time, we have no choice but to pursue litigation in order to protect our intellectual property, as well as protect our bibliographic formatting capability, an important publishing resource our EndNote user-community has relied upon for many years.”

I think Thomson Reuters is really out on a limb here. File formats exist as bits of information; it is software code that interprets those bits of information. There is certainly a history of “black box deciphering of file formats. That is to say, examining the files themselves in the absence of the underlying source code to determine content and structure. There is precedent for this. All of that said, it looks like this conflict is going to wind up in the courts, and we might get a judgment on whether it was legal.

The text was modified to update a link from http://www.opendwg.org/node/86 to http://waybackmachine.org/20081029000000/http://www.opendwg.org/node/86 on January 28th, 2011.

This is a brief update on the EndNote/Zotero lawsuit. The story thus far: The Thomson Scientific division of Reuters, maintainer of the EndNote software, is suing George Mason University over the upcoming release of Zotero, the Firefox plugin for managing citations. More specifically, the complaint filed in Virginia state courts says “GMU reverse engineered or de-compiled the EndNote Software and the proprietary .ens files contained within the EndNote Software in order to determine how to convert the EndNote Software .ens style files into the open source Zotero .csl style files, in direct and material violation of the [sitewide] License Agreement [signed by GMU].” (Note, though, as others have pointed out, Zotero is not converting EndNote Style files to Citation Style Language (CSL) files.¹ It is parsing EndNote Style files and using them internally along side style definitions defined using CSL. Or, at least it was until the function was removed from the code.)

EndNode Styles Site now behind a Click-Through License

To download EndNote output styles, connection files, import filters, templates and the user manual you must agree to the following terms.

1. USE OF DOWNLOADED FILES

EndNote output styles, connection files, import filters, templates and the user manual are provided solely for use by licensed owners of EndNote and with the EndNote product.

2. RESTRICTIONS
End User may not modify, translate, decompile, reverse engineer, retransmit in any form or by any means (electronic, mechanical, photocopied, recorded or otherwise), resell or redistribute EndNote and its associated files, or any portion thereof, without the prior written consent of Thomson Reuters.

3. PROPRIETARY RIGHTS OF THOMSON REUTERS
End User agrees that Thomson Reuters is the sole and exclusive owner of all rights in the EndNote product. EndNote is protected by United States copyright and other laws, as well as international treaty provisions, and End User must treat EndNote as any other copyrighted and legally protected material. All rights in the EndNote product not expressly granted herein are reserved by Thomson Reuters.

This seems a little like closing the barn door after the horse has gone.² And it opens up all sorts of questions like: What about all of the EndNote Style files that are posted on other sites and all over the web? Do these sites now need to put their .ens files behind similar click-through licenses? Does the license apply to EndNote Style files that I create myself using the EndNote software?

“Gripes About the New Contract for Downloads”

I’m not very amused by the new covenants to download style files. I realize that a recent lawsuit against a competitor may have spooked people, but I don’t understand several issues:

Can these files be used by users of the EndNote trial as part of their evaluation process? The first term of the contract seems like it may prohibit that.

Why is the user manual not made available to everyone? It can be useful prior to the purchase of EndNote for informational evaluation. I do not see what value it has to your competitors, and I see only advantages for EndNote in increasing the distribution of the manual.

Why is the word “modify” in covenant 2? You do not ship every output styles, connection files, import filters, or template for all needs. Fortunately, EndNote has excellent editors to fill in these gaps. Can I no longer modify the components of EndNote for use form within EndNote, even though I have a license? If so, this seems to be a major loss of value!

Another common practice that is prohibited now is sharing a subset of files with other EndNote users. My previous research group had styles of journals we used and connection files for databases we used in a common location. This made it very easy to make a minimal installation of EndNote on new workstatons and also to grab missing files without slogging through the online repository. Many other groups and libraries seem to do something similar. Again, this practice would make EndNote more valuable.

More on What Zotero Does

Unfortunately, [the Reuters lawsuit] is also based on claims that appear to be factually false. Apparently, the Zotero team did create their own style format and is crowd-sourcing the creation of import styles. As you can see from this Zotero developer discussion thread, the developers considered and explicitly rejected supporting the redistribution of Thomson-supplied EndNote conversion files. In fact, while Zotero can read EndNote style files, it specifically does not convert them into Zotero’s own format, in large part to discourage the redistribution (deliberately or accidentally) of Thomson-created files. What the import feature does facilitate is (a) users who have already licensed EndNote and want to migrate to Zotero can use the EndNote styles that they have already paid for, and (b) Zotero users can take advantage of the EndNote import styles that individual journal publishers (as opposed to Thomson itself) make available for the convenience of their subscribers. These uses strike me as totally within bounds.

One minor quibble is that I don’t think Zotero created its own style format. Citation Style Language, according to the description on its homepage, was “developed alongside CiteProc.” Other than that, I highly recommend reading his thoughtful analysis.

Extension Granted

The text was modified to update a link from //forums.thomsonscientific.com/ts/board/message?board.id=en-general&thread.id=649 to http://forums.thomsonscientific.com/ts/board/message?board.id=en-general&thread.id=649 on January 13th, 2011.

Footnotes

1. Citation Style Language is an “XML language to describe bibliographic and citation formatting.” It is used by Zotero and other citation management tools.
2. An explanation of the cliché Why close the barn door after the horse is gone?

Via a posting by James Grimmelmann, I found a link to the text of the complaint filed by Thomson Reuters against George Mason University in Virginia’s state courts. The critical bits (to an untrained eye) are included below:

Paragraph 12: George Mason University [GMU] entered into a site license agreement, dated December 1, 2003 with ISI (which was subsequently renewed with Thomson) to access to use the EndNote Software (the “License Agreement”)…

Paragraph 14: …Section 3(c) of the License Agreement requires an express acknowledgement by GMU that the “Licensee recognizes the importance of the Company maintaining its proprietary rights over the [EndNote] Software and of avoiding improper use of the [EndNote] Software as defined in this Agreement. Licensee shall use best efforts to ensure that Users comply with the forgoing limitations on usage.” [Emphasis added in court document]

Paragraph 15: Specifically, Section 3(d) of the License Agreement expressly prohibits the following activities in connection with the use of the EndNote Software:

• duplication of the [EndNote] Software except as specified herein;
• separation of components ([EndNote] Software is licensed as a single product. Its component parts may not be separated for use on more than one computer);
• distribution or transfer of the [EndNote] Software in any form (printed, electronically relayed, posted to public list services or bulletin boards, or magnetically stored); or
• reverse engineering, de-compiling, translation, modification, distribution, broadcasting, dissemination, or creation of derivative works from the [EndNote] Software. [Emphasis added in court document]

Paragraph 16: In addition, the License Agreement limits GMU’s use of Thomson’s trademarks and trade names solely to uses “connected with the identification of the [EndNote] Software.”

Paragraph 21: On information and belief, GMU reverse engineered or de-compiled the EndNote Software and the proprietary .ens files contained within the EndNote Software in order to determine how to convert the EndNote Software .ens style files into the open source Zotero .csl style files, in direct and material violation of the License Agreement.

Paragraph 22: GMU is distributing the new beta version of Zotero, which allows for the unauthorized, free conversion of the EndNote Software .ens style files, in direct violation of the prohibited uses of the EndNote Software, as specified in Section 3(d) of the License Agreement.

Paragraph 23: GMU is willfully and intentionally destroying Thomson’s customer base for the EndNote Software, in direct violation of the License Agreement, by allowing and encouraging users of Zotero to freely convert the EndNote Software’s proprietary .ens style files into open source Zotero .csl style files and further distributing such converted files to others.

Paragraph 24: By letter dated August 4, 2008, … counsel for Thomson notified Dr. Alan. G. Merten, President of GMU, that GMU’s actions are a direct violation of the terms and conditions of the License Agreement, and requested that GMU immediately take all necessary steps to cure the unauthorized used of the EndNote Software through Zotero (the “Cease and Desist Letter”).

Paragraph 25: The Cease and Desist Letter demanded that GMU immediately terminate Zotero’s unauthorized use and conversion of EndNote Software’s proprietary .ens style files and revise the beta version of Zotero to prohibit future conversion of EndNote Software’s proprietary .ens style files.

Paragraph 26: Furthermore, the Cease and Desist Letter demanded that the beta version of Zotero be made unavailable until such revisions were made by GMU.

Paragraph 27: The Cease and Desist Letter set forth a deadline of August 15, 2008 for GMU to comply with Thomson’s demands.

Paragraph 28: On or about August 11, 2008, counsel for Thomson received a telephone call from David Drummey, in-house counsel at GMU, (“Mr. Drummey”) asking to extend the August 15, 2008 deadline to August 25, 2008, in order for GMU to assess the matter and respond to the Cease and Desist Letter. Thomson’s counsel granted this request to GMU.

Paragraph 29: On August 25, 2008, Mr. Drummey spoke via telephone with counsel for Thomson and stated that GMU was seeking a collaborative arrangement between Thomson and GMU as a means of resolving the conflict between the parties.

Paragraph 30: By e-mail to Mr. Drummey dated August 25, 2008, … counsel for Thomson granted an extension of the response deadline until August 20, 2008, insisting, however, that, prior to any collaboration discussion between the parties, GMU must first comply with the demands set forth in the Cease and Desist Letter as a result of the material breach of its obligations set forth in the License Agreement.

Paragraph 31: On August 28, 2008, Mr. Drummey contacted counsel for Thomson asking for yet another extension of the August 29, 2008 deadline. When told that Thomson would provide no further extensions, Mr. Drummey informed counsel to Thomson that GMU had no intention of curing its breach of the terms and conditions of the License Agreement and had referred the matter to outside counsel.

Paragraph 37. Moreover, GMU has breached its obligations under Section 7 of the License Agreement with respect to the use of Thomson’s registered trademark EndNote®, by willfully and inappropriately using the trademark to direct and induce users of Zotero to covert the EndNote Software’s proprietary .ens style files to Zotero’s style files.

Paragraph 40. As a direct and proximate result of the material breach of the License Agreement by GMU, and due to the freely distributable, open-source nature of the converted Zotero .csl style files, Thomson has suffered irreparable harm for which the law will afford it no adequate remedy.

Paragraph 41. As a direct and proximate result of the material breach of the License Agreement by GMU, Thomson has suffered monetary damages in an amount to be determined at trial, but which, due to the open-source nature of the Zotero .csl style files, at a minimum are equal to about ten million dollars ($10,000,000) annually until such time as GMU is permanently enjoined.

In addition to James’ excellent comments on the matter, I’d note that it doesn’t look like the Thomson legal team actually had anyone look at the Zotero code. The complaint alleges that “users of Zotero [are freely converting] the EndNote Software’s proprietary .ens style files into open source Zotero .csl style files and further distributing such converted files to others.” That isn’t happening. Zotero is reading the .ens style files into internal data structures in the browser, but it is not converting .ens style files to .csl style files and storing them on disk. Nor is Zotero or anyone associated with Zotero redistributing .ens style files. Users of Zotero would need to download .ens files on their own and add them to Zotero’s configuration.

As an updated to my previous posting on the Thomson-Reuters/Zotero lawsuit, I also noted change to the EndNote website. Richard Wallis notes:

Following that link to Endnote export styles [link] you end up on a page containing the following words: “EndNote output styles are provided solely for use by licensed owners of EndNote and with the EndNote product.”

What’s interesting is the fact that this sentence was recently (within the last 8 months) added to the site. The January 13th snapshot of that page in the Wayback Machine doesn’t have that sentence. I wonder when it was added. In both cases (the current page and the Wayback Machine snapshot), the note at the bottom says “This page was last modified on: November 4, 2005″ — I would disagree.

From paragraph 31 of the complaint, it looks like GMU is going to fight the lawsuit. (I’m assuming this is the intention behind GMU engaging external counsel.) A comment by Dan Cohen (director of the Center for History and New Media at GMU) in a posting on the Zotero forums says: “Thanks to everyone for your words of generous support. We will likely issue a statement in the near future.” Let’s keep watching for more information…

Thomson Reuters is suing George Mason University to stop distribution of the newest version of Zotero, a Firefox browser plugin for managing citation data. According to a story from Courthouse News Service, Reuters is claiming that George Mason is violating the terms of its license agreement by including a function in Zotero that will convert citation styles from the proprietary EndNote format to a format that can be used by Zotero. Reuters also asking for $10 million in damages for destroying the EndNote customer base. Since George Mason is a state institution, the Commonwealth of Virginia is also named in the suit.

According to the Courthouse News article, Reuters says: “A significant and highly touted feature of the new beta version of Zotero, however, is its ability to convert – in direct violation of the License Agreement – Thomson’s 3,500 plus proprietary .ens style files within the EndNote Software into free, open source, easily distributable Zotero .csl files.” In fact, there is a closed enhancement ticket in the Zotero project management tool for an EndNote to Zotero style converter. The EndNote license agreement says

RESTRICTIONS. End User may not modify, translate, decompile, reverse engineer, retransmit in any form or by any means (electronic, mechanical, photocopied, recorded or otherwise), resell or redistribute the Product, or any portion thereof, without the prior written consent of ResearchSoft. Except as expressly set forth in this Agreement, End User may not make any use of the Product.

One suspects the heart of the lawsuit claim is that the EndNote .ens style files are a portion of the EndNote product and therefore cannot be reverse engineered, which it appears the Zotero team has done.

The suit was filed on September 5th in the civil division of Richmond City Circuit Court, case number CL08004225-00. There is not yet any significant information online, but if you are interested in the lawsuit you may want to keep an eye on the Virginia Circuit Court Case Information website. (Because of the way the site is built, one cannot link directly to the case. Select “Richmond City Circuit” from the pull-down list, select “Begin”, then at the next screen select “Civil” and plug in the case number listed above.) No public word from Zotero (nothing in the project’s blog or the forums) or Thomson Reuters (on the EndNote news or the Thomson Scientific news pages) on the lawsuit.

This could be an interesting lawsuit to follow. The Zotero enhancement ticket has a discussion about the legality of using the EndNote Styles collection or EndNote styles posted on other sites. The EndNote software itself makes it possible to create and export the EndNote Style .ens files, so many EndNote Style files are the result of the work of entities other than Thomson Reuters. Presumably the person that created the ENS-to-CSL translator for Zotero did not have a copy of the EndNote software and so was not bound by the EndNote license agreement. Does the fact that the ENS files were published on open websites where they could be accessed without using EndNote itself make a difference?

Zotero is a project of the Center for History and New Media at George Mason and is sponsored/funded by the Institute for Museum and Library Services, the Mellon Foundation, and the Sloan Foundation.

Found via Mark Leggott’s and David Bigwood’s blogs.

Updates

Following that link to Endnote export styles [link] you end up on a page containing the following words: “EndNote output styles are provided solely for use by licensed owners of EndNote and with the EndNote product.”

What’s interesting is the fact that this sentence was recently (within the last 8 months) added to the site. The January 13th January 3rd snapshot of that page in the Wayback Machine doesn’t have that sentence. I wonder when it was added. In both cases (the current page and the Wayback Machine snapshot), the note at the bottom says “This page was last modified on: November 4, 2005″ — I would disagree.

Also — I just found a copy of the Thomson Reuters’ complaint [PDF] on the Courthouse News Service website.

31-Oct-2008 As Rick notes in the comments, the January 13th version on the Wayback Machine of the EndNote Styles page is no longer there. It has been replaced with a link to the January 3rd version, and you can periodically check the list of all versions of the page in available in the Wayback Machine. Odd — I didn’t think pages were supposed to go away on the Internet Archive.

The text was modified to remove a link to http://thomsonreuters.com/press_room/press_releases_archive/?monthToDisplay=8&yearToDisplay=2008&busi_unit=scientific on January 28th, 2011.