First of all I would like to say that I enjoyed reading the article, it is very well written and summarizes in a very precise form the writers point of view about the problem.

Nevertheless it starts with a very interesting point that is not further followed. The main question is, Do you compete at all?. Or stated in a more exact form, do you need to compete at all?. In those 2 questions I want to imply that maybe, google is no competition for you.

Lets have a look at the Google and Amazon search engines example from another point of view. The user.

What the user wants / needs is a solution for his problem, find lion king cupcakes. He doesn’t really think about metadata at all. He just thinks about his problem and as such goes to massive text search engine like google. There, google, excelling at his core business directly redirects the user to the best place available to satisfy the user perceived need. Amazon.

No competition among Google and Amazon as such, and I don’t think it is in the mind of neither of them to compete. Google is certainly not competing with Amazon since it is the first reference to happen. And I don’t really think Amazon thinks Google is competing with him since Google is driving clients right to its grasp.

Lets get back to the user. In his mind there are two processes, the first process is oriented towards locating something that satisfies his need. The second process is directed towards obtaining that very same thing in the (easiest?, cheapest?, most secure?, just choose your adjective) way. There we see a collaboration in locating the subject and after that Google disappears and Amazon is all alone in finally satisfying the client. Is Amazon doesn’t satisfy the client both Google and Amazon lose. On the other hand if the client leaves Amazon as a happy buyer both win.

So the question is, Why compete at all?. Google is not an specialized, categorizing, need oriented catalog. It is just a very smart search engine. In mi opinion it is so good at doing what it does because it presents the most simple interface ever shown. A simple textbox and a button. You cant simplify more without showing a blank page. The message is clear to the user:

“hey just tell me what you need in three or four words, ill do the first part of the road for you and will set you in the portal of the best specialist available”

Thinking in a Google interface with combos and such simply breaks the Google concept (have any of you ever used the advanced search interface?). But you can’t have specialized advice without those. In it is there where there is plenty of space to breathe and where I see lots of opportunities for cooperative-competing.

I see your point. It is true that user experience is what finally draws the line between a successful and commonly used application and a well thought application that is simply not used.

As far as my experience with library catalog search applications is concerned I have always found the interface hard to use and difficult to understand (IBM370 terminals and such). Maybe because it was mainly directed towards making the most of the metadata accessible through search. But as an user I always had the problem to first understand how to map the concepts in my mind to the metadata used to map de concepts by the application and second to understand the search results returned by the application. Luckily the librarian was always near to lend a helping hand.

To be frank that’s the main problem I see with metadata. It represents the world as the cataloguer understands it using his own knowledge base. But a different user may find it much harder to locate that very same book using concepts that are alien to him. So maybe a lot of metadata gets to a point where it is too much metadata and simply adds to the noise making it more difficult to retrieve the desired result from a search query.

Maybe that’s a strong point with Google. It has so many cataloguers available (web authors) that for every concept expressed by an user using words or word combinations it is able to locate several HTML pages where an author has expressed that very same concepts using similar words. It will provide also pages that are not closely related to the concepts the user had in mind. But attempting a best effort search over so many data (not metadata) is able to locate more results than the ones that would be possible to obtain through an exact term metadata search.

So maybe a good question is, when do a lot of metadata becomes too much metadata?

First in answer to your question I’m neither a librarian nor a library professional. I come from the other side of the problem, the technical one. My expertise has always been directed towards analysis, design and development of metadata driven applications such as document and record management systems.

Right now for example, I’m in the middle of a project involving fedora (Digital Object Repository) for the development of a records management system mainly oriented towards information preservation. Understanding by preservation that the content of the document must be accessible and usable in a far away future where technological innovation and evolution might render the previous format or system unusable. As such, we must not only provide the means so that the document format is updated accordingly to technical evolution but also so that the information provided to catalogue and locate the document (metadata) is evolved so that we also preserve the ability to locate that very same document.

A good example would be converting all the pages contained in the library of congress to tiff and store them all away in a single HDD with a sequential name file contained in a single folder. We preserve the information as such, but nobody is going to be able to locate anything.

As such, in this kind of application, metadata collection, cataloguing and evolution is the key to a successful application.

I located your page while performing a state of the art search and got a very pleasant surprise in finding that not only there is someone on the other side of the problem worrying about these things such as metadata and its technical implications, but also they know very well what they are talking about. No offense meant by the previous comment but it is rare to find non-technical people really worried about the technical problems related to their area of expertise as it is also very difficult to find technical people worrying about the real problem and not on how to solve it. Maybe that’s why on most of the cases we come up with a very good solution for a problem that nobody has.

Returning to the Google versus Amazon problem and your point of data contained in the document versus metadata about the document I would like to further follow your example search.

In order to do so I have written “lion king cupcakes” on the Google search interface and have located Amazon on the seventh position of the search result. Then I have changed the search to “Lion King cupboards” and have been amazed to find Amazon on the first and second position of the search. What’s the difference?

The main difference is that the product located under the “lion king cupboard” search had 5 user reviews while the “lion king cupcakes” had none. Not all the people that reviewed the first product had the same point of view. In fact what makes Amazon the first option to be located is that it contains 5 different points of view that depict the product as these 5 users see it. And two of them see the product fit for a cupboard.

To summarize it is not the metadata that Amazon knows about the product what drives me directly to it. It is the comment of a user in a Web 2.0 way that sees the product closely to the way I see it and that uses the very same words I have used in my search. It is also true that once I have located the product it is the metadata I know about the product what really makes it useful.

Maybe that’s the point behind the faceted metadata you mentioned earlier. The more facets your metadata has, the closest to a final user community it is and its usefulness in a search increases.

Another point is that maybe the process is divided in two parts. The first part is locating the object, the second part is making it usable. Metadata is the key to second part, but in searching and locating the object maybe too much metadata simply gets in the way.

[quote post="94"]Still, one has to wonder if at some point the raw keyword index across the entire web content is going to break down at some point.[/quote]

My personal opinion is that it will break at some point in the future. A good example of that is that our conversation now appears the fifth at Google while trying to locate pages using the “lion king cupcakes” term. And I would be very hard pressed to believe that for someone using that terms to locate a page, our conversation is useful in any way.

[quote post="94"]I think it would be useful to the end user to know whether ‘cupboard’ appeared in the “abstract” of the item or in a comment about the item. Well, not directly useful to the user, but the usefulness could be brought out in search results weighting, visual cues in the results listing, and post-search options (e.g. a checkbox to eliminate the occurrence of the word ‘cupboard’ in all item comments[/quote]

You have a good point there. Not all the words used in a textual object description have the same location weight. Google already knows that, but the only thing it is able to do is to assign weight depending on the page position, surrounding words and page references.

The real question behind is whether it can be done better without dramatically increasing the costs. And whether that cost increase has an adequate return in user perception.

Imagine that while writing our opinions we had written them all surrounded with appropriated metadata such as <abstract>… <academicExample>Lion King Cupcakes</academicExample> … </abstract> or to use a better example lets imagine a book content indexing where we place metadata while indexing the content such as:

And then <mainCharacter>Jhon</mainCharacter>, <secondaryCharacter>Mary </secondaryCharacter> traveled to <location>New York</location>.

On the first example in case Google had inferred that our intent is to buy something it could have stripped our conversation based on the fact that there those words where simply used as an example.

On the second example trying to locate a concrete book where the words Jhon and New York appear could be a real nightmare using a simple search and indexing engine. But maybe a book where Jhon is the main character is easier.

And so we get to what I think is what you implied on your article. We have two totally different technologies. The first is text indexing. This technology ensures the best probability to locate a page related to the terms used while searching, provided that we have a rich and varied description of the subject. But it is the job of the user to get something useful out of it.

On the other hand we have rich metadata cataloguing and truth inferencing engines where we can search in so many different ways that we can provide infinite customization to the query so that it reflects the exact user intent and locate only what the user wants / needs.
On the first case the job is performed by the user after the query, on the second case the job is performed before the query.

Which one is better?. Well the only thing I can say for sure is that the second is more expensive. My personal opinion is somewhere near yours in that a wise combination of both technologies might very well be the answer.

[quote post="94"]To take a page from Clayton Christensen’s theory of disruptive innovations: is automated description of textual content good enough for some less-demanding users? The answer I think is yes. Is it good enough for high-demanding users as compared to human-driven description? No. Will it ever be? I think the answer here, too, is “yes.” [/quote]

Very good point. The only thing I would add is that my impression as a technologists is that “will it ever be?, yes”, and very soon.

We have the tools, we have the knowledge and we have the chance.

Starting with projects such as Wordnet, or Eurowordnet.
Text indexing initiatives such as google. Metadata cataloguing and triple stores search engines. And a lot of other technologies.

And the most important tool of all, Web 2.0 interfaces that help to bring the user into the application my impression is that we are in the verge of a huge change.

My impression is that up to now internet has been considered only as a huge automatic vending machine where I can simply place an order and get a result. It is curious but with all the technology available I can hardly see any portal where all the things they do cant be traced back to the automatic vending machine example.

Lets examine two distinct and extreme situations that I think reflect the extent of the problem.

Would you ever consider closing the library and placing instead an automatic book dispenser machine such as most of the video stores do?. I think not.

Would you, in the other hand, close the retail shop and place all your employees in a portal where they can replicate the user retail shop experience through web technologies?. To the best of my knowledge “It has not been done”.

Let me further follow the last example so that it might be fully understood. In this way imagine all your employees working at a contact center and available to answer Internet Telephony, Video Conferences, Online Chats, email requests and any other way of internal communication and cooperative navigation. In this way, when a “would be customer” enters into the portal (retail shop) he might look around a bit (search and navigation) but up into the portal he suddenly can read a message such as “welcome, can I help you?”, or he might see that there are 4 people ready to help him and upon selection a chat window, or audio / video conference is started as in a normal “real life” buying experience.

In the middle of those two extreme initiatives comes this kind of technology we are talking about. What we are always trying to do is to emulate the user face to face real life buying experience through technology.

In real life the customer comes into the shop. The librarian is able to see how the user is dressed, what books is he looking at. He is able to speak with him and in case he is a very good sales man to drive the user towards the thing he needs, using non-structured language. Finally what drives the difference between a good seller and a bad one is that he can know faster what the user wants through both visual and conversational skills. To summarize, he is able to “bring the customer into the deal”.

I think this notion of “bringing the customer in” is what drives web 2.0 initiatives as such. And is the real breakthrough in my opinion.

Can Google be Out-Googled? http://bit.ly/bnnVEx