Programming Pauls of Wisdom

"Googling for someone else's Truth"

I got into true "disgusted of tunbridge wells" mode last week and wrote the following to guardian newspapers comments section. It has not been published so I put it here in case anyoone wants to read it.

In reply to your article "http://www.guardian.co.uk/international/story/0,,1509281,00.html#article_continue" Googling the Truth, I am afraid nothing can be further from the truth. I am coming across more exmaples of Google not simply being wring, but being directedly wrong - lying on someone else's behalf. The worst example I came across was last week
Using Google to search for the "truth" on Martin Luther King finds a Neo_nazi propaganda site in the top 3 results.
Type in "martin luther king" into google.co.uk. Third down in the results page is a site called "Martin Luther King Jr. - A True Historical Examination". I will not repeat its URL for reasons that will become clear, yet a quick look at the home page shows nothing much amiss, there are even flyers to download for your school! It is only a second reading that reveals this is not an educational or even balanced site. Hosted by stormfront ("White pride World Wide") it is part of a technically astute but viciously racist attempt to ... well you can guess. I can feel my knee-jerking reatively already. Protect those children doing school essays.
I am obviously not claiming Google has Aryan purity as a hidden agenda far from it, but Google, like all other search engines, relies on the creators of web pages to provide it with enough information to categorise and index the vast number of web pages out there. It works a bit like this; if the content of my site contains text the same or similar to the search terms and if enough web sites publish links from their sites to mine, my site will be deemed more popular and relevant and so rise up the rankings for a given search term. The exact method used by the different search engines is a commercial secret, but all now seem to use a variation on this simple theme. Unlike the previous generation of seach engines, which primarily relied on information on a web page to classify it, Google was supposed to have ended the late 90's free for all where it seemed stuffing a hundred repetitions of a keyword into meta-tags could get you the top spot on yahoo, but it seems they have just made the fooling of a search engine harder, because you need to fool other people into linking to you. But that means for me to trust those search results, I implicitly trust those people who linked.
Ultimately we humans are very hard put to trust strangers. Which is why we tend to have a herd mentatilty - if 100,000 people trust it it must be good. And so in the main search engines work well - they aggregate the opinions of thousands of people and present them to us in a nice list. But when that breaks down there is no way of distinguishing between the opinions of thousands of people like me and thousands of rabid neo-nazis. And that is the essential problem of search engines - I have no way of checking who has told Google that this web page contains the Truth, and no way of deciding between competing claims of Truth. Google hopes that measuring organisations size, news gathering staff and so on may help, which in the main will do so, but if it is Truth to be measured, how do you compare the blogs of a young Iraqi in Baghdad to the Fox TV reporters embedded with the Us Marines? Certainly not by size of news staff.
Tim O'Reilly has called this aggregating of others' links the new open source, but it is not quite - there is a vital feedback loop missing.
Perhaps a better solution might lie in the ability for me to place a link to martinlutherking dot org on the Guardians' web site, but mark that link as "I think this is an racist site and I do not trust it". At the moment HTML does not let me do this, but ever-resourceful Tim Berners-Lee and w3.org are working on a semantic web to do just that (http://www.w3.org/DesignIssues/Toolbox.html#Assertion). The semantic web exists and works now, and allows me to trust opinions of friends of friends (of friends...) so a future google could distinguish between people like me and people like Jean Marie Le Pen, amoungst other things. It just might work. But for now I will have to rely on reading carefully, analysing critically and taking the time to make my own opinions. And if research for a school essay project can teach that to children, maybe my knee will need less exercise.

posted by paul @ 2:39 AM