Build your own search engine

In a move that could potentially huge impact on the search market, Alexa, the Amazon owned company that is most famous for its website popularity toolbar, has opened up its search index and web crawler to the world on the Amazon Web Services Platform.

Alexa says that its Web Search Platform, currently in beta, is the first time that the heavyweight search engine crawling technology used by the likes of Yahoo and Google has been made available to developers and small companies.

The sheer cost of maintaining an index and crawling technology has been a barrier to many companies with search ambitions. Now, with Alexa making the vast processing power of its servers available to everyone, small companies and enthusiasts have the potential to mine Alexa’s vast database and use the web crawling technology to set up search services of their own.

John Batelle, who first broke the story at his Searchblog, puts it well when he says:

Alexa and Amazon are turning the index inside out, and offering it as a web service that anyone can mashup to their hearts content. Entrepreneurs can use Alexa's crawl, Alexa's processors, Alexa's server farm....the whole nine yards.

The figures involved certainly seem impressive. Alexa says it spiders 4 billion to 5 billion pages a month, archives 1 terabyte of data a day, with each data store holding up to 100 terabytes of information - although how these figures compare with a Google and Yahoo is difficult to know.

There are no licensing costs just modest fees for processing, bandwidth and storage.

Alexa says sites such as the experimental musipedia, where you can search for tunes and melodies by whistling or singing (I won’t be trying this, much to the relief of my colleagues), are the kind of innovative new search sites that will be possible.

This could be a very smart move as Alexa had effectively lost the search engine game against the big three of Google, Yahoo and Microsoft, so why not change the rules of the game and give yourself a headstart.

At this early stage there are, of course, lots of if and buts but democratising search in this way could lead to some innovative search sites and Alexa’s move has been favourably received in the blogosphere.

Whether Alexa can upset the search apple cart remains to be seen but if it lessens the possibility of a Googlopoly, where we rely on the next offering from Google Labs to expand our search frontier, it can only be a good thing.

You can read more about Alexa's Web Search Platform here.