Study shows Google favoured over other search engines by webmasters

Web site policy makers who use robots.txt files as gatekeepers to specify what is open and what is off limits to Web crawlers have a bias that favours Google over other search engines, say Penn State researchers whose study of more than 7,500 Web sites revealed Google’s advantage.

That finding was surprising, said C. Lee Giles, the David Reese Professor of Information Sciences and Technology who led the research team which developed a new search engine "BotSeer" for the study.

“We expected that robots.txt files would treat all search engines equally or maybe disfavour certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favoured and the search engines’ market share,” said Giles of Penn State’s College of Information Sciences and Technology (IST).

Robots.txt files are not an official standard, but by informal agreement, they regulate Web crawlers – also known as “spiders” and “bots” – which mine the Web 24/7 for everything from the latest news to e-mail addresses.

Web policy makers use the files found in a Web site’s directory to restrict crawler access to non-public information.

Robots.txt files also are used to reduce server load which can result in denial of service and shut down Web sites.

But some Web policy makers and administrators are writing robots.txt files which are not uniformly blocking access.

Instead, those robots.txt files give access to Google, Yahoo and MSN while restricting other search engines, the researchers learned.