[issue75] googlebot

[Thomas Arendsen Hein]

New submission from Thomas Arendsen Hein <thomas at intevation.de>:

70% of the traffic to the webserver which runs our public hg repositories is due
to requests from googlebot.com. A robots.txt disallowing access completely is
easy, but there should be a way to calm down search engines using http headers
like last-modified or expires.

----------
messages: 210
nosy: ThomasAH
priority: wish
status: unread
title: googlebot
topic: hgweb

____________________________________________________
Mercurial issue tracker <mercurial-bugs at selenic.com>
<http://www.selenic.com/mercurial/bts/issue75>
____________________________________________________