hgweb.cgi and robots.txt

Wed Oct 19 22:02:56 UTC 2016

Hi there. I host several public mercurial repositories using hgweb.cgi
and Apache.

Recently I am seeing bots hitting my web server hard, hgweb.cgi sucking
CPU and harddisk read bandwidth.

Is anybody seeing something similar?.

Does anybody use a "robots.txt" to control bots?.

How can "robots.txt" be served by hgweb.cgi?

-- 
JesÃºs Cea AviÃ³n                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://www.mercurial-scm.org/pipermail/mercurial/attachments/20161020/203d8e52/attachment.sig>