hgweb.cgi and robots.txt

[Simon King]

On Wed, Oct 19, 2016 at 11:02 PM, Jesus Cea <jcea at jcea.es> wrote:
> Hi there. I host several public mercurial repositories using hgweb.cgi
> and Apache.
>
> Recently I am seeing bots hitting my web server hard, hgweb.cgi sucking
> CPU and harddisk read bandwidth.
>
> Is anybody seeing something similar?.
>
> Does anybody use a "robots.txt" to control bots?.
>
> How can "robots.txt" be served by hgweb.cgi?

I'm not sure you can easily get hgweb.cgi to serve robots.txt, but
(assuming you are currently directing the root of your site at
hgweb.cgi), you can configure Apache to handle the "/robots.txt" path
specially. I think an Alias rule ought to work, something like:

Alias "/robots.txt" "/path/to/robots.txt"

https://httpd.apache.org/docs/current/mod/mod_alias.html#alias

Hope that helps,

Simon