hgweb.cgi and robots.txt

[Jesus Cea]

On 20/10/16 09:24, Simon King wrote:
> On Wed, Oct 19, 2016 at 11:02 PM, Jesus Cea <jcea at jcea.es> wrote:
>> Hi there. I host several public mercurial repositories using hgweb.cgi
>> and Apache.
>>
>> Recently I am seeing bots hitting my web server hard, hgweb.cgi sucking
>> CPU and harddisk read bandwidth.
>>
>> Is anybody seeing something similar?.
>>
>> Does anybody use a "robots.txt" to control bots?.
>>
>> How can "robots.txt" be served by hgweb.cgi?
> 
> I'm not sure you can easily get hgweb.cgi to serve robots.txt, but
> (assuming you are currently directing the root of your site at
> hgweb.cgi), you can configure Apache to handle the "/robots.txt" path
> specially. I think an Alias rule ought to work, something like:
> 
> Alias "/robots.txt" "/path/to/robots.txt"
> 
> https://httpd.apache.org/docs/current/mod/mod_alias.html#alias
> 
> Hope that helps,

My configuration is more complicated because I am using "mod_rewrite".
My final configuration is this. I paste it just in case somebody else
found this useful:

"""
RewriteRule     ^/robots.txt$   /datos/sources/hg/robots/robots.txt
[Last]
RewriteRule     ^/$             /datos/sources/hg/cgi-bin/hgweb.cgi
[Last,T=application/x-httpd-cgi]
RewriteRule     ^/(.*)$         /datos/sources/hg/cgi-bin/hgweb.cgi/$1
[Last,T=application/x-httpd-cgi]

<Directory "datos/sources/hg/robots">
  Require all granted
</Directory>

<Directory "/datos/sources/hg/cgi-bin">
    Options +ExecCGI
    AllowOverride None

    AuthType Digest
    AuthName "JCEA Mercurial repositories"
    AuthUserFile /datos/sources/hg/usersNO
    <RequireAll>
      Require all granted
      <LimitExcept GET HEAD>
        Require valid-user
      </LimitExcept>
    </RequireAll>
</Directory>
"""

Still, I think that "hgweb.cgi" should be able to serve a customized
"robots.txt" :-).

-- 
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://www.mercurial-scm.org/pipermail/mercurial/attachments/20170209/a5b43444/attachment.sig>