HELP: Fighting AI scrapers

[Jesus Cea]

Hi there.

Since AI scrapers are not obeing "robots.txt" and they pretend to be a 
regular browsers, they are coming from thousands of IPs at a time and 
they do costly mercurial requests hammering my servers... I wonder what 
to do about it.

What are you doing?

I am aware of Anubis, but it doesn't integrate well with my 
infrastructure. I wonder if somebody out there has a WSGI middleware 
mitigation or if there is some interest in having one...

What do you do to battle this, guys?

Thanks.

-- 
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - https://www.jcea.es/    _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mercurial-scm.org/pipermail/mercurial/attachments/20250918/819e0355/attachment.sig>