[ixpmanager] DoS Attack of IXP Manager Looking Glass

[Rob Lister]

Thanks Barry,

I've added to robots.txt:

User-agent: *
Disallow: /index.php/lg/

/lg/ was already there.

So if this script respects robots.txt or not remains to be seen. There
are few automated queries running against it, but not to the extent that
it hammers the server.

 From what we can tell, it seems to be someone coming in from a ToR exit 
node or VPN
endpoint. It comes from within the same member network and just runs a 
ton of queries
to download all the prefixes for the same ASN. I've no objection to 
reasonable usage
- the requests themselves seem small, but doing a lot of queries with 10
simultaneous connections causing issues.

Seems to run every few weeks, we block it, then they come back from a 
different IP.
We've contacted the member concerned to see if they can find out what's 
going on.

The birdseye API itself is not externally visible, but only what's 
accessible from
IXP Manager portal.

If it happens again then we'll look into limiting the number of 
simultaneous
connections from each IP with apache for this URL.

Rob


On 2021-01-19 09:03, Barry O'Donovan wrote:
> Rob Lister wrote on 18/01/2021 16:21:
>> In recent months we've had a few instances of people really hammering
>>  our RS Looking glass, seemingly to enumerate large numbers of
>> prefixes.
> 
> See Marco's suggestion re robots.txt.
> 
>> Whilst we are happy for the lg data to be available, this seems to be
>>  caused by someone walking the entire lg for prefixes for a
>> particular ASN, making hundreds of connections in parallel, maybe
>> 7-10 requests per second, > 3000 requests in a 5 minute period before
>> our monitoring alarms.
> 
> This feels like a crawler.


-- 
Rob Lister
rob at lonap.net
+44 20 3137 8330