We did have a huge amount of traffic from something that was significantly degrading services recently.
Because of this I:
- Implemented global rate limiters for web page requests
- Integreated with AbuseDB - if a rate limiter is triggered we cross reference the IP here and auto ban the IP if it has a high confidence of abuse. Over 6,500 IP's have been banned to date, ranging from spammers, anonymous bots of security scanners. Just nasty traffic types you don't want to serve.
- Spent time on bot identification, getting the DNS host addresses from the IP's - most bots we allow, others we don't if they are too aggressive and we don't want them crawling (for example some third party business intelligence bots).
It's all settled down a lot now and we are not seeing much evidence at all of false positives. Some of the bots we banned were doing multiple page views per second - we log the blocked accesses now and some of the bots have over 3,000,000 blocked page view requests.
Anyway, was a fun and interesting project to work on and I'm pleased everything is running a lot more smoothly now.
RE ai bots specifically, we generally are OK with them. Interestingly we do get new customers who tell us they asked ChatGPT what a good game engine is and it recommends us - so we don't want to impact that negatively. The AI bots do consume a fair number of resources from our POV but for now it appears to be beneficial for us.