If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.
With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.
Comments
If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
image
What sorts of measure you have taken to prevent bots from exhausting your compute and bandwidth
As soon as an IP is determined to be malicious, subsequent pageloads result in a single query to match the IP and they get dumped to a 403 static error.
With the bots that hammer a site looking for the top 1,000 or so exploitable URLs or partake in speed scraping, this has worked wonders for my server load. I catch them in their 1st-5th page load and the rest just result in basically no effort on the part of the script.
image
Sign in to comment. Sign in