In theory, sure. But that implies the people doing this care to be stealthy, or put in that much effort. Web scrapers, aggregators and prepackaged tools set up on quickly deployed Linux VMs or containers don’t really care all that much until they do. So if you don’t go through the trouble to spoof the user agent, it doesn’t change. Even now, it’s not necessarily that those user agents are blocked, they just aren’t counted as real users anymore
Comments
In theory, sure. But that implies the people doing this care to be stealthy, or put in that much effort. Web scrapers, aggregators and prepackaged tools set up on quickly deployed Linux VMs or containers don’t really care all that much until they do. So if you don’t go through the trouble to spoof the user agent, it doesn’t change. Even now, it’s not necessarily that those user agents are blocked, they just aren’t counted as real users anymore
Sign in to comment. Sign in