Wouldn’t most bots want to self-report as the most ‘ordinary user’ they could? Which probably means reporting as Chrome on Windows. They’d want to blend in with normal users as much as possible, wouldn’t they?
In theory, sure. But that implies the people doing this care to be stealthy, or put in that much effort. Web scrapers, aggregators and prepackaged tools set up on quickly deployed Linux VMs or containers don’t really care all that much until they do. So if you don’t go through the trouble to spoof the user agent, it doesn’t change. Even now, it’s not necessarily that those user agents are blocked, they just aren’t counted as real users anymore
Comments
Wouldn’t most bots want to self-report as the most ‘ordinary user’ they could? Which probably means reporting as Chrome on Windows. They’d want to blend in with normal users as much as possible, wouldn’t they?
In theory, sure. But that implies the people doing this care to be stealthy, or put in that much effort. Web scrapers, aggregators and prepackaged tools set up on quickly deployed Linux VMs or containers don’t really care all that much until they do. So if you don’t go through the trouble to spoof the user agent, it doesn’t change. Even now, it’s not necessarily that those user agents are blocked, they just aren’t counted as real users anymore
Sign in to comment. Sign in