David, you are right about that. By using proxy rotation and user-agent rotation you can make your scrapers a lot less traceable.

What's more, you can use real headless browsers to generate sessions, CSRF tokens or other tokens that belong to a "real" user and then reuse those throughout your API calls.

As for obfuscated Javascript, the code on the frontend doesn't really affect you. The thing you're looking at when developing a new scraper are the Network requests. It doesn't matter if the frontend Javascript is hard to read.

One of the most difficult things you can run into are obfuscated responses (which is rare), but since they do cotnain the data you're looking for you can understand how they work with some tinkering and some patience.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store