3+1 Chrome Tools you can use to scrape websites faster

Image for post
Image for post
Photo by Jens Kreuter on Unsplash

1. The copy functionality

Image for post
Image for post
Copy XPath directly from the Chrome Developer Tools
// So by copying the XPath for the element we get a query that will work for the specific element
// But our aim is to get all cite_note-* elements so we'll need to experiment with that.
//*[@id="cite_note-4"]

2. The $x command

Image for post
Image for post
// In our case it helps us find a way to match all cite_notes and make sure that our XPath syntax is correct.
$x('//li[contains(@id,"cite_note-")]')
Image for post
Image for post
This way we’re getting all 75 matching elements.

3. Copying requests (Reproducing complex requests in your code)

Image for post
Image for post
Copy the request as a cURL command

4. Curl to code service (Github)

Founder & CTO @ GuestFlip

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store