Learn to Code via Tutorials on Repl.it

← Back to all posts
How to send a request and capture data in source using Python requests and re
jhash (14)

If you ever wanted to do simple scraping in python, then this is the tutorial for you.

This example will send a request to Google and capture the word "Google" from the title tags.

r = requests.get("https://google.com")

Sends a normal request to google. Typically, your browser would do this to almost any site. We must set this request to a variable to have any use for it.


Prints response from website.

Now this is the tricky part. This will parse or capture specific data from the website.

print_google = re.search("<title>(.*)</title>", r.text).group(1)

You must be specific when telling Python what to capture. I told Python to capture or parse everything in-between the title tags in the response source.
The syntax for this is

variablename = re.search("leftstring(.*)rightstring", VariableNameThatIwantToParseStuffFrom).group(1)

You can see an example of this in the provided code below.