Introduction to the requests library in Python
requests library provides the tools to send requests over the Internet. The library allows you to perform
POST requests, which means that you can:
fetch the HTML of a website
call a third party API to fetch data
send data over to a server for processing
Fetching the HTML of a website
To fetch the HTML of a website:
import requests# The URL of the desired siteurl = "https://en.wikipedia.org/wiki/Python_(programming_language)"r = requests.get(url)print(r.text)<!DOCTYPE html><html class="client-nojs" lang="en" dir="ltr"><head><meta charset="UTF-8"/><title>Python (programming language) - Wikipedia</title>...
r.text is of type string.
Calling a third party API
Let us call a free API hosted by Github that tells us how many projects there currently are for some programming language:
The GitHub API returns a JSON, which we can extracted using
r.json(). In Python, JSONs are treated as dictionaries, so you can get to use all the properties and methods of dictionaries to interact with the data.
status_code tells you whether or not your request was successful. A status code of
200 means that the request was successful and that no problem was encountered. Unsuccessful requests would result in a status codes starting with a
In the final line, we've printed out all the keys in the response. The keys that you see here are unique to this GitHub API, so you won't see the same keys when you call other third party APIs. In this case, we are just interested in extracting value for the the
Raising an error
By default, no error will be raised when the status code of the response is
import requests# Some invalid urlurl = "https://en.wikipedia.org/wiki/Python_(programming_langcccuafawefwaege)"r = requests.get(url)print(r.status_code)404
If we want to raise an error instead, call
r.raise_for_status() like so:
import requests# Some invalid urlurl = "https://en.wikipedia.org/wiki/Python_(programming_langcccuage)"r = requests.get(url)r.raise_for_status()print(r.status_code)HTTPError: 404 Client Error: Not Found for url: https://en.wikipedia.Python_(programming_langcccuage)