Requests

Requests#

The requests is a Python library that allows you to send http requests. Read more about this library in the official documentation.

This notebook uses a custom kernel HTTP IPython that listens on port 3232. If there is some content, it prints it as the output of the cell.

In the following cell we create container of the httpbin which allows to check headers of the requests.

import requests
import docker
import requests

docker_client = docker.from_env()

from src.rerun_docker import reload_docker_container
ans = reload_docker_container(
    name="httpbin",
    image="kennethreitz/httpbin",
    ports={80: 80},
    detach=True,
    remove=True
)

Headers#

Pass the headers to your request by specifying the headers argument as a dict[str, str].

The following cell sends the GET request with the specified headers.

ans = requests.get(
    "http://localhost:3232",
    headers={"a": "hello"}
)

GET /headers HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive
a: hello

Default headers#

There are some headers that the requests library generates itself - even if you’ve specified empty headers, there will be some headers in the request.

The following example shows that requests with headers specified as empty lead to requests that still have some headers.

prepared_request = requests.Request(
    'GET', 
    'http://localhost:3232', 
    headers={}
).prepare()

session = requests.Session()
response = session.send(prepared_request)

GET / HTTP/1.1
Host: localhost:3232
Accept-Encoding: identity
User-Agent: python-urllib3/2.5.0

Even though we explicitly specified headers={}, it still results in some filled headers.

Data#

Send data using the requests.post function, with parameters:

  • data: bytes, filelike, dict or list of tuples.

  • json: for any object that can be serialised as json.

How specify the data for the requests.post method determines how it will be represented in http.

Form#

To represent data as a form pass dictionary or list of tuples in the data parameter.

The following cell displays the POST request, where data passed as dictionary.

ans = requests.post("http://localhost:3232", data={"a": [20, 30]})

POST / HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive
Content-Length: 9
Content-Type: application/x-www-form-urlencoded

a=20&a=30

The Content-Type header takes the corresponding value.

Raw data#

Any other type of input passed to the data will result in the sending of data with an uspecified type.

The following cell specifies the data argument as a string. Despite the fact that the string follows the JSON format, it is treated as a raw information by the requests anyway.

requests.post("http://localhost:3232", data='{"a": 10}')

<Response [200]>

POST / HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive
Content-Length: 9

{"a": 10}

As a result, the Content-Type header is not provided.

JSON#

If you pass information through json parameter, the request will be set as JSON.

Consider the case in which as json argument provided the dictionary.

ans = requests.post("http://localhost:3232", json={"a": 10})

POST / HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive
Content-Length: 9
Content-Type: application/json

{"a": 10}

The Content-Type header takes value application/json.

Query params#

With the params argument in methods, you can specify specify the query parameters. requests automatically add them to the URL.

The following cell displays the complete URL for a get request when the params argument specified.

ans = requests.get(
    "http://localhost:3232",
    params={"a": 10, "b": "some_value"}
)

GET /?a=10&b=some_value HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive

Reponse#

The methods that provide requests retun the requests.Response instance. The following tale shows the most useful attributes of the requests.Response class.

Attribute / Method

Description

status_code

Integer HTTP status code (e.g., 200, 404).

headers

Response headers as a case-insensitive dict.

text

Body decoded to a string using detected encoding.

content

Raw response body as bytes.

json()

Parses the body as JSON and returns a Python object.

url

Final request URL (after redirects).

ok

True if status code < 400.

reason

Reason phrase from the server (e.g., “Not Found”).

cookies

Cookies provided by the server.

elapsed

Time taken for the request.

history

List of intermediate responses (e.g., redirects).

raise_for_status()

Raises an exception for HTTP status codes ≥ 400.

The follwoing cell sends the request and saves the ouput as response.

response = requests.get("http://localhost:3232/anything")

GET /anything HTTP/1.1
Host: localhost:3232
User-Agent: python-requests/2.32.5
Accept-Encoding: gzip, deflate, zstd
Accept: */*
Connection: keep-alive

The following cell displays the attribute url of the response.

response.url

'http://localhost:3232/anything'

Wrong status#

With the raise_for_status function, you can automatically raise a special requests.expceptions.HTTPError exception for incorrect HTTP codes.

The following cell correctly assigns response with a correct HTTP code.

response = requests.get("http://localhost:80/status/200")
response

<Response [200]>

The raise_for_status method of this response does not have any effect.

response.raise_for_status()

In contrast, consider the response with a 400 status code.

response = requests.get("http://localhost:80/status/400")
response

<Response [400]>

The raise_for_status mehtod raises the corresponding exception.

try:
    response.raise_for_status()
except Exception as e:
    print(e)
    print(type(e))

400 Client Error: BAD REQUEST for url: http://localhost:80/status/400
<class 'requests.exceptions.HTTPError'>

Content#

The response has several attributes that represent the content at different stages of processing:

  • raw: returns the raw data read from the socket.

  • iter_content: for reading streaming data.

  • content: the pure butes.

  • text: content decoded with detected encoding.

  • json: returns a python dictionary if the response content can be decerialized from JSON.

Examine different components of the response as defined in the following cell.

response = requests.get("http://localhost:80/anything")

The following cell displays the content attribute.

response.content

b'{\n  "args": {}, \n  "data": "", \n  "files": {}, \n  "form": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate, zstd", \n    "Connection": "keep-alive", \n    "Host": "localhost", \n    "User-Agent": "python-requests/2.32.5"\n  }, \n  "json": null, \n  "method": "GET", \n  "origin": "172.17.0.1", \n  "url": "http://localhost/anything"\n}\n'

For the text ouput the same, but encoded as a str data type.

response.text

'{\n  "args": {}, \n  "data": "", \n  "files": {}, \n  "form": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate, zstd", \n    "Connection": "keep-alive", \n    "Host": "localhost", \n    "User-Agent": "python-requests/2.32.5"\n  }, \n  "json": null, \n  "method": "GET", \n  "origin": "172.17.0.1", \n  "url": "http://localhost/anything"\n}\n'

With json method you can get the body of the response as python dictionary.

response.json()

{'args': {},
 'data': '',
 'files': {},
 'form': {},
 'headers': {'Accept': '*/*',
  'Accept-Encoding': 'gzip, deflate, zstd',
  'Connection': 'keep-alive',
  'Host': 'localhost',
  'User-Agent': 'python-requests/2.32.5'},
 'json': None,
 'method': 'GET',
 'origin': '172.17.0.1',
 'url': 'http://localhost/anything'}
Contents