Skip to content
elephantoo

HTTP requests & web APIs

Lesson 30 of 38 15 min read

Call REST APIs with requests: GET/POST, params, JSON, auth, timeouts, error handling, sessions, retries and pagination.

Modern software is glued together by web APIs: payment gateways, weather services, GitHub, Slack, your own company's microservices. They speak HTTP and usually exchange JSON. The third-party requests library is the most popular way to call them from Python — it's simple, robust and used by millions of projects. In this lesson you'll make GET and POST requests, handle errors and timeouts properly, authenticate, and build a small reusable API client.

Terminal
pip install requests

The examples use two free public test APIs: JSONPlaceholder (fake blog data) and httpbin.org (echoes back what you send).

HTTP in two minutes#

An HTTP request has a method, a URL, headers and optionally a body. The server returns a response with a status code, headers and a body.

MethodTypical meaning
GETread a resource
POSTcreate a resource / submit data
PUT / PATCHreplace / partially update
DELETEdelete
StatusMeaning
2xxsuccess (200 OK, 201 Created, 204 No Content)
3xxredirect (requests follows these automatically)
4xxclient error (400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 429 Too Many Requests)
5xxserver error (500, 502, 503)

Your first GET request#

Python
import requests

response = requests.get("https://jsonplaceholder.typicode.com/todos/1", timeout=10)

print(response.status_code)
print(response.headers["Content-Type"])
data = response.json()            # parse the JSON body into Python objects
print(data)
print(data["title"])
Output
200
application/json; charset=utf-8
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}
delectus aut autem

The response object gives you status_code, headers (a case-insensitive dict), text (body as a string), content (raw bytes) and json() (parsed JSON).

Query parameters#

Don't glue query strings together by hand; pass a dict with params and requests encodes it safely:

Python
import requests

r = requests.get(
    "https://jsonplaceholder.typicode.com/posts",
    params={"userId": 2, "_limit": 3},
    timeout=10,
)
print(r.url)
for post in r.json():
    print(post["id"], post["title"][:40])
Output
https://jsonplaceholder.typicode.com/posts?userId=2&_limit=3
11 et ea vero quia laudantium autem
12 in quibusdam tempore odit est dolorem
13 dolorum ut in voluptas mollitia et saepe

Sending data: POST with JSON#

Python
import requests

new_post = {"title": "Learning Python", "body": "requests is great", "userId": 1}
r = requests.post("https://jsonplaceholder.typicode.com/posts", json=new_post, timeout=10)

print(r.status_code)          # 201 Created
print(r.json())
Output
201
{'title': 'Learning Python', 'body': 'requests is great', 'userId': 1, 'id': 101}

json= serialises your dict and sets the Content-Type: application/json header. Use data= for HTML-form-style submissions and files= for uploads. requests.put, requests.patch and requests.delete work the same way.

Headers and authentication#

APIs usually require an API key or token, normally sent in a header. Never hard-code secrets in source code — read them from environment variables:

Python
import os
import requests

token = os.environ.get("DEMO_API_TOKEN", "test-token-123")
headers = {
    "Authorization": f"Bearer {token}",
    "Accept": "application/json",
    "User-Agent": "elephantoo-tutorial/1.0",
}

r = requests.get("https://httpbin.org/headers", headers=headers, timeout=10)
echoed = r.json()["headers"]
print(echoed["Authorization"])
print(echoed["User-Agent"])

r = requests.get("https://httpbin.org/basic-auth/ada/s3cret", auth=("ada", "s3cret"), timeout=10)
print(r.status_code, r.json())
Output
Bearer test-token-123
elephantoo-tutorial/1.0
200 {'authenticated': True, 'user': 'ada'}

In development, people often keep secrets in a .env file loaded with the python-dotenv package — and add .env to .gitignore.

Handling errors properly#

Network code fails in many ways: DNS errors, refused connections, slow servers, 404s, 500s, invalid JSON. Two crucial facts:

  1. requests has no default timeout. Without timeout=, a stuck server can hang your program forever.
  2. HTTP error codes don't raise exceptions by themselves. Call raise_for_status().
Python
import requests


def fetch_json(url):
    try:
        r = requests.get(url, timeout=5)
        r.raise_for_status()                   # 4xx/5xx -> HTTPError
        return r.json()
    except requests.Timeout:
        print("timed out")
    except requests.HTTPError as e:
        print("HTTP error:", e.response.status_code)
    except requests.ConnectionError:
        print("could not connect")
    except requests.JSONDecodeError:
        print("response was not JSON")
    return None


print(fetch_json("https://jsonplaceholder.typicode.com/users/1")["name"])
fetch_json("https://jsonplaceholder.typicode.com/users/9999")
fetch_json("https://httpbin.org/status/503")
fetch_json("https://httpbin.org/html")
fetch_json("https://no-such-host.invalid")
Output
Leanne Graham
HTTP error: 404
HTTP error: 503
response was not JSON
could not connect

All of these exceptions inherit from requests.RequestException, which you can catch as a fallback.

Sessions: connection reuse and shared settings#

When you call the same API repeatedly, use a requests.Session. It reuses TCP connections (much faster), and lets you set headers, auth and cookies once:

Python
import requests

with requests.Session() as session:
    session.headers.update({"Accept": "application/json", "User-Agent": "elephantoo/1.0"})
    users = session.get("https://jsonplaceholder.typicode.com/users", timeout=10).json()
    for user in users[:3]:
        todos = session.get(
            "https://jsonplaceholder.typicode.com/todos",
            params={"userId": user["id"], "completed": "false"},
            timeout=10,
        ).json()
        print(f"{user['name']:<20} {len(todos)} open todos")
Output
Leanne Graham        9 open todos
Ervin Howell         12 open todos
Clementine Bauch     13 open todos

Automatic retries#

Transient failures (a 503, a 429 rate limit, a dropped connection) are often fixed by waiting and trying again. requests can do this with urllib3's Retry, mounted on a session:

Python
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=3,
    backoff_factor=0.5,                       # wait 0.5s, 1s, 2s between tries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET"],                  # only retry safe, idempotent methods
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))

r = session.get("https://jsonplaceholder.typicode.com/posts/1", timeout=10)
print(r.status_code, r.json()["id"])
Output
200 1

Be careful retrying non-idempotent requests like POST — you might create the same order twice.

Pagination#

APIs return large collections in pages. Loop until there's nothing left — a generator keeps the calling code clean:

Python
import requests


def iter_posts(session, page_size=25):
    page = 1
    while True:
        r = session.get(
            "https://jsonplaceholder.typicode.com/posts",
            params={"_page": page, "_limit": page_size},
            timeout=10,
        )
        r.raise_for_status()
        batch = r.json()
        if not batch:
            return
        yield from batch
        page += 1


with requests.Session() as s:
    posts = list(iter_posts(s))
print(len(posts), posts[-1]["id"])
Output
100 100

Real APIs signal the next page in different ways — a next URL in the JSON, a Link header (r.links["next"]["url"]), or a cursor token. Read the API's docs.

Worked example: a small API client class#

Wrapping an API in a class keeps URLs, auth and error handling in one place:

Python
from dataclasses import dataclass

import requests


@dataclass
class Todo:
    id: int
    title: str
    completed: bool


class TodoClient:
    def __init__(self, base_url="https://jsonplaceholder.typicode.com", timeout=10):
        self.base_url = base_url.rstrip("/")
        self.timeout = timeout
        self.session = requests.Session()
        self.session.headers["Accept"] = "application/json"

    def _get(self, path, **params):
        r = self.session.get(f"{self.base_url}{path}", params=params, timeout=self.timeout)
        r.raise_for_status()
        return r.json()

    def todos_for(self, user_id: int) -> list[Todo]:
        return [Todo(t["id"], t["title"], t["completed"]) for t in self._get("/todos", userId=user_id)]


client = TodoClient()
todos = client.todos_for(1)
done = sum(t.completed for t in todos)
print(f"user 1: {done}/{len(todos)} done")
print(todos[0])
Output
user 1: 11/20 done
Todo(id=1, title='delectus aut autem', completed=False)

In tests, you'd replace the network calls with fakes (see the pytest lesson) so tests are fast and don't depend on the internet.

Beyond requests: httpx and async#

httpx offers an almost identical API plus async support and HTTP/2 — useful with FastAPI and asyncio (covered in the concurrency lesson):

Python
import httpx

r = httpx.get("https://jsonplaceholder.typicode.com/users/2", timeout=10)
print(r.status_code, r.json()["username"])
Output
200 Antonette

Common mistakes#

  • No timeout — your program can hang forever.
  • Not checking the status code — use raise_for_status().
  • Hard-coding API keys in code or committing them to git.
  • Building query strings by hand — use params=.
  • Hammering an API in a tight loop — respect rate limits (429), cache results, and back off.
  • Creating a new connection for every call in a loop — use a Session.

What's next#

An API client without tests is a liability. Next you'll learn the industry-standard testing tool: pytest.

Check your understanding

Quick quiz

0/3 answered

  1. 1.Why should you always pass timeout= to requests.get()?

  2. 2.What does response.raise_for_status() do?

  3. 3.How should you send a JSON body in a POST request with requests?

Finished reading?

Mark this lesson complete to track your progress.