Pagination and Filtering
22 min · Intermediate
Need help? Ask a tutor on WhatsApp (opens in a new tab)A collection can hold millions of rows. Sending all of them in one response is slow and can crash the server or the client. Pagination returns a small page at a time, and filters let the client ask only for what it needs.
Never return everything
Pagination is splitting a long list into pages that the client asks for one at a time, so each response stays small and fast.
Every list endpoint needs a limit. Pick a default page size, such as 50, and a hard maximum, such as 100. A client that asks for more still gets only the maximum.
DEFAULT_LIMIT, MAX_LIMIT = 50, 100
def clamp(limit): return max(1, min(limit or DEFAULT_LIMIT, MAX_LIMIT))
print(clamp(None), clamp(500), clamp(10))The clamp keeps one careless request from loading the whole table.
Offset and limit
The simplest way uses two query parameters. skip says how many items to jump over, and limit says how many to return. In SQL this is OFFSET and LIMIT.
items = list(range(1, 11))skip, limit = 4, 3page = items[skip:skip + limit]print(page)print("next skip:", skip + limit)Example 1 · A reusable paginate helper
def paginate(items, skip=0, limit=50, max_limit=100): limit = max(1, min(limit, max_limit)) end = skip + limit return { "items": items[skip:end], "total": len(items), "next_skip": end if end < len(items) else None, }
data = list(range(1, 11))print(paginate(data, skip=0, limit=4))print(paginate(data, skip=8, limit=4))print(paginate(data, limit=1000)["next_skip"]){'items': [1, 2, 3, 4], 'total': 10, 'next_skip': 4}
{'items': [9, 10], 'total': 10, 'next_skip': None}
NoneWhat this code does
- The reply tells the client where to go next;
Nonemeans the last page. - A huge limit is cut down to the maximum.
Example 2 · Building the next link
from urllib.parse import urlencode
def next_link(path, skip, limit, total, **filters): if skip + limit >= total: return None query = {**filters, "skip": skip + limit, "limit": limit} return f"{path}?{urlencode(query)}"
print(next_link("/products", 0, 20, 45, category="books"))print(next_link("/products", 40, 20, 45, category="books"))/products?category=books&skip=20&limit=20 None
What this code does
- The filters travel with the link, so page two keeps the same filter.
- On the last page there is no link.
Slicing does the same job in Python. The reply should also tell the client where the next page starts.
Cursors keep pages stable
Cursor is a marker, often the id or time of the last item on a page, that the client sends back to get the next page. It is stable even when items are added or removed.
With offset paging, a new item added at the front shifts everything, so the next page repeats an item. A cursor remembers the last item seen, and the next page starts after it.
items = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]after_id, limit = 8, 3page = [i for i in items if i < after_id][:limit]print(page)Example 3 · Offset drift versus a cursor
items = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1] # newest first
page1 = items[0:3]items.insert(0, 11) # a new item arrivesoffset_page2 = items[3:6]
after_id = page1[-1]cursor_page2 = [i for i in items if i < after_id][:3]
print("page 1:", page1)print("offset page 2:", offset_page2)print("cursor page 2:", cursor_page2)page 1: [10, 9, 8] offset page 2: [8, 7, 6] cursor page 2: [7, 6, 5]
What this code does
- Offset paging repeats item 8 because everything moved down by one place.
- The cursor asks for items below 8, so the answer is right.
The page is found by value, not by position, so inserts elsewhere cannot move it. Databases do this fast with an index on the id.
Filtering and sorting
Filters, sorting and paging all belong in the query string, because they choose which part of the same collection to show. Keep names simple and validate every value.
from urllib.parse import urlencode
query = {"category": "books", "max_price": 20, "sort": "-price", "limit": 10}print("/products?" + urlencode(query))Example 4 · Filters, sorting and paging in FastAPI
from fastapi import FastAPI, Queryfrom fastapi.testclient import TestClient
app = FastAPI()PRODUCTS = [ {"id": 1, "name": "Atlas", "category": "books", "price": 25}, {"id": 2, "name": "Novel", "category": "books", "price": 12}, {"id": 3, "name": "Pen", "category": "office", "price": 2}, {"id": 4, "name": "Guide", "category": "books", "price": 18},]
@app.get("/products")def list_products( category: str | None = None, max_price: float | None = None, sort: str = "id", skip: int = Query(0, ge=0), limit: int = Query(50, ge=1, le=100),): rows = [p for p in PRODUCTS if (category is None or p["category"] == category) and (max_price is None or p["price"] <= max_price)] rows.sort(key=lambda p: p[sort.lstrip("-")], reverse=sort.startswith("-")) return {"total": len(rows), "items": [p["name"] for p in rows[skip:skip + limit]]}
client = TestClient(app)print(client.get("/products?category=books&max_price=20&sort=price").json())print(client.get("/products?sort=-price&limit=2").json())print(client.get("/products?limit=500").status_code){'total': 2, 'items': ['Novel', 'Guide']}
{'total': 4, 'items': ['Atlas', 'Guide']}
422What this code does
totalcounts the rows after filtering, before paging.- A limit above the maximum is rejected with 422 before the function runs.
A leading minus on sort is a common way to say "descending".
Next step
Check your understanding
12 quick questions