Python Pagination Patterns — Core Concepts

Why Pagination Exists

When an API endpoint could return thousands or millions of results, sending everything in one response is impractical. It wastes bandwidth, slows down clients, and puts unnecessary load on the database. Pagination solves this by breaking results into pages.

The question isn’t whether to paginate, but how.

Three Pagination Strategies

Offset-Based Pagination

The simplest approach: specify a page number (or offset) and a page size.

Request: GET /users?page=3&per_page=20 Means: Skip the first 40 users, return the next 20.

In SQL: SELECT * FROM users ORDER BY id LIMIT 20 OFFSET 40

Pros: Simple to implement and understand. Users can jump to any page directly.

Cons: Performance degrades on large datasets. To skip 10,000 rows, the database still has to scan through all 10,000. Also, if new data is inserted between page requests, users might see duplicates or miss items.

Cursor-Based Pagination

Instead of “give me page 3,” you say “give me items after this marker.”

Request: GET /users?cursor=eyJpZCI6IDQwfQ==&limit=20 Means: Return 20 users after the one encoded in the cursor.

The cursor is typically a Base64-encoded reference to the last item seen (like {"id": 40}). The server uses it to pick up exactly where the previous page left off.

Pros: Consistent results even when data changes. No performance degradation on deep pages because the database uses an index seek instead of scanning.

Cons: No random page access — you can’t jump to “page 50.” Clients must traverse sequentially.

Keyset Pagination

A variant of cursor-based pagination that uses actual column values instead of encoded cursors.

Request: GET /users?after_id=40&limit=20 Means: Return 20 users where id > 40.

In SQL: SELECT * FROM users WHERE id > 40 ORDER BY id LIMIT 20

Pros: All the benefits of cursor pagination, plus the parameters are human-readable and bookmarkable.

Cons: Requires a unique, ordered column. Doesn’t work well for complex sort orders involving multiple columns.

Choosing the Right Pattern

Use offset pagination when:

• Your dataset is small (under ~10,000 items)
• Users need to jump to specific pages
• The data doesn’t change frequently between requests

Use cursor/keyset pagination when:

• Your dataset is large or growing
• You’re building infinite scroll or “load more” experiences
• Data is frequently inserted (feeds, logs, events)
• API performance must stay constant regardless of page depth

Response Envelope

A well-designed paginated response includes metadata about the pagination state:

{
  "data": [...],
  "pagination": {
    "total": 5842,
    "page": 3,
    "per_page": 20,
    "next": "/users?page=4&per_page=20",
    "prev": "/users?page=2&per_page=20"
  }
}

For cursor-based:

{
  "data": [...],
  "pagination": {
    "has_next": true,
    "next_cursor": "eyJpZCI6IDYwfQ==",
    "limit": 20
  }
}

Include next and prev links so clients don’t need to construct URLs themselves. This follows the HATEOAS principle and makes your API easier to use.

Common Misconception

“Offset pagination is always fine for small apps.” Even small apps can have tables that grow. If your user table is 500 rows today but 500,000 rows next year, migrating from offset to cursor pagination is a breaking API change. If you’re building an API that others depend on, start with cursor pagination — it works well at any scale.

The one thing to remember: Offset pagination is simple but breaks at scale; cursor pagination maintains consistent performance regardless of dataset size — choose based on where your data is headed, not where it is now.