API Pagination Clients — Core Concepts

Why APIs paginate

APIs paginate because sending unbounded data is dangerous. A query that returns 10 rows today might return 10 million rows next year. Without pagination, that single request could exhaust server memory, time out the connection, and consume all available bandwidth.

Pagination also enables parallel processing: you can process page 1 while fetching page 2, keeping memory constant regardless of the total dataset size.

The three pagination schemes

Offset-based pagination

The simplest form: GET /users?offset=0&limit=20, then GET /users?offset=20&limit=20. The client tracks position by counting items.

Pros: easy to implement, supports random access (“jump to page 5”).

Cons: unreliable when data changes. If a user is deleted between page requests, the offset shifts and you either skip an item or see one twice. This is called the sliding window problem. For datasets that change frequently, offset pagination produces inconsistent results.

Cursor-based pagination

The server returns an opaque cursor token with each page: { "data": [...], "next_cursor": "abc123" }. The client sends it back: GET /users?cursor=abc123&limit=20.

The cursor encodes the server’s position (usually the last item’s ID or timestamp), making it immune to insertions and deletions between pages. Stripe, Slack, Twitter, and most modern APIs use cursor-based pagination.

Cons: no random access. You can’t “jump to page 5” because cursors are sequential. For API consumers, this is rarely a problem.

Link-header pagination

The GitHub API pioneered this approach. Pagination URLs are in the response Link header:

Link: <https://api.github.com/users?page=2>; rel="next",
      <https://api.github.com/users?page=50>; rel="last"

The client follows rel="next" until it’s absent. This decouples the pagination implementation from the URL structure — the server can change its pagination strategy without breaking clients.

Lazy iteration: the Python pattern

The best practice for consuming paginated APIs in Python is wrapping pagination in a generator:

• The generator fetches one page at a time
• It yields individual items from each page
• The next page is only fetched when the current page’s items are exhausted

This means for user in client.users.list() works like iterating a regular list, but memory usage stays proportional to one page regardless of the total result count.

When to collect vs. when to stream

Sometimes you need all results in memory (sorting across pages, deduplication). Use list(client.users.list()) — but be aware this loads everything. For datasets over 10,000 items, consider whether you actually need them all at once or if processing page-by-page suffices.

Rate limits and pagination

Paginated endpoints count against your rate limit. Fetching 100 pages of 20 items consumes 100 API calls. Strategies to minimize this:

• Increase page size — request 100 items per page instead of 20 (if the API allows it)
• Add delays between pages — prevent rate limit errors on aggressive pagination
• Stop early — if you’re looking for a specific item, break out of the loop once found

Common misconception

Developers often write pagination consumers that load all pages eagerly, storing every result in a list before processing any of them. This defeats the purpose of pagination — you’ve just recreated the “dump everything at once” problem on the client side. Lazy iteration (generators) solves this by processing items as they arrive.

The one thing to remember: Cursor-based pagination is the most reliable scheme for changing data, and Python generators make it transparent — your code iterates naturally while pages load on demand.