Pagination is rarely just a matter of clicking “next.” A reliable scraper must identify how a site or API represents progress, request the next batch without losing state, stop at the right time, and prevent duplicate records. This guide explains the main web scraping pagination patterns—numbered pages, next links, cursor APIs, load-more controls, and infinite scroll—so you can choose an appropriate implementation and build a durable collection workflow.
Overview
Pagination divides a large result set into smaller responses. The first response may contain ten products, fifty search results, or a page of records, followed by a mechanism for requesting more. That mechanism can be visible in the HTML, embedded in a link, returned as JSON, or generated only after JavaScript runs.
The central task in web scraping pagination is to turn that mechanism into a controlled loop:
- Request or load the current batch.
- Extract records and pagination state.
- Validate the batch and remove duplicates.
- Determine whether another batch exists.
- Continue until a defined stopping rule is reached.
Start by inspecting the network requests made by the page. If the browser receives a JSON response containing the records, an API-oriented scraper is often simpler and more stable than parsing rendered markup. If the data exists directly in the initial HTML, a standard HTTP client and parser may be sufficient. When content appears only after interaction, a browser automation tool such as Playwright, Puppeteer, or Selenium may be appropriate. The choice should follow the page’s delivery mechanism rather than the visual appearance of its pagination.
For background on choosing an implementation, see Playwright vs Puppeteer vs Selenium for Web Scraping and Playwright vs BeautifulSoup for Web Scraping.
Core concepts
Numbered pagination
Numbered pagination uses a page number, offset, or similar parameter. Typical requests look like ?page=2, ?offset=50, or ?start=50. Do not assume that page numbers and offsets are interchangeable. A page number usually identifies a batch, while an offset identifies the number of records to skip.
A basic Python pattern might look like this:
page = 1
seen = set()
while True:
response = fetch(url, params={"page": page})
records = parse_records(response)
new_records = [r for r in records if r["id"] not in seen]
if not new_records:
break
save(new_records)
seen.update(r["id"] for r in new_records)
page += 1The important details are not the loop syntax but the stopping conditions. Stop when the response contains no records, when the response reports that there is no next page, or when a configured maximum has been reached. A maximum page limit protects the workflow from malformed pagination or an endpoint that keeps returning the same data.
Next-page links
Some sites include a direct link such as <a rel="next" href="...">. Others place a “Next” link in a navigation container or expose a URL in a JSON field. Following the supplied URL is generally safer than reconstructing it, because the link may preserve filters, sorting, locale settings, or an opaque token.
Normalize relative URLs, retain the original query parameters where appropriate, and record each visited URL. If the next link points to a URL already visited, stop and log the condition. This prevents a loop caused by broken navigation.
Cursor-based API pagination
A cursor pagination scraper does not normally increment a page number. Instead, each response returns a cursor, token, or continuation value for the next request. For example, a response might contain:
{
"items": [{"id": "a1"}],
"next_cursor": "eyJvZmZzZXQiOjUwfQ=="
}Send the cursor exactly as returned, and treat it as opaque. Do not decode, edit, sort, or mathematically increment it unless the API documentation explicitly defines that behavior. A cursor can represent a position, a snapshot, or other server-side state. Stop when the cursor is absent, null, empty, or equal to a previously used cursor.
Cursor APIs can change while you are collecting data. To reduce inconsistency, save the cursor alongside each batch and design the process so it can resume from the last successful request. Stable record identifiers and deduplication remain necessary even when the API provides cursors.
Load-more buttons and infinite scroll
A load-more interface is often a visual wrapper around one of the patterns above. Clicking the button may trigger a request containing an offset, cursor, or page number. Before automating clicks, inspect the browser’s network panel and look for the underlying request. Replaying that request can be faster and easier to test than repeatedly rendering the page.
When the request cannot be used independently, automate the interaction. Wait for the new records to appear, compare the record count or identifiers before and after the click, and stop when the button is disabled, removed, hidden, or fails to add new content. For scrape infinite scroll workflows, scroll in measured steps and wait for a specific response or DOM change instead of using an arbitrary long delay.
Browser automation introduces timing issues: a page may render placeholders, request several resources, or update the DOM after the network response. Use explicit waits for meaningful conditions, and capture diagnostic HTML, screenshots, or response metadata when a batch fails. The troubleshooting guidance in Web Scraping Troubleshooting Guide is useful when selectors or JavaScript behavior change.
Stopping rules and duplicate prevention
Every pagination job should have more than one stopping rule. Useful safeguards include:
- Stop after an empty batch.
- Stop when no new unique identifiers appear.
- Stop when the next URL or cursor repeats.
- Stop when the interface reports that all results are loaded.
- Stop at a configured maximum number of pages or records.
- Stop after repeated request or parsing failures, then alert rather than silently continuing.
Deduplicate using a stable key such as an item ID, canonical URL, or a carefully constructed combination of fields. Avoid using the record’s position in the page as its identity; positions change when new records are inserted or results are reordered.
Related terms
- Offset pagination
- Requests identify how many records to skip. It is straightforward but may produce overlap or gaps when the underlying dataset changes during collection.
- Page pagination
- Requests identify a numbered batch, usually through a page parameter or path segment.
- Cursor pagination
- Requests use a continuation value returned by the previous response. The cursor should normally be treated as opaque.
- Continuation token
- A token that tells an API where to continue. It may be called a cursor, next token, bookmark, or continuation key.
- Canonical URL
- A normalized URL used to identify a page or record consistently, especially when query parameters or tracking values vary.
- Idempotent collection
- A workflow that can be retried without creating duplicate output. Stable keys, upserts, and saved progress help achieve this.
Practical use cases
Pagination patterns appear in more workflows than product listings. A documentation crawler may follow next links through articles. A monitoring job may collect every result from a filtered search endpoint. An internal data pipeline may consume cursor-based records from an API. A research workflow may capture content behind a load-more interface for later analysis.
For each project, write down the pagination contract before coding:
- Input: Which URL, endpoint, filters, headers, or session state are required?
- Progress marker: Is progress represented by a page, offset, next URL, cursor, or browser action?
- Record identity: Which field reliably identifies one record?
- Completion signal: What proves that no further records are available?
- Recovery point: What state must be saved to resume after a failure?
- Quality checks: How will you detect an empty response, changed schema, duplicate batch, or unexpected login page?
Keep raw responses or a small diagnostic sample during development. Log the page number, cursor, URL, record count, unique record count, response status, and elapsed time. These fields make it much easier to distinguish a genuine end of data from a selector failure or a blocked request.
Respect the site’s access requirements and avoid collecting data you are not authorized to use. Apply conservative request pacing, identify your client appropriately where required, and handle errors without repeatedly sending requests. See How to Build Reliable Web Scrapers and Rate Limiting in Web Scraping for workflow considerations.
When the result is ready, export it with its source URL, collection timestamp, and stable identifier. Those fields support later validation and make downstream exports to CSV, spreadsheets, or databases more useful. The guide to exporting scraped data covers common delivery options.
When to revisit
Revisit a pagination implementation whenever the site changes its URL structure, frontend framework, API response shape, authentication flow, or result ordering. A scraper can continue returning HTTP success responses while silently collecting zero records, the same batch repeatedly, or an interstitial page. Treat unexpected record counts as an operational warning rather than assuming the job succeeded.
Review the workflow after a change to filters or sorting, because those changes can alter cursor behavior and create overlap. Recheck offset-based jobs when the source dataset changes frequently; records inserted between requests can shift later offsets. Also revisit browser-based workflows when load-more controls, lazy loading thresholds, or visible labels change.
A practical maintenance checklist is:
- Run a small test collection and compare record counts with a known page range.
- Confirm that the first and last batches contain the expected fields.
- Check for repeated URLs, cursors, or identifiers.
- Verify that the completion signal still works.
- Review logs for retries, redirects, authentication pages, and schema changes.
- Update selectors, request parameters, or parsing rules only after capturing the new response shape.
Pagination is best treated as a state-management problem, not merely a loop. Identify the state, save it, validate every transition, and give the process clear ways to stop. That approach makes numbered pages, next links, cursor APIs, load-more buttons, and infinite-scroll interfaces manageable as variations of the same dependable automation pattern.
