Scraping every record from a website is rarely just a matter of incrementing a page number. This practical checklist shows how to handle numbered pages, next-page links, cursor-based APIs, load-more controls, and infinite scroll while preventing skipped records, repeated requests, and silent duplicates.
Overview
Before writing a web scraper, identify how the target delivers the next batch of records. A visible page number may represent a simple page=2 parameter, but the same interface could use a hidden cursor, a POST request, a JavaScript data endpoint, or a button that appends results to the current document.
A reliable pagination workflow has five parts:
- Discover the pagination mechanism: inspect links, network requests, forms, scripts, and response data.
- Request the next batch: preserve the parameters, headers, cookies, or request body required by the source.
- Extract records: use stable selectors or structured response fields rather than relying only on visual position.
- Detect completion: stop when the source signals that there are no more records or when progress stops.
- Validate and deduplicate: compare counts, identifiers, URLs, and page boundaries before treating the crawl as complete.
Keep the pagination layer separate from extraction and storage. That separation makes it easier to replace a page-number loop with cursor handling when a site changes, and it allows the same extraction logic to be tested against saved HTML fixtures. For a testing workflow, see How to Test Web Scrapers with HTML Fixtures, Snapshots, and CI Regression Checks.
Also confirm that your collection method is appropriate for the site, respects applicable terms and access controls, and uses a reasonable request rate. If authentication is required, treat session handling as a separate concern; the patterns in Scraping Authenticated Websites can help structure that work.
Checklist by scenario
Numbered pagination
Numbered pagination is usually the simplest pattern, but do not assume that every number exists or that pages are immutable. First capture the initial URL and inspect links such as ?page=2, /page/2, or form parameters. Then:
- Record the starting page and any filters or sort parameters.
- Confirm whether page numbering starts at zero or one.
- Preserve all query parameters that define the result set.
- Extract a stable item identifier from every page.
- Stop on an empty result set, a known last-page indicator, or a page with no new identifiers.
- Set a maximum page limit as a safety guard against malformed links or loops.
A simple Python outline looks like this:
for page in range(1, max_pages + 1):
response = fetch(url, params={"page": page, **filters})
records = extract_records(response.text)
new_records = add_unseen(records)
if not records or not new_records:
break
The exact request client can vary. Compare implementation trade-offs in Requests vs HTTPX vs aiohttp for Web Scraping.
Next-page links
Some sites do not expose reliable page numbers. Instead, each response contains a link to the next result set. Prefer the canonical next link supplied by the document or API over constructing a URL yourself.
- Find the next link using a semantic relation, such as
rel="next", when available. - Resolve relative URLs against the current response URL.
- Stop when the next link is absent, empty, disabled, or identical to a previously visited URL.
- Store visited URLs to prevent cycles.
- Keep filters and sorting consistent unless the next link explicitly changes them.
Selectors should survive minor layout changes. Use a combination of attributes, link relationships, and stable content structure rather than a deeply nested CSS path. The guide to Resilient Web Scraper Selectors covers practical choices for CSS, XPath, text, and ARIA-based extraction.
Cursor-based APIs
Cursor pagination returns a token representing the position of the next batch. The token may appear in a response field such as next_cursor, nextCursor, continuation, or a nested pagination object. Treat it as opaque: store it and send it back exactly as returned instead of trying to decode or increment it.
cursor = None
seen_cursors = set()
while True:
payload = {"limit": 100}
if cursor:
payload["cursor"] = cursor
data = request_json(endpoint, json=payload)
records = data.get("items", [])
save_unseen(records)
next_cursor = data.get("next_cursor")
if not records or not next_cursor or next_cursor in seen_cursors:
break
seen_cursors.add(next_cursor)
cursor = next_cursor
Check whether the API expects the cursor in query parameters, a JSON body, or a header. Also preserve the same filters, account scope, and sort order for every request. If the source uses GraphQL, pagination fields and variables may be embedded in the query; see Scraping GraphQL APIs for a structured approach.
Load-more buttons and infinite scroll
A load-more button is often a front end for an ordinary API request. Use browser developer tools to click the control and inspect the resulting network request. Look for a changing offset, cursor, page token, or request body. If a direct endpoint is available and permitted, it is generally easier to test and monitor than repeatedly rendering the page.
When browser automation is necessary, define a clear stopping rule:
- Click only after the previous batch has finished loading.
- Wait for a specific response, new item count, or DOM change rather than an arbitrary delay alone.
- Measure the number of records before and after each action.
- Stop when the button disappears, becomes disabled, or produces no new identifiers.
- Capture errors and empty responses instead of interpreting them as completion.
Infinite scroll should be treated as an interaction pattern, not as a special data format. The important questions remain: what request loads the next batch, what token or offset advances the request, and how can you prove that new records arrived?
What to double-check
Request continuity: Compare the first and later requests. Verify that cookies, authorization state, headers, query filters, language settings, and sort order are still present when required.
Progress: Log the page number or cursor, response status, number of extracted records, number of new records, and a representative identifier. A scraper that returns successful responses can still be stuck on the same page.
Completeness: If the source displays a total count, compare it with your unique-record count. Where no total is available, compare adjacent batches, inspect the final response, and sample records from the beginning, middle, and end of the crawl.
Deduplication: Prefer a source identifier. If none exists, use a carefully designed compound key such as normalized URL plus a relevant date or title. Do not use the record's position on a page as its identity; positions change when results are inserted or reordered.
Extraction stability: Make sure pagination controls are not being extracted as data records. For tables, check whether headers, footers, advertisements, or repeated mobile markup create false rows. For table-specific workflows, review HTML Table Scraping in Python.
Failure visibility: Record non-success responses, redirects, challenge pages, empty documents, parse errors, and unexpected content types. Monitoring guidance is available in the Web Scraper Monitoring Runbook.
Common mistakes
- Stopping at the first empty page: An empty response may be a transient failure or a blocked request. Distinguish an error from a valid end-of-results response.
- Incrementing a cursor: Cursor values are usually opaque. Reuse the returned value exactly.
- Ignoring filters: Dropping a category, date range, or sort parameter on later requests can create mixed or incomplete data.
- Using fixed sleeps as the only wait strategy: A fixed delay may be too short for a slow response and unnecessarily long for a fast one. Wait for observable progress and retain a timeout.
- Trusting HTTP status alone: A successful response can contain an error page, login screen, or empty shell. Validate content type and expected fields.
- Failing to cap the crawl: Add limits for pages, requests, cursors, and elapsed time so a broken stopping condition cannot run indefinitely.
- Deduplicating after export: Deduplicate during collection when possible, while retaining enough logging to investigate why duplicates appeared.
When to revisit
Review a pagination scraper before a seasonal planning cycle, after a site redesign, or whenever the source changes its front end or API. Recheck it when record counts suddenly fall, duplicate rates rise, response sizes change, or the final page appears unusually large or small.
Use this short maintenance checklist:
- Run a known fixture or snapshot test for the pagination and extraction logic.
- Manually inspect the first request, a middle request, and the final request.
- Confirm that the stopping condition still works for an empty final batch.
- Compare unique identifiers with a previous crawl or an available source count.
- Review logs for repeated URLs, repeated cursors, empty pages, redirects, and parsing warnings.
- Update selectors, request templates, and browser automation code when the source workflow changes.
The practical goal is not merely to fetch more pages. It is to produce a defensible collection in which every request advances, every record can be identified, and completion can be explained. Keep pagination, extraction, validation, and storage independently testable, and a change from numbered pages to cursors or infinite scroll becomes a manageable maintenance task rather than a full rewrite.