Web Scraping with Python: A Practical Guide to Requests, BeautifulSoup, and Pagination

Overview

A small scraping script can be useful for a one-time task, but a reusable scraper needs more than a selector and a loop. Pages can time out, links can be relative, markup can change, and pagination may stop in more than one way. A dependable workflow separates these concerns so each part is easier to test and update.

This guide uses Requests to retrieve pages and BeautifulSoup as the HTML parser. The pattern is appropriate for pages whose useful content is present in the server-delivered HTML. If the initial response contains only an application shell and the data appears after JavaScript runs, inspect the site's available data endpoints or use a browser automation approach where appropriate. The related guide to headless browsers for web scraping covers that separate workflow.

Before collecting data, identify the pages you are allowed to access, keep request volume reasonable, and avoid collecting sensitive information unnecessarily. A scraper should be designed around the site's structure and your intended use, not around bypassing access controls.

Step-by-step workflow

1. Define the record before writing selectors

Start by describing one output record. For a product listing, that might be a name, price, detail URL, and source page. For an article index, it could be a title, author, publication date, and URL. This prevents the parser from becoming a collection of unrelated selectors.

Also decide which fields are required, which may be empty, and how duplicate records will be identified. A stable detail-page URL is often a better key than a title, because titles can repeat or change.

2. Create a small, explicit fetch function

Use a timeout, check the response status, and keep fetching separate from parsing. A session can reuse connection settings and makes it easier to apply consistent headers.

from urllib.parse import urljoin
import csv
import requests
from bs4 import BeautifulSoup


def fetch_html(session, url):
    response = session.get(url, timeout=20)
    response.raise_for_status()
    return response.text


session = requests.Session()
session.headers.update({
    "User-Agent": "ExampleResearchBot/1.0"
})

The user-agent should identify your application accurately rather than pretending to be a different client. For more detail on request identity, see how to use user agents correctly in web scraping.

3. Parse one page into records

Inspect a representative page and find the repeated container for each record. Prefer selectors tied to meaningful classes, attributes, or semantic elements. Avoid long chains based on incidental nesting, because small layout changes can break them.

def parse_list_page(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    records = []

    for card in soup.select("article.card"):
        link = card.select_one("a.card-title")
        if not link:
            continue

        title = link.get_text(" ", strip=True)
        detail_url = urljoin(page_url, link.get("href", ""))
        price_node = card.select_one(".price")

        records.append({
            "title": title,
            "url": detail_url,
            "price": price_node.get_text(" ", strip=True)
                     if price_node else ""
        })

    next_link = soup.select_one("a[rel='next']")
    next_url = urljoin(page_url, next_link["href"]) \\
        if next_link and next_link.get("href") else None

    return records, next_url

The selectors in this example are placeholders; replace them after inspecting the target HTML. Notice that urljoin handles relative links such as /items/42 and converts them into usable absolute URLs.

4. Follow pagination with a stopping condition

Pagination may use a next link, numbered URLs, a cursor, or a “load more” control. For ordinary linked pages, return the next URL from the parser and stop when it is absent. Also track visited URLs so a malformed or repeating link cannot create an endless loop.

def scrape_pages(start_url, max_pages=50):
    all_records = []
    visited = set()
    current_url = start_url

    with requests.Session() as session:
        session.headers.update({
            "User-Agent": "ExampleResearchBot/1.0"
        })

        for _ in range(max_pages):
            if not current_url or current_url in visited:
                break

            visited.add(current_url)
            html = fetch_html(session, current_url)
            records, next_url = parse_list_page(html, current_url)
            all_records.extend(records)
            current_url = next_url

    return all_records

A maximum page limit is a useful safety guard even when the site appears to have a clear final page. For intermittent failures, add deliberate retry behavior with a backoff, and log the URL and error rather than silently dropping the page. Rate control should be part of the design; the guide to rate limiting in web scraping covers practical strategies.

5. Export a stable, reviewable result

Keep the scraped records as dictionaries until parsing is complete, then write them to a format suited to the next handoff. CSV is convenient for spreadsheets and simple pipelines.

records = scrape_pages("https://example.com/catalog")

with open("catalog.csv", "w", newline="", encoding="utf-8") as output:
    writer = csv.DictWriter(
        output,
        fieldnames=["title", "url", "price"]
    )
    writer.writeheader()
    writer.writerows(records)

For larger or nested records, JSON may preserve the structure more naturally. If the destination is a shared workspace, plan the export separately from the parser. See options for exporting scraped data to Google Sheets, Airtable, and CSV.

Tools and handoffs

Requests and BeautifulSoup work well as a compact starting point because fetching, parsing, and output can remain ordinary Python functions. Add tools only when they solve a specific problem:

• Requests Session: centralizes headers and connection behavior for a run.
• BeautifulSoup: parses HTML and provides CSS selector support for common extraction tasks.
• CSV or JSON: creates a simple boundary between collection and analysis.
• Logging: records failed URLs, page counts, and extracted record counts for diagnosis.
• Browser automation: may be needed when content is rendered only after client-side JavaScript executes.

Keep the handoff explicit. A useful run should report where the data came from, when it was collected, how many pages were processed, and how many records were written. If another system consumes the output, document the field names and whether empty values are expected.

When a scraper grows into a shared internal service, separate the fetching job from the delivery mechanism. The guide to building a web scraping API for internal teams provides a useful next step, while webhooks versus polling helps frame delivery choices.

Quality checks

Successful HTTP responses do not guarantee useful data. Add checks that detect silent failures:

• Confirm that each page contains the expected repeated container before treating it as valid.
• Compare the number of records with a reasonable expectation for the page.
• Check that required fields, especially URLs and titles, are not empty.
• Normalize whitespace and convert relative URLs before exporting.
• Detect duplicate URLs or record keys before downstream analysis.
• Save a small sample of raw HTML during development so selector changes can be investigated.
• Record failed pages instead of hiding exceptions.

Run the parser against more than one page type if the site has promoted items, empty categories, or alternate templates. If the same item can appear on several pages, deduplicate after extraction using a stable key. The guide to deduplicating scraped data discusses that stage in more detail.

When selectors stop matching, first inspect the downloaded HTML rather than assuming BeautifulSoup failed. The page may have changed its markup, returned an access message, moved the data into JSON-LD, or started rendering the content in a browser. For a broader diagnostic checklist, use the web scraping troubleshooting guide.

When to revisit

A scraper should be treated as maintained code, not a finished bookmark. Revisit it when the target site's layout, URL scheme, pagination controls, or rendering method changes. Also review it when the output schema changes, a downstream import begins rejecting rows, or a run produces an unusual record count.

Make updates easier by keeping selectors in one parser function, fetch settings in one place, and export logic separate from both. Before changing production behavior, test the parser against saved HTML samples and compare the new output with a known-good result. After deployment, monitor page counts, empty-field rates, duplicate counts, and failed URLs.

For a practical maintenance routine, start with one target page, verify the response and selectors, run a small page limit, inspect the exported rows, and only then expand the collection. This process catches most structural problems early while keeping request volume controlled. When the target requires JavaScript, the pagination becomes cursor-based, or the workflow needs scheduled delivery, revisit the design rather than forcing the original Requests-and-BeautifulSoup pattern beyond its useful limits.