Web Scraping Troubleshooting Guide: Fix Selectors, Pagination, JavaScript Rendering, and Rate Limits

Overview

Effective web scraping troubleshooting starts by separating the failure into layers. A scraper can fail before it reaches the page, while downloading the response, while locating elements, while interpreting values, or while storing the result. Testing these layers independently prevents guesswork.

A useful diagnostic sequence is:

1. Request: Did the target URL return a response, and was it the expected page?
2. Document: Does the downloaded HTML contain the data you need?
3. Selector: Does the CSS or XPath expression still identify the intended elements?
4. Pagination: Does the next-page mechanism still work, and does the loop terminate?
5. Rendering: Is the data inserted by JavaScript after the initial HTML loads?
6. Output: Are fields being cleaned, deduplicated, and saved correctly?

Record evidence at each stage. Save the URL, response status, final URL after redirects, a short response preview, the number of matched elements, and the reason a page was considered complete. These details turn “the web scraper is not working” into a specific, testable problem.

Before changing code, confirm that your collection is permitted and that your request rate is appropriate for the site. For request identity and session behavior, see How to Use User Agents Correctly in Web Scraping. For pacing and retry design, see Rate Limiting in Web Scraping: Strategies That Reduce Blocks.

What to track

Request and response health

Log the status code, response time, content type, response size, redirect destination, and whether the response contains an expected page marker. A successful HTTP response is not proof that the correct content was received. A login page, an error document, or an empty shell can all return a response that looks successful at the transport layer.

Use a small validation function rather than relying only on status codes:

def looks_like_target(response):
    content_type = response.headers.get("content-type", "")
    body = response.text
    return (
        response.status_code == 200
        and "text/html" in content_type
        and "product-list" in body
    )

The marker should be a stable piece of page structure, not a fragile string such as a changing timestamp. If validation fails, save the response for inspection before retrying repeatedly.

Selector behavior

For CSS selector debugging, track the number of matches per page and the percentage of records with each required field. A selector that returns zero elements is an obvious failure, but a selector that returns too many elements can be just as damaging. Compare a known-good fixture with a newly downloaded page.

With Beautiful Soup, inspect the first match and its surrounding markup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
items = soup.select("article.card")
print("matches:", len(items))
if items:
    print(items[0].prettify()[:1500])

Prefer selectors based on stable attributes or semantic structure. Avoid depending on automatically generated class names, deeply nested paths, or a fixed element position when a simpler relationship is available.

Pagination and completeness

Track the page number, requested URL, next-page URL, item count, duplicate count, and stop reason. Pagination bugs commonly produce infinite loops, repeated pages, skipped pages, or a clean-looking dataset that is incomplete.

Do not stop only because a request succeeded. Stop when there is no next link, the next link repeats a previously visited URL, the page returns no new records, or the site’s documented endpoint indicates the final page. A visited-URL set provides a simple safeguard:

visited = set()
url = start_url

while url and url not in visited:
    visited.add(url)
    page = fetch(url)
    records = parse_records(page)
    save(records)
    url = parse_next_url(page)

JavaScript rendering signals

To determine whether you need to scrape JavaScript websites with a browser, compare the raw response with the browser’s rendered DOM. If the required text appears in the browser but not in the downloaded HTML, the page may be populated through client-side requests or JavaScript execution. Other signals include an empty results container, script references to a data endpoint, or content that appears only after scrolling or interaction.

Use the least complex option that meets the requirement. First inspect embedded JSON, JSON-LD, or a network request that returns structured data. The guide How to Parse JSON-LD for Structured Web Scraping covers one useful alternative to extracting visible markup. If a browser is necessary, Playwright can wait for a meaningful condition rather than an arbitrary delay:

await page.goto(url, wait_until="domcontentloaded")
await page.locator("article.card").first.wait_for()
html = await page.content()

A condition-based wait is easier to maintain than a fixed sleep because it ties the workflow to the content you actually need.

Data quality and delivery

Monitor null rates, field lengths, duplicate keys, date parsing failures, and the number of records exported. A scraper can collect pages successfully while silently producing unusable data. Store a run summary with counts for requested pages, parsed pages, records found, records rejected, records saved, and errors.

For downstream workflows, review How to Export Scraped Data to Google Sheets, Airtable, and CSV and consider whether a webhook or polling model better fits your delivery process.

Cadence and checkpoints

Use two maintenance rhythms: a checkpoint after every run and a deeper review monthly or quarterly, depending on how often the target changes and how important the output is.

Every run

• Confirm the response passed validation.
• Compare page and record counts with a recent baseline.
• Check required-field completeness and duplicate rates.
• Record retries, throttling responses, timeouts, and browser errors.
• Preserve a sample of raw input and parsed output.

Monthly or quarterly

• Run the parser against saved HTML fixtures and a current sample.
• Review whether selectors still describe stable page structure.
• Test the first, middle, and final pagination paths.
• Compare raw HTML with rendered content for key fields.
• Review retry limits, request spacing, timeouts, and logging volume.
• Check that exports, deduplication, and downstream alerts still work.

Keep a small regression suite of representative pages: a normal page, an empty-results page, a final pagination page, a page with missing optional fields, and a page that previously caused a bug. These fixtures make Python web scraping debugging faster and reduce the risk of fixing one layout variation while breaking another.

How to interpret changes

Look for patterns rather than reacting to a single unusual run. A sudden drop to zero matches usually points to a selector, response, or rendering change. A gradual decline in records may indicate pagination drift, changed filtering, or a site section becoming unavailable. A rise in duplicate records often means the next-page link is repeating, query parameters are being discarded, or the parser is collecting both mobile and desktop variants.

Separate infrastructure symptoms from parser symptoms:

Change one variable at a time and rerun against the same fixture. If you replace a selector, test both old and new expressions temporarily and compare their matched nodes. If you switch from direct requests to a headless browser, measure whether the required data actually improves before accepting the added complexity. For browser options and trade-offs, see Best Headless Browsers for Web Scraping.

When a run produces more data than expected, treat that as a defect until explained. Validate uniqueness with a stable key, normalize URLs before comparison, and inspect a sample of newly added records. The related guide on deduplicating scraped data at scale provides a useful follow-up workflow.

When to revisit

Revisit this troubleshooting checklist after any target-site redesign, change in authentication or navigation, new pagination behavior, move from server-rendered HTML to client-rendered content, or change in the fields your pipeline requires. Also review it when record counts shift unexpectedly for more than one run, when error rates rise, or when downstream users report missing or duplicated data.

At the end of each monthly or quarterly review, update three things: the saved fixtures, the expected metrics, and the documented stop conditions. Remove obsolete selectors and record why replacements were made. If the scraper has grown beyond a single script, document its inputs, outputs, retry behavior, and alert thresholds so another developer can diagnose it without reconstructing the system.

Use this practical sequence for the next failure:

1. Save the failing URL, response, logs, and timestamp.
2. Confirm that the response is the intended document.
3. Compare raw HTML with the rendered browser view.
4. Test selectors against a saved fixture and a fresh page.
5. Trace pagination using visited URLs and explicit stop reasons.
6. Check rate-limit behavior, retries, and timeout settings.
7. Validate record counts, required fields, duplicates, and export output.
8. Add a regression test for the failure before deploying the fix.

A scraper that is measured at each layer is easier to repair and safer to operate. Treat the article as a recurring maintenance checklist, not just a one-time debugging reference. For larger internal workflows, continue with How to Build a Web Scraping API for Internal Teams and the data cleaning checklist for web scraping pipelines.