Reliable Web Scraping Requests: Rate Limits, Retries, Backoff, Timeouts, and Polite Crawling
Web ScrapingRate LimitingRetriesHTTPAutomation

Reliable Web Scraping Requests: Rate Limits, Retries, Backoff, Timeouts, and Polite Crawling

SScraper.page Editorial Team
2026-11-21
7 min read

Estimate scraper runtime, handle HTTP 429 and 503 responses, apply backoff, set timeouts, and build resumable polite crawling jobs.

Reliable scraping starts with controlling requests, not simply sending more of them. This guide shows how to estimate crawl duration, configure pacing and concurrency, distinguish retryable failures from permanent ones, and build a resumable workflow that responds politely to rate limits, timeouts, and temporary outages.

Overview

A web scraper can fail even when its selectors and parsing logic are correct. Targets may return HTTP 429 responses when request volume is too high, HTTP 503 responses during service interruptions, connection errors when a network path is unstable, or successful HTML responses that contain a block page instead of the expected content. A resilient workflow treats request delivery as its own engineering problem.

The goal is not to retry every failed request indefinitely. The goal is to make a measured decision for each response:

  • Proceed: accept the response when its status, content type, and basic structure are expected.
  • Wait and retry: use controlled backoff for temporary failures such as rate limiting, server errors, or transient network exceptions.
  • Skip or quarantine: record permanent failures, invalid URLs, authentication problems, and content that requires a different workflow.
  • Stop safely: pause the job when failures indicate that request pressure, credentials, or target behavior needs review.

Before starting, review the target's published access guidance and robots.txt where applicable. Treat this review as one input to your operating policy, not as a substitute for reasonable pacing, access controls, or site-specific terms. Scrape only data you are permitted to access, avoid bypassing authentication or technical restrictions, and keep an audit trail of the decisions your job makes.

For adjacent reliability concerns, see the guides to monitoring empty pages and silent scraper failures and testing scrapers with fixtures and regression checks.

How to estimate

Start with a small planning model. Let:

  • N = number of planned URLs or requests
  • r = average permitted request rate per worker, in requests per second
  • W = number of concurrent workers
  • m = expected average attempts per URL, including retries
  • T = average response and processing time per attempt, in seconds

If a worker is limited by its delay between requests, an initial duration estimate is:

estimated seconds ≈ (N × m) ÷ (r × W)

If response time is the limiting factor, use a second estimate:

estimated seconds ≈ (N × m × T) ÷ W

Use the more conservative result, then add time for backoff, connection setup, parsing, checkpoint writes, and failed requests. These formulas are planning tools rather than guarantees. A target may apply separate limits by IP address, account, endpoint, or time window, and concurrency does not always increase useful throughput.

A practical planning example: suppose a job has 12,000 URLs, one worker averages one request every two seconds, and you allow four workers. With no retries, the pacing estimate is approximately 12,000 ÷ (0.5 × 4), or 6,000 seconds. If your test run suggests an average of 1.1 attempts per URL, use 13,200 attempts instead, producing roughly 6,600 seconds before additional backoff and processing overhead. Round the schedule upward and validate it with a small batch before committing to the full run.

Measure the first few hundred requests rather than guessing. Record status-code counts, median and high-percentile latency, retry counts, response sizes, and the number of valid records extracted. If error rates rise as concurrency increases, reduce concurrency even if the nominal request rate appears acceptable.

Inputs and assumptions

Request pacing

Use a delay between requests rather than relying only on a global worker count. A fixed delay is easy to explain and audit. A token-bucket or leaky-bucket limiter is useful when you need a defined average rate with a controlled burst size. Start conservatively, then adjust based on observed responses. Do not treat a successful test batch as proof that unlimited parallelism is safe.

Concurrency

Set separate limits for total workers and per-host workers. A scraper that visits several domains should not let activity on one host determine pressure on another. Keep connection pools aligned with the worker limit; a large pool can quietly create more simultaneous activity than intended.

Timeout budgets

Use distinct connection and read timeouts when the HTTP client supports them. A connection timeout covers establishing a route, while a read timeout covers waiting for response data. Set a total request budget as well when possible. Without a cap, a small number of stalled requests can occupy all workers and make the job appear frozen.

Retry classification

Retry temporary conditions such as HTTP 429, selected 5xx responses, connection resets, and short-lived DNS or transport errors. Handle HTTP 400, 401, 403, and 404 carefully: these often require correcting the URL, credentials, permissions, or extraction policy rather than repeating the same request. A 403 may have several meanings, so record it and inspect the target's guidance instead of assuming that retries are appropriate.

Backoff and Retry-After

Use exponential backoff with jitter so workers do not retry in a synchronized wave. One general form is:

wait = min(max_delay, base_delay × 2^attempt) + random_jitter

Honor a valid Retry-After value when the server provides one, subject to a maximum safety limit and your job's deadline. Keep the retry count finite. After the limit, place the URL in a failure queue with the final status, exception, attempt count, and timestamps.

Checkpointing

Persist progress as the job runs. A checkpoint can contain the URL or stable item ID, processing state, attempt count, last status, next eligible retry time, and extracted-record checksum. Write checkpoints atomically so an interrupted process does not mark unfinished work as complete. This makes a job resumable instead of forcing a full restart.

Worked examples

Example 1: A rate-limited listing crawl

A listing site returns 200 responses during a small test but begins returning 429 after concurrency rises from two to eight workers. Keep the request limiter, reduce per-host concurrency to two, and treat 429 as retryable. Read and honor Retry-After when present; otherwise use exponential backoff with jitter. Do not immediately add more proxies or increase the rate. First establish whether the lower-pressure configuration completes a representative batch with stable extraction results.

For pagination, persist the next URL or cursor only after the current page has been parsed and validated. This prevents a restart from skipping a page. The guide to pagination, cursors, infinite scroll, and deduplication covers the corresponding state-management patterns.

Example 2: Temporary server failures

A product crawl receives several 503 responses, followed by successful responses when requests are spaced farther apart. Classify 503 as retryable within a finite limit, apply backoff, and record the recovery time. If failures continue across multiple intervals, pause the affected host and alert an operator rather than consuming the entire retry budget. The final report should distinguish recovered retries from URLs that remained unsuccessful.

Example 3: A slow authenticated workflow

An authenticated site requires cookies or a session token, and some pages take longer to respond. Keep authentication state separate from retry state: a timeout should not automatically trigger a new login, and a 401 should not be retried forever with the same credentials. Validate the session before resuming, rotate no credentials without an approved process, and store secrets outside logs. See authenticated scraping session patterns for a broader treatment.

Minimal control-flow template

for item in queue:
    wait_until_allowed(item.host)
    for attempt in range(max_attempts):
        try:
            response = fetch(item, connect_timeout, read_timeout)
            if response.status in retryable_statuses:
                sleep(retry_after_or_backoff(response, attempt))
                continue
            if response.status in permanent_statuses:
                record_failure(item, response.status)
                break
            validate_content(response)
            save_result_and_checkpoint(item, response)
            break
        except transient_network_error as error:
            sleep(backoff_with_jitter(attempt))
    else:
        record_failure(item, "retry limit exceeded")

Adapt the names and exception classes to your HTTP client. Requests, HTTPX, and aiohttp expose different retry and timeout APIs, so consult the client-specific implementation before copying configuration directly; their trade-offs are compared in this HTTP client guide.

When to recalculate

Recalculate the schedule and retry assumptions whenever the inputs change. Review them before increasing URL volume, adding workers, changing endpoints, introducing authentication, or moving to a new network environment. Revisit the model when latency, response size, status-code distribution, or extraction yield changes materially.

Use a short canary batch after a long pause or a target-site redesign. Compare its request rate, successful-record rate, 429 and 5xx counts, timeout frequency, and average attempts per URL with the previous baseline. A change in pricing, proxy capacity, or infrastructure cost also warrants a fresh estimate if those inputs affect the job's operating decision.

Keep a failure-triage checklist with every run:

  1. Confirm the URL and response are in scope for the job.
  2. Check robots guidance, access permissions, and authentication state.
  3. Separate 429, 5xx, timeouts, connection errors, and permanent client errors.
  4. Inspect whether response bodies contain expected content rather than a block or login page.
  5. Compare current concurrency and pacing with the last successful run.
  6. Resume from checkpoints and retry only the recorded eligible failures.
  7. Stop and review the policy when failures persist across backoff windows.

Reliable web scraping is a repeatable operating process: estimate demand, limit pressure, retry selectively, save progress, and measure outcomes. Recalibrating those inputs as conditions change keeps automation useful without turning transient problems into an endless request loop.

Related Topics

#Web Scraping#Rate Limiting#Retries#HTTP#Automation
S

Scraper.page Editorial Team

Developer Workflow Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.