How to Scrape JavaScript-Rendered Websites by Finding Their Network APIs

Overview

A browser does not usually “see” a JavaScript website as one document. It first downloads an HTML shell, then loads JavaScript bundles, requests data, and inserts that data into the page. If the information you need arrives through a predictable network request, you may be able to scrape the underlying response directly.

This approach can be faster and easier to test than rendering every page. It also produces structured records that are often simpler to store than scraped DOM text. However, it is not a shortcut around access controls, authentication requirements, or a site's terms. Only collect data you are permitted to access, avoid bypassing technical restrictions, and use conservative request rates.

The basic workflow is:

1. Open the target page in a normal browser.
2. Inspect requests in DevTools.
3. Find the request that supplies the visible data.
4. Record its URL, method, parameters, headers, and response shape.
5. Reproduce the request with an HTTP client.
6. Handle pagination, retries, and validation.
7. Compare the downloaded records with the rendered page.

If the page contains embedded JSON before any API call is made, check that option first. The guide to extracting JSON-LD and embedded JSON covers that useful alternative.

Checklist by scenario

Scenario 1: The page loads a visible list

Open DevTools with the page loaded and select the Network panel. Filter by Fetch/XHR, then reload the page. Interact with the page if necessary: change a filter, search for a term, or open a list view. Look for requests whose responses contain item names, IDs, prices, dates, or other values visible in the interface.

Click a likely request and record:

• Request URL and HTTP method.
• Query-string parameters or form data.
• Request and response headers that appear relevant.
• Status code and response content type.
• Whether the response contains a list, a nested object, or a cursor.

Use the response preview and raw response tabs to confirm that it is the source of the rendered values. A request named config, analytics, or session may be important to the application but unrelated to the records you want.

Scenario 2: The request is easy to reproduce

When the response is public and needs only ordinary parameters, start with a small Python test. Keep the first request narrow so that you can inspect failures clearly.

import requests

url = "https://example.com/api/items"
params = {"page": 1, "limit": 25, "query": "example"}

response = requests.get(url, params=params, timeout=30)
response.raise_for_status()
data = response.json()

print(response.url)
print(data.keys() if isinstance(data, dict) else type(data))

Do not assume that a successful HTTP status means the data is correct. Check for an expected top-level key and a representative field before processing a large result set.

Scenario 3: The site uses POST or GraphQL

Some interfaces send filters in a POST body rather than in the URL. In DevTools, inspect the Payload or Request tab and reproduce the same structure.

payload = {
    "filters": {"status": "active"},
    "page": 1,
    "pageSize": 25
}

response = requests.post(
    "https://example.com/api/search",
    json=payload,
    timeout=30
)
response.raise_for_status()
records = response.json()

For GraphQL, the request commonly includes a query document and a variables object. Copy the smallest working request, then remove variables one at a time to learn which inputs are required. Avoid copying opaque browser state into permanent code until you understand whether it expires and whether you are authorized to use it.

Scenario 4: The page uses infinite scroll

Scroll until new records appear and watch for the next request. Compare successive requests. A page number may increase, or the response may return a cursor such as nextCursor. Follow the site's actual pagination mechanism rather than guessing offsets.

For a cursor workflow, the loop should stop when the response has no next cursor, returns no new records, or reaches a deliberate limit. The article on pagination with next links, cursor APIs, and infinite scroll provides patterns for each case.

Scenario 5: The endpoint requires a browser session

If a request works only after a user signs in or completes an allowed application flow, treat it as authenticated data. Do not publish or hard-code session cookies, bearer tokens, or personal credentials. Prefer an official API, an application integration, or a controlled workflow with credentials stored outside source code. A browser can still be useful for diagnosis, but direct requests should not be used to evade an access boundary.

What to double-check

Headers and parameters

Use DevTools' “Copy as cURL” feature as a diagnostic starting point, not as a finished scraper. It may contain many incidental headers. Test which values are actually required by removing them in a controlled environment. A stable request generally needs fewer moving parts than a verbatim browser export.

Pay particular attention to:

• Accept and Content-Type when negotiating JSON.
• Locale, currency, sort, filter, and date-range parameters.
• Trailing slashes, URL encoding, and repeated query parameters.
• POST JSON structure, including nested arrays and null values.
• Response compression or content types that your client must handle.

Pagination and duplicates

Save a stable identifier for every record and track identifiers already seen. This protects against duplicate pages and makes reruns easier to compare. Check whether the API's reported total is authoritative or merely an estimate. For cursor APIs, store the cursor only as an implementation detail and validate that each request advances.

Response-to-page validation

Choose several records visible in the browser and locate the same values in the raw response. Compare fields that can expose a mismatch, such as title, identifier, status, timestamp, and detail URL. Also compare counts when the interface displays a count. If the response contains more data than the page shows, document which filters or permissions explain the difference.

For downstream storage, choose a format that preserves the response shape. The comparison of JSONL, CSV, and SQLite for scraped data can help when records contain nested fields or need incremental updates.

Reliability controls

Set a timeout, handle transient failures, and limit concurrency. Log the request parameters, status code, response size, and parsing result without logging secrets. Validate required fields before writing a record. For a broader reliability checklist, see selectors, retries, rate limits, and data validation.

Common mistakes

• Scraping the initial HTML only. An empty shell is not evidence that the data is unavailable; inspect subsequent requests.
• Choosing the wrong request. Several calls may run together. Confirm that the response contains the exact values rendered on screen.
• Copying every browser header. This creates brittle code and can accidentally expose credentials. Start with the minimum required request.
• Ignoring filters. A request may silently default to a different region, date range, sort order, or page size.
• Assuming page numbers are universal. Some endpoints use offsets, cursors, tokens, or opaque continuation links.
• Parsing without validation. A changed response can produce plausible but incomplete output. Check schemas, counts, and required fields.
• Rendering when a direct response is available. Headless browsers remain useful when JavaScript computes a request, when interaction is essential, or when the data is not exposed cleanly. They are not automatically the best first tool. Review the comparison of Playwright, Puppeteer, and Selenium when browser automation is necessary.

When to revisit

Recheck the workflow before a recurring collection run and whenever the site's frontend changes. A redesign may rename endpoints, alter response fields, move filters into a POST body, or replace offset pagination with cursors. Seasonal planning cycles are also a sensible review point if your process depends on dates, categories, regions, or changing inventory.

Use this short maintenance checklist:

1. Reload the page and confirm the request still appears under the same interaction.
2. Compare the current URL, method, parameters, and response schema with your saved notes.
3. Run a small sample and compare several records with the rendered page.
4. Test the first pagination step and the stopping condition.
5. Review logs for rising error rates, empty responses, duplicates, or missing fields.
6. Remove obsolete headers, selectors, and assumptions from the code.
7. Record the date of the check and the exact page or workflow tested.

If the network endpoint is no longer stable or the page requires genuine browser interaction, reassess the tool rather than forcing a direct-request approach. For HTML that still needs parsing after retrieval, compare BeautifulSoup, lxml, Selectolax, and Cheerio, or use the Python requests and BeautifulSoup workflow as a maintainable baseline.