How to Extract JSON-LD and Embedded JSON from Web Pages

Overview

There are several different meanings of “JSON on a web page.” A page may include JSON-LD for search engines, a serialized state object used by a frontend application, a configuration object inside a JavaScript block, or a payload embedded in a custom data attribute. Each format needs a slightly different extraction strategy.

The most reliable workflow is to first download or render the page, inspect its HTML, locate likely containers, extract the raw text, and then parse or validate the result as JSON. Avoid treating every script tag as JSON. Many scripts contain ordinary JavaScript, templates, tracking code, or text that only resembles a JSON object.

This distinction matters when you scrape JSON from a webpage for monitoring, research, testing, or an internal data pipeline. A scraper that relies on one fragile selector may work for a week and then fail after a frontend change. A parser that validates candidates and records failures is easier to maintain.

Core framework

1. Retrieve the HTML you actually need

Start with a normal HTTP request when the required data is present in the server response. Use a browser such as Playwright when the page creates the data only after JavaScript runs. This is the same decision used for other scraping tasks: compare the initial response with the DOM after rendering rather than assuming that every modern page requires a browser.

For static HTML, a small Python setup is enough:

import requests

url = "https://example.com/page"
response = requests.get(
    url,
    headers={"User-Agent": "Mozilla/5.0"},
    timeout=20,
)
response.raise_for_status()
html = response.text

Use an appropriate user agent, reasonable timeouts, retries where appropriate, and a request rate that does not place unnecessary load on the site. For broader reliability practices, see How to Build Reliable Web Scrapers.

2. Identify the data container

Inspect the source HTML, not only the browser’s visual output. Common locations include:

• <script type="application/ld+json"> for JSON-LD structured data.
• Script elements with framework-specific state or configuration.
• Custom attributes such as data-state or data-config.
• Inline JavaScript assignments such as window.__INITIAL_STATE__ = {...}.
• HTML comments, hidden elements, or network responses loaded after page initialization.

Prefer semantic and stable signals, such as the JSON-LD type attribute or a known state variable, over “the third script on the page.” When no stable marker exists, collect candidates and validate them instead of assuming a fixed position.

3. Extract, normalize, and parse

JSON is strict. Property names and string values must use double quotes, trailing commas are not allowed, and JavaScript values such as undefined or functions are not valid JSON. Extract the text first, remove only surrounding whitespace, and pass it to a JSON parser.

A useful result model includes the source URL, container type, raw text when permitted, parsed value, and an error message if parsing fails. This makes troubleshooting possible without silently discarding data.

Practical examples

Extract JSON-LD with BeautifulSoup

JSON-LD is usually the simplest case because its script element has a recognizable type. A page can contain one block or several, and a block may contain an object or an array.

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
json_ld_items = []

for script in soup.select('script[type="application/ld+json"]'):
    raw = script.string or script.get_text()
    raw = raw.strip()
    if not raw:
        continue

    try:
        json_ld_items.append(json.loads(raw))
    except json.JSONDecodeError as error:
        print(f"Invalid JSON-LD: {error}")

print(json_ld_items)

Do not assume that the first JSON-LD object is the only one or that it represents the page’s main entity. Inspect each parsed item. A block can contain an array, an object with an @graph property, or a type you do not need. Filter after parsing:

def objects_with_type(value, wanted_type):
    if isinstance(value, dict):
        if value.get("@type") == wanted_type:
            yield value
        for item in value.get("@graph", []):
            if isinstance(item, dict) and item.get("@type") == wanted_type:
                yield item
    elif isinstance(value, list):
        for item in value:
            yield from objects_with_type(item, wanted_type)

articles = [
    item
    for block in json_ld_items
    for item in objects_with_type(block, "Article")
]

Parse embedded JSON from a script tag

Some applications place a complete JSON object in a script tag with an identifier or a nonstandard type. If the script body is pure JSON, parsing it is straightforward:

script = soup.find("script", id="initial-state")
if script is None:
    raise ValueError("Initial state script was not found")

raw_state = script.get_text(strip=True)
state = json.loads(raw_state)
print(state.get("items", []))

If the script contains an assignment such as window.__INITIAL_STATE__ = {...};, it is not pure JSON. A regular expression can help locate a marker, but it should not be used as a full JavaScript parser. Braces can appear inside strings, nested objects can contain similar text, and escaped characters make naive patterns unreliable. Where possible, find an official page-specific endpoint or use a browser to observe the network response. If the object is known to be a JavaScript literal rather than strict JSON, use a parser designed for that syntax instead of applying unsafe evaluation.

Decode escaped JSON

Embedded data may be HTML-escaped or JSON-encoded more than once. For example, a data attribute can contain entities such as ", while a string may contain escaped quotes such as '. BeautifulSoup decodes normal HTML entities in attributes, but the remaining value may still need JSON parsing:

element = soup.select_one("[data-state]")
if element:
    raw = element["data-state"]
    try:
        state = json.loads(raw)
    except json.JSONDecodeError as error:
        print("The attribute is not valid JSON:", error)

When a value has been serialized twice, parse it in stages only when the type at each stage is clear:

value = json.loads(raw)
if isinstance(value, str):
    value = json.loads(value)

Log the representation and length of failed values during development, but be careful with tokens, personal data, and other sensitive content.

Find data after JavaScript rendering

When the initial response does not contain the required script or state object, render the page with Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as browser_api:
    browser = browser_api.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/page", wait_until="networkidle")

    rendered_html = page.content()
    browser.close()

soup = BeautifulSoup(rendered_html, "html.parser")

Rendering the DOM can reveal elements created by JavaScript, but it may still be better to capture the underlying response when the data comes from an API. A response is often cleaner and less coupled to the page’s markup. For tool selection, compare Playwright, Puppeteer, and Selenium, or review Playwright versus BeautifulSoup.

Common mistakes

Confusing page source with the live DOM

Requests returns the server response. A browser inspector may show a DOM modified after scripts run. If a value appears in one but not the other, determine when it is created before changing selectors.

Assuming every script is JSON

Only parse a script as JSON when its contents are expected to be JSON. Catch JSONDecodeError, preserve enough context to diagnose the failure, and continue when one unrelated block is invalid.

Using brittle regular expressions

Regular expressions are useful for locating a marker, but nested structured data is difficult to extract safely with a simple pattern. Prefer DOM selection, a known endpoint, or a syntax-aware parser.

Ignoring arrays and multiple blocks

Return all relevant JSON-LD blocks unless the page contract explicitly guarantees one. Validate the resulting shape before reading fields such as name, url, or offers.

Skipping validation and change detection

Check required keys, expected types, and acceptable empty values. Store a small fixture from a permitted test page and run the parser against it during changes. This catches markup changes before they affect downstream data.

Overlooking access and data-handling boundaries

Before automating, review the site’s terms, access controls, and applicable requirements. Keep collection narrow, identify your client where appropriate, rate-limit requests, and avoid collecting data that the workflow does not need.

When to revisit

Revisit this extraction logic whenever the site changes its frontend framework, structured-data templates, script identifiers, rendering sequence, or API responses. Also review it when a parser begins returning empty arrays, malformed JSON, unexpected types, or a sudden drop in records.

A practical maintenance routine is to keep separate tests for JSON-LD, pure script-tag JSON, escaped attribute JSON, and browser-rendered state. Record the selector or endpoint used for each source and add a clear failure message when it disappears. If several pages share a template, test representative page types rather than relying on one URL.

For the next implementation, begin with a static request, inspect the raw HTML, and extract JSON-LD with a type selector. Add validation and logging before adding browser automation. If the data is created only after load, capture the relevant rendered content or network response. Finally, connect the parser to the reliability practices in this web scraping troubleshooting guide and export only validated records. This sequence keeps the code understandable while giving you a clear path when the underlying page changes.