Reliable selectors are the difference between a scraper that keeps working and one that silently returns empty or incorrect data after a redesign. This guide compares CSS, XPath, stable attributes, visible text, ARIA-based locators, and positional selectors, then turns the comparison into a practical review checklist for browser automation and HTML parsing projects.
Overview
A selector identifies the element that contains the data or action your scraper needs. The strongest selector is not necessarily the shortest one; it is the one that expresses a stable relationship between the target and the page.
In general, prefer selectors based on an element's purpose or an explicitly maintained attribute. Use layout details only when no better contract exists. A class such as product-card may be useful if it represents a durable component, while a generated class such as css-1a2b3c is a warning sign. A selector that depends on the fifth div inside a container is usually more vulnerable to change than one that identifies a product link by its stable attribute or accessible name.
| Strategy | Example | Best use | Main risk |
|---|---|---|---|
| Stable attribute | [data-testid="price"] | Explicit application hooks | The attribute may be removed or renamed |
| Semantic CSS | article.product a.details | Structured HTML and component classes | Class names can change with redesigns |
| XPath relationship | //article[.//h2[normalize-space()="Example"]]//a | Relationships and conditional matching | Long expressions become difficult to review |
| Visible text | get_by_text("Next") | Human-facing controls and labels | Copy, localization, or whitespace changes |
| ARIA role and name | get_by_role("button", name="Load more") | Interactive browser automation | Depends on correct accessible markup |
| Position | li:nth-child(3) | Last-resort, tightly controlled layouts | Insertion or reordering changes the target |
For static pages, CSS and XPath are commonly used with HTML parsers. For JavaScript-rendered pages, browser libraries such as Playwright offer role, label, text, CSS, and XPath locators. The same principle applies in both cases: select by meaning first, then by structure, and use position only when the position itself is the data.
For a syntax refresher, see the practical CSS selectors and XPath recipe guide. If the content does not exist in the initial HTML, investigate the page's network requests before adding increasingly complex selectors; the network API guide for JavaScript-rendered sites covers that workflow.
Checklist by scenario
1. Extracting repeated records
Start by locating the repeated unit: a product card, result row, article, or listing item. Select the parent collection with a stable class, attribute, or element type, then extract fields relative to each item rather than querying the entire document repeatedly.
for card in soup.select('article.product-card'):
title = card.select_one('h2.product-card__title')
price = card.select_one('[data-field="price"]')
link = card.select_one('a[href]')This pattern limits accidental matches. A price selector scoped to one card is safer than a global selector that happens to find the first price on the page. When no stable hook exists, combine modest structural clues, such as an article element containing a heading and a link, and validate the result.
2. Finding a link or button
For browser automation, prefer the role and accessible name when they are reliable:
page.get_by_role("button", name="Load more").click()
page.get_by_role("link", name="Product details").first.click()Use a label locator for form controls when the label is stable. Use visible text when the text is the actual business identifier, such as a category name or a pagination control. If the same label appears several times, narrow the search to a region or parent component instead of immediately adding a positional index.
3. Handling changing text
Exact text is useful for a stable identifier but fragile for descriptions, prices, dates, and localized interfaces. XPath can normalize surrounding whitespace:
//button[normalize-space()="Load more"]For text that changes predictably, match a stable portion only when that cannot create false positives. Better still, locate the containing record with a stable attribute and read the text as data rather than using it to identify the element.
4. Selecting by relationship
XPath is valuable when the target is defined by its relationship to another element. For example, a detail link belonging to the card whose heading contains a known title can be expressed as:
//article[.//h2[normalize-space()="Example product"]]//a[contains(@href, "/products/")]Keep relationship-based XPath readable. If an expression requires several unrelated ancestors, multiple indexes, and implementation-specific classes, it may be compensating for a missing stable hook.
5. Working with forms and repeated controls
Use a field's name, label, or stable identifier before relying on its position:
page.get_by_label("Email address").fill(email)
page.locator('input[name="email"]').fill(email)For repeated controls, scope the locator to the relevant row or card. If a button has no useful name, ask whether the page exposes a stable attribute such as data-action. When you control the application, adding a documented test or extraction hook is often more durable than reverse-engineering layout.
What to double-check
- Uniqueness: Does the selector match exactly one intended element in the normal page state? If it matches several elements, is that multiplicity expected?
- Scope: Is the query scoped to the relevant card, table row, dialog, or navigation region?
- Meaning: Does the selector describe what the element is, or only where it happens to be today?
- Rendered state: Are you testing the initial HTML, post-JavaScript DOM, expanded content, or a modal state? A selector can be correct for one state and absent in another.
- Optional elements: What happens when a badge, image, discount, or secondary link is missing?
- Pagination and duplication: Could hidden templates, mobile navigation, or duplicated responsive markup produce a second match?
- Encoding and whitespace: Are non-breaking spaces, line breaks, Unicode characters, or localized text affecting a text-based match?
- Output validation: Does the extracted value have the expected type and shape? A selector can return a value while still selecting the wrong element.
Record the selector's purpose beside the code. A comment such as “price inside each product card, maintained by the data-field attribute” is more useful than “find price.” For critical fields, log a small sample of the matched element or its surrounding text during tests, without storing unnecessary sensitive data.
Common mistakes
Relying on generated classes
Build systems and CSS-in-JS tools may produce class names that are not intended as public interfaces. They can still be useful as one part of a selector, but avoid making a generated token the only anchor when a semantic element or stable attribute is available.
Using long absolute XPath
An expression such as /html/body/div[2]/main/div[1]/section/div[3] describes a current DOM path, not the target's purpose. A wrapper, advertisement, or navigation change can invalidate it. Rewrite it around a meaningful ancestor, attribute, heading, or relationship.
Overusing nth-child and nth()
Positional selectors are appropriate when order is explicitly meaningful, such as selecting the first page of a controlled table for a test. They are poor defaults for product cards, search results, or navigation items because insertion and reordering are normal.
Assuming ARIA is decorative
ARIA roles and accessible names can make excellent browser locators, but only when the page exposes them consistently. Inspect the accessibility tree and test states such as disabled, expanded, and dialog-open. Do not infer that a visual label is automatically the accessible name.
Testing only the happy path
A selector may pass on a fully populated page and fail when there are zero results, a missing image, an authenticated state, a consent dialog, or a pagination boundary. Include these states in fixture pages and assert both presence and absence where appropriate.
When to revisit
Review selectors before deploying a new scraper, after a target site's redesign, and whenever monitoring reports empty pages, sudden field-level nulls, duplicate records, or unusual value distributions. Also revisit them when your browser library, parser, frontend framework, or rendering workflow changes. These changes can alter the DOM or the accessible tree even when the page looks similar.
Use this short maintenance routine:
- Save representative HTML or rendered-page fixtures for normal, empty, and edge-case states.
- Run every selector against those fixtures and report zero-match, multi-match, and type-validation failures separately.
- Inspect the changed markup and identify the new stable contract before editing the selector.
- Prefer a small, readable replacement over adding another positional exception.
- Deploy with monitoring that distinguishes an unavailable page from a successful page with no records.
- Document why the final selector was chosen and what change would require another review.
When a selector fails repeatedly, consider whether selectors are the right extraction boundary. Some JavaScript-rendered pages expose structured data or network responses that are more stable than the rendered layout. For downstream reliability, pair selector tests with the web scraper monitoring runbook, and validate the resulting records using an appropriate output format such as JSONL, CSV, or SQLite.
The practical rule is simple: anchor each extraction to the most stable meaning available, scope it narrowly, validate what it returns, and record the assumptions that make it reliable. That process makes CSS versus XPath a secondary implementation choice rather than the source of fragility.