HTML Table Scraping in Python: pandas, Beautiful Soup, lxml, and Selectolax Compared
Web ScrapingPythonHTML TablesPandasBeautiful SouplxmlSelectolax

HTML Table Scraping in Python: pandas, Beautiful Soup, lxml, and Selectolax Compared

SScraper.page Editorial Team
2026-11-17
7 min read

Compare pandas, Beautiful Soup, lxml, and Selectolax for reliable Python HTML table extraction, including spans, headers, malformed markup, and validation.

For most projects, pandas.read_html() is the fastest way to turn ordinary HTML tables into data frames, while Beautiful Soup, lxml, and Selectolax provide more control when the markup is irregular or the table must be selected precisely. This guide compares the four approaches, explains how they handle headers, spans, nested tables, hidden content, data types, and output formats, and gives a validation workflow for dependable Python table extraction.

Overview

The best way to scrape an HTML table in Python depends on what “best” means for the task. If the source contains conventional <table> elements and the desired result is a DataFrame, pandas is usually the most direct starting point. If the page contains several similar tables, nested markup, custom row selection, or inconsistent cells, a lower-level parser can make the extraction logic easier to inspect and maintain.

These libraries solve related but different problems:

  • pandas: detects HTML tables and converts them into DataFrames, with useful handling for headers, missing cells, and basic type inference.
  • Beautiful Soup: provides a readable tree API for locating and traversing HTML, but requires you to write the table-to-record conversion yourself.
  • lxml: offers fast HTML parsing and XPath support, making it useful when selectors and malformed markup need close control.
  • Selectolax: is a lightweight tree parser suited to high-volume extraction and CSS-style selection, but it also requires custom table normalization.

None of these tools renders JavaScript. A table created only after a browser executes scripts may not appear in the downloaded HTML. In that case, inspect the page’s network requests or use an appropriate rendering workflow before choosing a parser. The guide How to Scrape JavaScript-Rendered Websites by Finding Their Network APIs covers that distinction in more detail.

How to compare options

Compare table extraction libraries against the actual failure modes in your source pages rather than against a single speed test. First save representative HTML samples: include a normal table, a table with missing cells, one using rowspan or colspan, and a page containing unrelated tables. Then measure whether each approach returns the right table, the expected row count, stable column names, and the intended data types.

The following summary is a practical starting point. “Strong” means the library provides a useful built-in path or mature primitives; it does not mean that every page will be handled automatically.

OptionNested tablesRowspan and colspanMalformed markupData typesTypical output
pandas.read_htmlCan be ambiguousOften convenient, verify resultsParser-dependentAutomatic inference with cleanupDataFrame
Beautiful SoupEasy to inspect and filterCustom implementationDepends on parser backendCustom conversionLists, dictionaries, or DataFrame
lxmlPrecise with XPathCustom normalizationUseful recovery optionsCustom conversionElement trees or custom records
SelectolaxPrecise with selectorsCustom normalizationDesigned for practical HTML parsingCustom conversionElement nodes or custom records

Also compare operational concerns: installation constraints, parser dependencies, memory usage, how easily the code can be tested, and whether the output needs to remain a DataFrame or should be written as CSV, JSON Lines, or SQLite. Output format affects downstream reliability; the comparison in JSONL vs CSV vs SQLite for Scraped Data can help with that decision.

Feature-by-feature breakdown

pandas: the shortest path to a DataFrame

For a conventional table, start with pandas:

import pandas as pd

frames = pd.read_html(html)
for index, frame in enumerate(frames):
    print(index, frame.shape)
    print(frame.head())

The function can discover multiple tables, so do not assume the first result is correct. Use table count, expected column names, captions, or a distinctive cell value to select the result. After selection, normalize whitespace and column labels, then explicitly convert important fields instead of trusting inference alone.

pandas is convenient for header rows and many ordinary span patterns, but a visually correct table can still produce surprising multi-level column labels or shifted values. Treat rowspan, colspan, footnotes, and nested tables as cases that require inspection. A DataFrame is not proof that the extraction is correct.

Beautiful Soup: readable, flexible traversal

Beautiful Soup is a good fit when selection logic is more important than parser speed. It lets you find a table by an ID, class, caption, or nearby heading and then traverse rows and cells explicitly:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
table = soup.select_one("table#results")

records = []
for row in table.select("tr"):
    cells = [cell.get_text(" ", strip=True)
             for cell in row.select("th, td")]
    if cells:
        records.append(cells)

This approach makes it easy to skip layout rows or identify a header based on its text. Its trade-off is that you must decide how to handle uneven row lengths, repeated headers, spans, and nested tables. If you need a rectangular DataFrame, normalize the records before constructing one. You can also choose a different parsing backend where the input requires it, but test that choice against saved fixtures.

lxml: XPath and controlled extraction

lxml is useful when the table has stable structural cues or when CSS selection is not expressive enough. XPath can target a table by its attributes, its caption, or a relationship to a heading:

from lxml import html

root = html.fromstring(page_html)
tables = root.xpath("//table[@id='results']")

rows = tables[0].xpath(".//tr")
records = [
    [" ".join(cell.itertext()).strip()
     for cell in row.xpath("./th | ./td")]
    for row in rows
]

Be careful with XPath scope. Using .//tr can include rows from a nested table; selecting direct children or filtering nested table descendants may be necessary. lxml gives you precise primitives, not automatic semantic table reconstruction, so span expansion and header mapping remain application logic.

Selectolax: lightweight selection for repeated extraction

Selectolax can be attractive when a scraper processes many pages and the required logic is mostly CSS selection and text extraction. Its API is compact, but table conversion is still a custom step:

from selectolax.parser import HTMLParser

parser = HTMLParser(page_html)
table = parser.css_first("table#results")
records = []

for row in table.css("tr"):
    cells = [cell.text(strip=True) for cell in row.css("th, td")]
    if cells:
        records.append(cells)

Benchmarking should use your own documents. Parser speed is only one part of total runtime; downloading pages, selecting the right table, normalizing spans, converting types, and writing output may dominate the process.

Headers, spans, hidden tables, and malformed HTML

Headers may be stored in <th> elements, the first row, multiple header rows, or a caption outside the table. Define the expected schema explicitly when the data feeds another system. For rowspan and colspan, preserve the original HTML as a fixture and verify a few known coordinates after extraction. If semantic relationships use the headers attribute, a simple “first row is the header” rule may lose information.

A CSS rule such as display:none does not necessarily prevent a parser from seeing source HTML. Conversely, content inserted by JavaScript will not be available in a plain HTTP response. Distinguish source-hidden content from browser-rendered content before debugging the parser. Malformed markup should be tested with the same parser and options used in production; recovery behavior can differ between parser backends.

Best fit by scenario

  • One or several ordinary tables: begin with pandas.read_html(), inspect every returned frame, and add schema checks.
  • A page with many similar tables: use Beautiful Soup, lxml, or Selectolax to identify the table by a distinctive attribute, heading, or caption before converting it.
  • Complex XPath relationships: choose lxml when table selection depends on document structure.
  • High-volume, selector-driven extraction: evaluate Selectolax with a benchmark based on real pages and the complete pipeline.
  • Heavy irregularity or span-heavy layouts: use a lower-level parser and write explicit normalization logic rather than hiding assumptions inside a convenience call.
  • Data that is not in the initial HTML: locate the underlying request or use a rendering strategy; changing table parsers will not create missing content.

Regardless of library, keep selection separate from normalization. First identify the intended table, then convert rows into records, then clean values and validate the schema. This separation makes redesigns easier to diagnose. For selector design, see Resilient Web Scraper Selectors: CSS, XPath, Text, and ARIA Strategies.

When to revisit

Revisit this comparison when a source site redesign changes table IDs, nesting, header rows, or the way content is rendered. Also review it during Python dependency upgrades, when a parser backend changes, or when a pipeline begins processing substantially larger pages. New parser options may alter the trade-off between convenience and control, but a package update should not be considered successful until saved HTML fixtures still produce the expected records.

Use a small validation checklist after every change:

  1. Confirm that the target table was found and that its identifying marker is still present.
  2. Compare extracted row and column counts with an expected range.
  3. Check required column names and several known cell values.
  4. Look for duplicate headers, unexpected nested-table rows, shifted span values, and empty results.
  5. Validate numeric, date, and missing-value conversions before writing output.
  6. Log the source URL, retrieval time, parser path, and validation result.

If a scraper starts returning empty tables or plausible-looking but incorrect data, treat that as a monitoring problem rather than silently accepting the output. The Web Scraper Monitoring Runbook provides a broader failure-detection workflow. For the HTTP layer that supplies the HTML, compare clients using Requests vs HTTPX vs aiohttp for Web Scraping if needed.

In short, choose pandas for a clean table-to-DataFrame path, and choose Beautiful Soup, lxml, or Selectolax when selection and normalization need to be explicit. The dependable choice is the one that passes your fixtures, schema checks, and row-count validation—not necessarily the one with the shortest code.

Related Topics

#Web Scraping#Python#HTML Tables#Pandas#Beautiful Soup#lxml#Selectolax
S

Scraper.page Editorial Team

Developer Resources Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.