PythonBeginner-friendly

beautifulsoup4

HTML and XML parsing library for extracting data from web pages with a simple Pythonic API.

scrapinghtmlparsingwebxml

The Problem

You've fetched HTML from a web page with requests. Now you're staring at a wall of tags, trying to extract product prices, article text, or table rows with regex. Regex breaks the moment the site adds a CSS class. You need a proper HTML tree parser.

Mental Model

BeautifulSoup turns an HTML string into a traversable tree. Every tag becomes an object with .text, .attrs, and child/parent navigation. You find elements by tag name, CSS class, id, or attribute — like jQuery selectors but in Python.

60-second start

Scrape article titles from a news page

import requests
from bs4 import BeautifulSoup

html = requests.get('https://news.ycombinator.com').text
soup = BeautifulSoup(html, 'html.parser')
titles = soup.select('span.titleline > a')
for t in titles:
    print(t.text, t['href'])

Use it when

Scraping static HTML pages, extracting data from email templates, parsing HTML in tests, processing XML feeds, cleaning up messy HTML.

Skip it when

JavaScript-rendered pages (use Playwright or Selenium). High-volume scraping at scale (use Scrapy). When the site has a proper API — always prefer the API.

Alternatives

lxmlMuch faster for large documents. BS4 can use lxml as its parser backend.

parselScrapy's selector library — XPath and CSS selectors, faster than BS4.

playwrightNeeded for JS-rendered pages; can extract from the live DOM.

Ready to explore the full API?

See every class, function, and relationship in an interactive graph.