PythonIntermediate

Scrapy

Fast, batteries-included web scraping and crawling framework for Python.

scrapingcrawlingspiderasyncpipelines

The Problem

requests + BeautifulSoup works for one page, but you need to scrape 10,000 product pages, follow pagination, handle retries, avoid getting banned, store results to a database, and run it on a schedule. Doing all that by hand is a full project in itself.

Mental Model

Scrapy is a framework, not a library. You define Spiders (what to crawl and how to parse), Items (what data to extract), and Pipelines (what to do with it). Scrapy's async engine handles concurrency, retries, politeness delays, and deduplication — you just write the parsing logic.

60-second start

Build a basic spider

scrapy startproject myproject
cd myproject
scrapy genspider quotes quotes.toscrape.com
import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = ['https://quotes.toscrape.com']

    def parse(self, response):
        for q in response.css('div.quote'):
            yield {
                'text': q.css('span.text::text').get(),
                'author': q.css('small.author::text').get(),
            }
        next_page = response.css('li.next a::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)
scrapy crawl quotes -o quotes.json

Use it when

Large-scale crawling, production spiders that run regularly, structured data extraction with pipelines (to DB, CSV, JSON), sites with pagination or login.

Skip it when

Single-page quick scrapes (use requests+BS4). JavaScript-heavy sites (use Playwright; or scrapy-playwright to integrate). When a proper API exists.

Alternatives

requests + BS4Simpler for one-off scrapes, but no concurrency, retries, or pipelines built in.

playwrightRequired for JS-rendered pages; can be integrated with Scrapy via scrapy-playwright.

apifyManaged scraping platform — handles infrastructure, proxies, and scheduling for you.

Ready to explore the full API?

See every class, function, and relationship in an interactive graph.