Scrapy Python Library Cheatsheet

Web scraping is a powerful technique used to extract data from websites, and Scrapy is one of the most popular frameworks for Python developers to perform this task efficiently. With its flexibility and scalability, Scrapy simplifies the process of gathering data from the web. This cheatsheet serves as a quick reference guide for Scrapy, providing key commands and tips to help you navigate through the web scraping journey.

Installation

Before diving into Scrapy, make sure you have it installed. Use the following command to install Scrapy using pip:

pip install scrapy

Creating a Scrapy Project

To start a new Scrapy project, run the following commands in your terminal:

scrapy startproject project_name
cd project_name

This creates a basic project structure with essential files and directories.

Writing a Spider

Spiders are the core components of Scrapy responsible for defining how to navigate a website and extract data. Create a spider using the following command:

scrapy genspider spider_name example.com

This generates a spider template in the spiders directory. Open the spider file and start coding.

Extracting Data

Use selectors to extract data from the HTML response. Scrapy supports CSS selectors and XPath expressions. Here’s an example:

# Using CSS Selector
title = response.css('h1::text').get()

# Using XPath
author = response.xpath('//div[@class="author"]/text()').get()

Crawling Multiple Pages

To crawl multiple pages, define a callback method to handle the next requests. Use the yield keyword to create a new request:

def parse(self, response):
    # Extract data from the current page
    # ...

    # Follow links to the next page
    next_page = response.css('a.next_page::attr(href)').get()
    if next_page:
        yield scrapy.Request(url=next_page, callback=self.parse)

Handling Pagination

To handle pagination, use the start_requests method to generate initial requests for each page:

def start_requests(self):
    urls = ['http://example.com/page1', 'http://example.com/page2']
    for url in urls:
        yield scrapy.Request(url=url, callback=self.parse)

Item Pipelines

Item pipelines are used to process scraped data. Define pipelines in the settings.py file and enable them:

# settings.py
ITEM_PIPELINES = {
    'myproject.pipelines.MyPipeline': 300,
}

Running the Spider

Execute the spider using the following command:

scrapy crawl spider_name

To export data to various formats like JSON or CSV, use the -o option:

scrapy crawl spider_name -o output.json

Handling Dynamic Content

For pages with dynamic content loaded via JavaScript, consider using the scrapy-selenium middleware to interact with the page.

pip install scrapy-selenium

Configure SELENIUM_DRIVER_EXECUTABLE_PATH in settings.py and add the middleware:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800,
}

SELENIUM_DRIVER_EXECUTABLE_PATH = 'path/to/chromedriver'

Scrapy is a robust framework that streamlines the web scraping process. This cheatsheet provides a quick reference for common tasks, but Scrapy offers much more functionality. Explore the official documentation for in-depth information and advanced features to supercharge your web scraping projects.

FAQ