Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 61k 11.4k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 578

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.3k 159

  4. w3lib w3lib Public

    Python library of web-related functions

    Python 418 107

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 85 29

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 68 11

Repositories

Showing 10 of 29 repositories
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy’s past year of commit activity
    Python 60,986 BSD-3-Clause 11,400 453 (18 issues need help) 198 Updated Mar 27, 2026
  • scrapyd Public

    A service daemon to run Scrapy spiders

    scrapy/scrapyd’s past year of commit activity
    Python 3,088 BSD-3-Clause 578 6 0 Updated Mar 27, 2026
  • sphinx-scrapy Public

    Sphinx extension for documentation in the Scrapy ecosystem

    scrapy/sphinx-scrapy’s past year of commit activity
    Python 1 BSD-3-Clause 1 0 1 Updated Mar 26, 2026
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib’s past year of commit activity
    Python 418 BSD-3-Clause 107 10 5 Updated Mar 20, 2026
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter’s past year of commit activity
    Python 68 BSD-3-Clause 11 10 2 Updated Mar 16, 2026
  • scrapyd-client Public

    Command line client for Scrapyd server

    scrapy/scrapyd-client’s past year of commit activity
    Python 776 BSD-3-Clause 145 5 0 Updated Feb 27, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint’s past year of commit activity
    Python 21 MIT 4 42 (2 issues need help) 0 Updated Feb 25, 2026
  • itemloaders Public

    Library to populate items using XPath and CSS with a convenient API

    scrapy/itemloaders’s past year of commit activity
    Python 48 BSD-3-Clause 16 18 (1 issue needs help) 4 Updated Jan 29, 2026
  • queuelib Public

    Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

    scrapy/queuelib’s past year of commit activity
    Python 295 BSD-3-Clause 55 4 2 Updated Jan 29, 2026
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego’s past year of commit activity
    DIGITAL Command Language 85 BSD-3-Clause 29 7 (3 issues need help) 0 Updated Jan 29, 2026
View all repositories