Rank #06 · Daily · DevTools
Scrapling: Batteries-Included Python Scraping & Crawling for Agent/Data Pipelines
One Python library for modern scraping: fast HTTP, stealth browsers, spiders, exports, plus MCP/CLI for AI agents.
What it does
Scrapling positions itself as a full-stack web scraping and crawling framework that covers the path from a single request to a site-scale crawl:
- Fetch:
Fetcher/ sessions with TLS impersonation, stealthy headers, optional HTTP/3;StealthyFetcher/ Playwright Chromium or Chrome with Cloudflare Turnstile helpers, proxy rotation, ad/domain blocking, DoH, CDP remote browsers, and background API capture (record_requests). - Parse: CSS/XPath, BeautifulSoup-style finders, text/regex search, adaptive element tracking, similar-element matching, chained navigation,
page.markdown()for LLM/RAG-ready output. - Crawl: Scrapy-like
SpiderAPI with concurrency, multi-session routing, pause/resume (crawldir), AutoThrottle, robots.txt, streaming, templates (CrawlSpider,SitemapSpider, feed spiders,ShopifySpider), and JSON/JSONL/CSV/XML exporters. - Agent/CLI: MCP server (
scrapling[ai]), Agent Skill docs, interactive shell, andscrapling extractone-liners.
Docs: scrapling.readthedocs.io. Install splits parser vs fetchers—pip install scrapling alone is parser-only; add pip install "scrapling[fetchers]" and scrapling install for browsers. Docker images ship with browsers. Python ≥3.10. BSD-3-Clause.
Who it’s for
Python scrapers who want one toolkit instead of stitching Requests + BeautifulSoup + Scrapy + Playwright; and teams feeding agent/data pipelines that need Markdown corpora or MCP browse/extract tools. Respect site ToS and robots.txt—the README disclaimer is explicit educational/research framing.
Stars / language / license
~92,315 stars · Python · BSD-3-Clause
Data verified: 2026-11-02 ~22:16 Asia/Shanghai (GitHub API get_repository + README get_file_contents)
Trending note (selection ~22:14 Asia/Shanghai): ~+892 stars today (live Trending UI; late-day climb on Daily)
Category tags
DevTools · web-scraping · crawling · Python · MCP · data-pipelines · stealth-browser
How it differs from similar projects
Versus Scrapy alone, Scrapling bundles stealth browser fetchers, adaptive parsing, and AI/MCP surfaces in one package. Versus Playwright scripts, it adds spider orchestration, exports, and a Scrapy-familiar parse API (including a drop-in scrapling_response decorator path). Versus thin AI “scrape a URL” wrappers, it is a full crawler framework with pause/resume and templates that agents can call through MCP. Parser speed / similarity tables in the README are project benchmarks—use as orientation, not TidyHow verification. The late-day Rising climb fits a broader pattern: agent stacks still need reliable, programmable data acquisition—not only chat UIs.