Rank #03 · Weekly · DevTools
Scrapling: One Python Framework From Single Request to Full Crawl
Fetch, parse, and crawl with one library, then hand the results to an LLM pipeline or let an agent drive it over MCP.
Canonical digest: /daily/scrapling (also on Weekly board).
What it does
- Fetchers. A fast HTTP
Fetcherwith browser TLS impersonation.StealthyFetcherand Playwright-based browser fetchers handle JS-heavy or protected pages, with proxy rotation, session support, and optional Cloudflare Turnstile/Interstitial handling. - Parser. CSS/XPath selectors, BeautifulSoup-style finders, text/regex search, and adaptive element tracking that relocates elements after a site's layout changes. Pages convert to Markdown for LLM/RAG ingestion via
page.markdown(). - Spiders. A Scrapy-like
SpiderAPI with concurrency, multi-session routing, pause/resume (crawldir), AutoThrottle, robots.txt handling, and JSON/JSONL/CSV/XML exporters. Templates include CrawlSpider, SitemapSpider, and ShopifySpider. - Agent and CLI surface. An MCP server (
scrapling[ai]), an Agent Skill, an interactive shell, andscrapling extractone-liners.
Note on installation: pip install scrapling installs only the parser. Add "scrapling[fetchers]" and run scrapling install for browsers. Docker images ship with browsers included. Docs: scrapling.readthedocs.io.
Who it's for
Python developers who are tired of stitching together Requests, BeautifulSoup, Playwright, and Scrapy, and AI data-pipeline builders who need clean Markdown corpora or a scraping tool their agent can call. Follow site terms and robots.txt — the README frames the project for educational and research use.
Stars / language / license
99,895 stars · Python · BSD-3-Clause
Data verified: 2026-11-09 09:19 Asia/Shanghai (GitHub API get_repository + prior README structure)
Trending note: +8,342 stars this week (weekly Trending; selection captured 09:10 Asia/Shanghai)
Category tags
DevTools · web-scraping · crawling · Python · MCP · data-pipelines · RAG
How it differs from similar projects
Compared with Scrapy, Scrapling adds stealth browser fetching and adaptive parsing in the same package. Compared with Playwright scripts, it adds spider orchestration and exporters. Compared with hosted "URL → Markdown" APIs (Firecrawl-style services), it's self-hosted, BSD-licensed, and a full crawler, not just a single-page converter. For AI pipelines, the pattern is: spider to collect pages, markdown() to clean them, MCP tools for ad-hoc agent lookups. The README's parser speed tables are project benchmarks that TidyHow hasn't reproduced.