Rank #03 · Weekly

Rank #03 · Weekly · DevTools

Scrapling: One Python Framework From Single Request to Full Crawl

· Weekly Top10 · License: BSD-3-Clause

Rank #03 Weekly — D4Vinci/Scrapling

Fetch, parse, and crawl with one library, then hand the results to an LLM pipeline or let an agent drive it over MCP.

Canonical digest: /daily/scrapling (also on Weekly board).

  • Board: Rank #03 · Weekly ·
  • Repo: D4Vinci/Scrapling
  • Stars / lang / license: 99,895 stars · Python · BSD-3-Clause
  • License label: License: BSD-3-Clause
  • Data verified: 2026-11-09 09:19 Asia/Shanghai (GitHub API get_repository + prior README structure)
  • Trending note: +8,342 stars this week (weekly Trending; selection captured 09:10 Asia/Shanghai)
  • Tags: DevTools · web-scraping · crawling · Python · MCP · data-pipelines · RAG

What it does

Note on installation: pip install scrapling installs only the parser. Add "scrapling[fetchers]" and run scrapling install for browsers. Docker images ship with browsers included. Docs: scrapling.readthedocs.io.

Who it's for

Python developers who are tired of stitching together Requests, BeautifulSoup, Playwright, and Scrapy, and AI data-pipeline builders who need clean Markdown corpora or a scraping tool their agent can call. Follow site terms and robots.txt — the README frames the project for educational and research use.

Stars / language / license

99,895 stars · Python · BSD-3-Clause

Data verified: 2026-11-09 09:19 Asia/Shanghai (GitHub API get_repository + prior README structure)

Trending note: +8,342 stars this week (weekly Trending; selection captured 09:10 Asia/Shanghai)

Category tags

DevTools · web-scraping · crawling · Python · MCP · data-pipelines · RAG

How it differs from similar projects

Compared with Scrapy, Scrapling adds stealth browser fetching and adaptive parsing in the same package. Compared with Playwright scripts, it adds spider orchestration and exporters. Compared with hosted "URL → Markdown" APIs (Firecrawl-style services), it's self-hosted, BSD-licensed, and a full crawler, not just a single-page converter. For AI pipelines, the pattern is: spider to collect pages, markdown() to clean them, MCP tools for ad-hoc agent lookups. The README's parser speed tables are project benchmarks that TidyHow hasn't reproduced.

Repo link

https://github.com/D4Vinci/Scrapling