Scrapling - Adaptive Web Scraping Framework from Request to Crawl

Scrapling describes itself as “an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl” — parsing, fetching and spider-style crawling in one Python package. It requires Python 3.10 or higher and is BSD-3-Clause licensed, with its translator submodule adapted from Parsel.

The project’s disclaimer sets the terms of use plainly: “This library is provided for educational and research purposes only,” and “Always respect the terms of service of websites and robots.txt files.” Those are the project’s own words and they belong at the top of any honest description of it.

The genuinely novel feature

Most scraping libraries differ in ergonomics. Scrapling’s distinguishing idea is maintenance: the parser “learns from website changes and automatically relocates your elements when pages update.”

Mechanically, you save an element’s signature on first run and ask for relocation later:

from scrapling.fetchers import StealthyFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
products = p.css('.product', auto_save=True)   # store the signature
products = p.css('.product', adaptive=True)    # find it again after a redesign

Elements are “relocated after website changes using intelligent similarity algorithms.” This addresses the actual cost of scraping, which is not writing selectors but repairing them every time a site ships a redesign. Whether the similarity matching holds up across a substantial redesign is something to test on your own targets — but the problem it picks is the right one.

Crawling, with brakes

The spider API is deliberately Scrapy-shaped: start_urls, async parse callbacks, pause and resume checkpoints, streaming.

from scrapling.spiders import Spider, Response

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com/"]

    async def parse(self, response: Response):
        for item in response.css('.product'):
            yield {"title": item.css('h2::text').get()}

MySpider().start()

Two features here are about being a good citizen rather than a fast one. AutoThrottle “tunes the delay of each domain” based on response speed and backs off when blocked or rate-limited. And there is an optional robots_txt_obey flag that respects robots.txt directives.

That flag being optional is worth noticing. Turning it on is the difference between a crawler that honours a site’s stated wishes and one that does not, and given the project’s own disclaimer about respecting robots.txt, it is the setting to enable by default rather than discover later.

The anti-bot features, described rather than recommended

The fetchers include fingerprint and TLS impersonation, proxy rotation, domain and ad blocking, and DNS-over-HTTPS. The project claims its fetchers “bypass anti-bot systems like Cloudflare Turnstile out of the box,” and that StealthyFetcher “can easily bypass all types of Cloudflare’s Turnstile/Interstitial with automation.”

Those are the project’s own claims, unverified here, and they describe capability rather than permission. An anti-bot system is a site operator stating they do not want automated access; defeating it does not change that, and it may put you on the wrong side of the terms of service the project itself tells you to respect. The legitimate uses are real — your own sites, sites you have agreement to crawl, research within a site’s stated rules — and they are the ones worth building on.

The extras that catch people out

The base package is parser-only, which catches people out:

pip install scrapling                 # parser engine only
pip install "scrapling[fetchers]"     # needed for fetchers and spiders
scrapling install                     # downloads browsers and dependencies

Importing scrapling.fetchers or scrapling.spiders fails without the fetcher extras. [all] pulls everything, and Docker images are published as pyd4vinci/scrapling.

Performance and coverage figures in the README — a test-coverage percentage, benchmark speedups — are the project’s own measurements.

Worth a look if

You maintain scrapers that keep breaking. The adaptive relocation is the feature to evaluate, and it is evaluated cheaply: add auto_save=True to selectors you already have and see whether they survive the next time the target changes.

Enable robots_txt_obey, leave AutoThrottle on, and be deliberate about the stealth features — the project’s own disclaimer is the right standard to hold yourself to.

Primary link

Open on github.com