Measure everything. Publish the method.
Spider Research is the lab layer of Spider: open benchmarks and field notes on web data quality for AI. Every number links to a method; every method links to a repo.
Spider passed 80/80 anti-bot tasks on Sep 18, 2026, over loopback through the session proxy that production requests use. Spider's figure is the CDP harness run. Other providers retain the March 22, 2026 readings described in the blog as LLM-judged; they were not re-measured. Production requests use the session proxy.
The record.
Each entry is a dated snapshot of a maintained measurement. Where a repo exists, the numbers reproduce from a clone.
- Spider Browser scores 85% on Browser Use's stealth benchmarkReproduce
Spider Browser scored 85% on Browser Use's open stealth benchmark, beating every other cloud browser tested against 80 anti-bot protected sites.
- Introducing Silk: our custom AI model for web data extraction
Silk is Spider's own extraction model. It turns raw HTML into structured data and solves captchas on our GPUs, with no external API calls and no per-token billing.
- Spider Browser vs. Kernel vs. Browserbase: 999 URLs benchmarkedReproduce
Kernel benchmarked cold start. We benchmarked reliability: 999 URLs, 254 domains, 18 categories, 100% success and 2.5s median end-to-end latency.
- Crawl4AI vs Firecrawl vs Spider: 1,000 URLs benchmarkedReproduce
Crawl4AI vs Firecrawl on 1,000 real URLs: 89.7% vs 95.3% success rate, Firecrawl pricing per 1K pages, RAG recall@5, and the method to rerun it yourself.
- Rust vs. Python for web scraping: why we rewrote everything
Why Spider abandoned its Python scrapers and rebuilt in Rust: the benchmarks, the architecture decisions, and what the rewrite cost us.
- Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
- The true cost of web scraping at scale
What 10K to 10M pages a month really costs on self-hosted Scrapy, Firecrawl, Apify, Crawl4AI and Spider, counting proxies, infrastructure and engineer time.
Same tasks, same scoring, every provider.
A leaderboard is only as honest as its instrument. Every Spider benchmark holds three invariants:
Every provider runs the same 80-task list against the same pages, measuring completeness under anti-bot pressure. Each row comes from its own run, not one shared session.
A task passes when the page comes back and fails when the response matches one of the block patterns. Every provider gets the same check, and no model sits in the loop.
We publish the task list and the scoring, and the runs we cite stay on this page. Clone spider-rs/benchmark and rerun the numbers yourself.
Method · spider-rs/benchmark ↗
On the bench.
- Quality Bench v1
Extraction fidelity, HTML against ground-truth markdown.
- Dataset drop
The 1,000-URL evaluation set behind the benchmark.
- Silk structure eval
Structure-conformance scoring for Silk output.
Judge the output yourself.
Post a URL, read what comes back. A failed request bills $0.
