About

We build the data infrastructure for AI.

Spider started as a Rust crawler and became a question: how do you know web data is good? We run it as a quality lab and answer in public. Open source core, open benchmarks, and a bill that reads $0 when a page fails.

2023

spider, open source

A low-latency Rust crawler. Concurrent by default, designed to crawl thousands of pages a second on a single host. MIT-licensed at github.com/spider-rs/spider.

2024

Spider Cloud

The same crawler, hosted. Stealth, proxies, anti-bot, and auto-scaling rolled in. JavaScript rendering included; pay only for bandwidth and compute used.

2025

AI extraction

Send a prompt and a URL, get structured JSON. Selectors and parsers are still there if you want them, but most teams stop reaching for them.

2026

The quality lab

Silk, our extraction model. StealthBench V1, published with methodology. The work shifted from moving pages fast to proving the data is right: fidelity, completeness, freshness, provenance.

Support: support@spider.cloud
Sales: sales@spider.cloud
GitHub: github.com/spider-rs
Twitter / X: @spider_rust
Discord: discord.spider.cloud