Concepts
How crawling, scraping, and delivery fit together, and the knobs you'll reach for most often.
Crawling vs. scraping
Spider has two core operations. Scraping fetches a single page and returns its content. Crawling starts from a URL and follows links to fetch many pages across a site. Both take the same parameters for output format, proxy usage, and request mode. See Scraping and crawling for endpoint specifics.
Request modes
Every request uses one of three modes. The default smart inspects each page and picks between a lightweight HTTP fetch and a full Chrome browser based on what the page needs.
| Mode | When to use | Speed | Cost | JS rendering |
|---|---|---|---|---|
| smart | Default. Works for most sites. | Fast | Low to medium | Auto-detected |
| http | Static HTML, APIs, known simple pages. | Fastest | Lowest | No |
| chrome | SPAs, JS-rendered content, bot-protected sites. | Slower | Higher | Yes |
Concurrent crawling
The Rust engine fetches, renders, and processes pages in parallel, so a 500-page crawl doesn't take 500 times longer than a single page. The server manages the concurrency; there are no thread pools or connection limits to wire up. For large jobs, pair it with streaming so you can process pages the moment they arrive.
Output formats
The return_format parameter controls how Spider delivers page content. Markdown is the default for AI workloads. It keeps the structure, strips navigation and ads, and costs far fewer tokens than raw HTML.
| Format | What you get | Best for |
|---|---|---|
| raw | The HTML as the server returned it. | Parsing with your own tools, archiving. |
| markdown | Clean text with the structure kept. Navigation, scripts, and boilerplate stripped. | LLMs, RAG pipelines, content analysis. |
| text | Plain text without any markup. | Simple text extraction, word counts. |
| bytes | Binary data for non-HTML resources. | PDFs, images, file downloads. |
Streaming
With streaming on, Spider returns each page as a JSON line the moment it finishes, with no buffering of the full result set. Lower memory, no HTTP timeouts, faster time to first result. See Concurrent streaming for full examples.
import requests, json, os
headers = {
'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
'Content-Type': 'application/jsonl', # Enable streaming
}
response = requests.post(
'https://api.spider.cloud/crawl',
headers=headers,
json={"url": "https://example.com", "limit": 50, "return_format": "markdown"},
stream=True,
)
for line in response.iter_lines():
if line:
page = json.loads(line)
print(f"Received: {page['url']} ({page.get('status')})")Screenshots
The /screenshot endpoint captures full-page or viewport-sized images as PNG, JPEG, or WebP, returned as base64 or raw binary. Use it for visual regression tests, for archiving how a page looked, or to pair a picture with extracted text. It always renders in Chrome, so JavaScript-heavy pages come out right.
AI extraction
Spider can pull structured data from pages with AI. With an AI Studio subscription, describe the fields you want and Spider returns structured JSON instead of raw content. Good for product details, contact info, or any repeatable shape, with no CSS selectors required. See JSON scraping for the parameter reference.
Credits
Usage is measured in credits at $1 per 10,000 credits. Each crawled page has a base cost; Chrome rendering, proxy usage, and AI extraction add on top. Failed requests, timeouts, and blocked pages cost zero. Every response includes a costs field with a per-request breakdown. View the live balance and history on the usage page.
