Reliability

Streams drop, servers restart, and a deploy kills any long-running connection. On a crawl of thousands of pages the results have to land even when your client doesn't. Data connectors and webhooks run on Spider's side, so the data lands whatever happens to your connection. Four patterns for combining them with streaming.

Stream + data connector

Stream JSONL to process pages as they arrive, and attach a data connector so Spider writes every page to your storage too. If the connection drops, the connector already has the data.

Stream JSONL with S3 backup

import requests, os, json

headers = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/jsonl",
}

response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={
    "url": "https://example.com",
    "limit": 100,
    "return_format": "markdown",
    "data_connectors": {
        "s3": {
            "bucket": "my-crawl-data",
            "access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
            "secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
            "region": "us-west-2",
            "prefix": "crawls/"
        },
        "on_find": True
    }
}, stream=True)

# Process in real time while S3 has your backup
for line in response.iter_lines():
    if line:
        page = json.loads(line)
        print(f"Got {page['url']}")

Tip

Ctrl+C the client and the connector keeps writing. Delivery runs on Spider's side, independent of your stream. The data connectors page covers each provider.

Background crawl with webhook

Set run_in_background: true, point a connector at your storage, and add an on_website_status webhook. The API returns at once. Pages pile up in your bucket or database while Spider crawls, and the webhook fires when it finishes. Best for cron jobs and large batch crawls.

Background crawl with Supabase and webhook

import requests, os

headers = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/json",
}

response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={
    "url": "https://example.com",
    "limit": 500,
    "return_format": "markdown",
    "run_in_background": True,
    "data_connectors": {
        "supabase": {
            "url": "https://your-project.supabase.co",
            "anon_key": os.getenv("SUPABASE_ANON_KEY"),
            "table": "crawled_pages"
        },
        "on_find": True
    },
    "webhook": {
        "url": "https://your-server.com/crawl-done",
        "on_website_status": True
    }
})

# Returns immediately with a crawl_id
print(response.json())

JSONL checkpointing

For a small crawl, track progress on the client with blacklist. Collect URLs as you consume the stream. If the connection drops, pass them back so Spider skips the pages you already have. This holds up to a few hundred URLs. Past that the request payload gets too large, so use a data connector instead: Spider tracks delivery on its side and there is nothing to replay on reconnect.

Client-side checkpointing with blacklist

import requests, os, json

API = "https://api.spider.cloud/crawl"
HEADERS = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/jsonl",
}

def crawl_with_checkpoint(url: str, limit: int):
    processed = set()

    while len(processed) < limit:
        try:
            body = {
                "url": url,
                "limit": limit - len(processed),
                "return_format": "markdown",
            }
            if processed:
                body["blacklist"] = list(processed)

            resp = requests.post(API, headers=HEADERS, json=body, stream=True)

            for line in resp.iter_lines():
                if line:
                    page = json.loads(line)
                    processed.add(page["url"])
                    yield page

        except requests.exceptions.ConnectionError:
            if not processed:
                raise
            print(f"Disconnected after {len(processed)}/{limit}, resuming...")
            continue

crawl_with_checkpoint("https://example.com", limit=100)

Tip

For a large crawl, skip the blacklist and use a data connector. Spider delivers pages whatever your connection does, so there is nothing to resume.

Webhook queue pipeline

Enable the on_find webhook and push each page into the queue you already run, such as SQS, Redis Streams or RabbitMQ. Spider finds pages, the queue buffers them, workers consume at their own pace.

Crawl with on_find webhook

import requests, os

headers = {
    "Authorization": f"Bearer {os.getenv('SPIDER_API_KEY')}",
    "Content-Type": "application/json",
}

response = requests.post("https://api.spider.cloud/crawl", headers=headers, json={
    "url": "https://example.com",
    "limit": 200,
    "return_format": "markdown",
    "webhook": {
        "url": "https://your-server.com/spider-webhook",
        "on_find": True,
        "on_website_status": True
    }
})

print(response.json())

Webhook receiver pushing to SQS

from fastapi import FastAPI, Request
import boto3, json

app = FastAPI()
sqs = boto3.client("sqs")
QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789/spider-pages"

@app.post("/spider-webhook")
async def handle_webhook(request: Request):
    payload = await request.json()

    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps(payload),
    )

    return {"ok": True}  # return 200 fast

Tip

Return 200 as soon as the message is queued. Heavy inline work makes Spider time out waiting for your response.

Choose a pattern

They combine. A common setup streams with a connector as the backup, then adds a webhook queue for post-processing.

Need	Pattern
Live output with guaranteed delivery	Stream + connector
Large batch or cron jobs	Background + connector + webhook
Small crawls that must survive a disconnect	JSONL checkpointing
An event-driven pipeline	Webhook queue

Tip

New here? Read data connectors, webhooks and streaming first.