Authentication

Every request carries your API key as a Bearer token in the Authorization header, over HTTPS.

Get a key

Send the key

Put the key in the Authorization header of every request. The API rejects plain HTTP.

cURL

curl 'https://api.spider.cloud/crawl' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "limit": 1}'

Python

import requests
import os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={"url": "https://example.com", "limit": 1}
)

print(response.json())

Node.js

const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com', limit: 1 }),
});

const data = await response.json();
console.log(data);

Let the SDK send it

The client libraries read SPIDER_API_KEY from the environment and set the header for you.

Python SDK

# pip install spider_client
from spider import Spider

# Reads SPIDER_API_KEY from the environment
app = Spider()
result = app.crawl_url('https://example.com', params={'limit': 1})
print(result)

Node.js SDK

// npm install @spider-cloud/spider-client
import { Spider } from "@spider-cloud/spider-client";

// Reads SPIDER_API_KEY from the environment
const app = new Spider();
const result = await app.crawlUrl("https://example.com", { limit: 1 });
console.log(result);

Tip

Never paste a key into source code. Read it from an environment variable or a secrets manager.

Keep the key safe

Five habits that cover most leaks.

Read the key from the environment. Store it in SPIDER_API_KEY and read it at runtime.
Keep it out of version control. Add .env to .gitignore before the first commit.
Rotate a leaked key. Regenerate it from the dashboard the moment you suspect it has been exposed.
One key per environment. Development, staging and production each get their own.
Share it only with the people who need it. A key is an account, not a config value.

Regenerate a key

Regenerate from the API keys page at any time. The old key stops working the moment the new one exists, so update every client first.

When the key is wrong

A missing, invalid or revoked key gets a 401 Unauthorized. The error codes page lists every other response.

401 response

{
  "error": "Unauthorized",
  "status": 401
}