AnyCrawl

Swiftproxy(https://www.swiftproxy.net/?ref=AnyCrawl) — High-performance residential proxies built for scraping, automation, and large-scale data collection. Access 80M+ rotating residential IPs across 195+ countries with stable connections, high anonymity, and developer-friendly integration. Ideal for AI agents, crawlers, browser automation, and anti-bot bypass workflows. Free trial available. Use code PROXY90 for an exclusive 10% discount.

Rapidproxy — RapidProxy is a high-performance proxy provider offering clean residential proxies and native static ISP IPs for web scraping, browser automation, social media automation, e-commerce, multi-account management, and large-scale data operations. With 90M+ residential IPs, smart rotation, stable sessions, high concurrency, AI-powered CAPTCHA bypass, and non-expiring traffic, RapidProxy helps developers run reliable automation tasks at scale. Residential proxies start from $0.65/GB. Use code RAPID10 for 10% off — try it now.

TalorData provides a fast, reliable SERP API that delivers structured, real-time search data from Google, Bing, Yandex, and DuckDuckGo, built for AI agents and SEO automation. Sign up to receive free trial and a 10% discount.

AnyCrawl is a high‑performance crawling and scraping toolkit:

• SERP crawling: multiple search engines, batch‑friendly
• Web scraping: single‑page content extraction
• Site crawling: full‑site traversal and collection
• High performance: multi‑threading / multi‑process
• Batch tasks: reliable and efficient
• AI extraction: LLM‑powered structured data (JSON) extraction from pages

LLM‑friendly. Easy to integrate and use.

📖 See full docs: Docs

If you enable authentication (ANYCRAWL_API_AUTH_ENABLED=true), generate an API key:

pnpm --filter api key:generate
# optionally name the key
pnpm --filter api key:generate -- default

The command prints uuid, key and credits. Use the printed key as a Bearer token.

If running AnyCrawl via Docker:

• Docker Compose:

docker compose exec api pnpm --filter api key:generate
docker compose exec api pnpm --filter api key:generate -- default

• Single container (replace <container_name_or_id>):

docker exec -it <container_name_or_id> pnpm --filter api key:generate
docker exec -it <container_name_or_id> pnpm --filter api key:generate -- default

💡 Use the Playground to test APIs and generate code in your preferred language.

If self‑hosting, replace https://api.anycrawl.dev with your own server URL.

curl -X POST https://api.anycrawl.dev/v1/scrape \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "url": "https://example.com",
  "engine": "cheerio"
}'

More parameters: see Request Parameters.

Cache details (self-host / S3 / map index): see docs/cache.md.

The public scrape and crawl engine values remain cheerio, playwright, and puppeteer. For self-hosted browser engines, playwright and puppeteer are launched through CloakBrowser by default; callers should not send a cloakbrowser engine value.

CloakBrowser requires Node.js 20 or newer. Docker images pre-install its browser binary during image build. For local or custom deployments, set CLOAKBROWSER_CACHE_DIR to a stable writable path and CLOAKBROWSER_AUTO_UPDATE=false to avoid browser downloads during worker startup. If you manage the binary yourself, set CLOAKBROWSER_BINARY_PATH.

Browser engines use CloakBrowser's native fingerprint and context/viewport defaults. Crawlee fingerprint injection and its automatic headless UA override are disabled for this runtime; explicit user-agent and viewport overrides remain supported. Browser processes are reused only for the same configured proxy URL, while page contexts remain isolated by default.

ANYCRAWL_BROWSER_GEOIP=true (default) resolves timezone and locale through the selected proxy when a browser starts. The optional GeoIP dependencies are installed; its database (~70 MB) is downloaded and cached on first use. Failed resolution fails the launch instead of continuing with an unknown region. For a known region, set both ANYCRAWL_BROWSER_TIMEZONE and ANYCRAWL_BROWSER_LOCALE to avoid lookup, or explicitly disable automatic lookup with ANYCRAWL_BROWSER_GEOIP=false. Use sticky or region-stable proxies: a fixed proxy URL does not guarantee a fixed exit, and GeoIP databases may disagree. Changing the exit behind the same URL cannot change an already running browser's process-wide timezone.

API and worker deployments should use the same browser region settings. Browser cache identity includes the native fingerprint policy and these settings, preventing reuse of entries from the old injected-fingerprint policy. Existing cached data is retained.

curl -X POST "https://api.anycrawl.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_ANYCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "json_options": {
      "schema": {
        "type": "object",
        "properties": {
          "company_mission": { "type": "string" },
          "is_open_source": { "type": "boolean" },
          "employee_count": { "type": "number" }
        },
        "required": ["company_mission"]
      }
    }
  }'

AnyCrawl supports Atlas Cloud as an OpenAI-compatible LLM provider for extraction and summarization workloads.

• Official site: Atlas Cloud
• LLM base URL: https://api.atlascloud.ai/v1
• Recommended env model format: atlascloud/deepseek-v3

ATLASCLOUD_BASE_URL=https://api.atlascloud.ai/v1
ATLASCLOUD_API_KEY=your-atlascloud-api-key
DEFAULT_LLM_MODEL=atlascloud/deepseek-v3
DEFAULT_EXTRACT_MODEL=atlascloud/deepseek-v3

If you prefer file-based AI config, add an atlascloud provider entry in ai.config.json and map it to any Atlas Cloud model exposed through the OpenAI-compatible chat API.

curl -X POST https://api.anycrawl.dev/v1/crawl \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "url": "https://example.com",
  "engine": "playwright",
  "max_depth": 2,
  "limit": 10,
  "strategy": "same-domain"
}'

More parameters and endpoints: see Request Parameters.

Scrape many known URLs in a single asynchronous job, all sharing the same scrape options. Create a job, then poll status and pull paginated results (like Crawl, but with a fixed URL set and no link discovery).

curl -X POST https://api.anycrawl.dev/v1/batch/scrape \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "urls": ["https://example.com/a", "https://example.com/b"],
  "engine": "cheerio",
  "formats": ["markdown"]
}'
# -> { "success": true, "data": { "job_id": "...", "status": "created", "total": 2 } }

# Poll status / fetch results
curl -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' https://api.anycrawl.dev/v1/batch/scrape/JOB_ID/status
curl -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' https://api.anycrawl.dev/v1/batch/scrape/JOB_ID

Credits are charged per successfully scraped URL (failed URLs are not charged). Full reference: Batch Scrape API.

curl -X POST https://api.anycrawl.dev/v1/search \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
  -d '{
  "query": "AnyCrawl",
  "limit": 10,
  "engine": "google",
  "lang": "all"
}'

• Google

1. Can I use proxies? Yes. AnyCrawl ships with a high‑quality default proxy. You can also configure your own: set the proxy request parameter (per request) or ANYCRAWL_PROXY_URL (self‑hosting).
2. How to handle JavaScript‑rendered pages? Use the Playwright or Puppeteer engines.

We welcome contributions! See the Contributing Guide.

Browser fingerprint and scoring checks have a separate live test type: pnpm test:browser-score --help. It records BrowserScan, CreepJS, Sannysoft, and server-verified reCAPTCHA demo results without combining them into a human score. See the browser-score testing guide for proxy configuration, comparison variants, thresholds, and saved artifacts.

Support us with a monthly donation and help us continue our activities. [Become a backer]

MIT License — see LICENSE.

We build simple, reliable, and scalable tools for the AI ecosystem.