Error codes

The API answers with standard HTTP status codes. An error also carries a JSON body. See authentication for key errors and rate limits for throttling.

Error body

Every error returns this shape.

Error body

{
  "error": "Rate limit exceeded",
  "status": 429
}

Each page in a crawl result carries the HTTP status the target site returned. It says nothing about your request to Spider.

Status code	Description	Consumes credits	Cause	Resolution
2xx	Success	Yes	The target page was fetched.	Nothing to do.
404	Target page not found	Yes	The URL does not exist on the target site.	Check the URL and that the page still exists.
403	Forbidden	Yes	The target site blocked the request.	Turn on proxy_enabled or set request to "browser" for JavaScript rendering.
401	Unauthorized	Yes	The target page needs a login.	Use automation_scripts to run the login flow first.
400	Bad request	Yes	The target server rejected the request.	Check that the URL is well formed and reachable.
429	Rate limit exceeded	Yes	The target site is rate limiting requests.	Add a delay between requests or lower concurrency.
500	Server error	No	The target server hit an internal error.	Retry. If it keeps failing, the target site itself is down.
503	Service unavailable	No	The target server is temporarily unavailable.	Retry after a short delay. The site may be under maintenance.

The HTTP status of the response to your request. It tells you whether Spider accepted and ran it.

Status code	Description	Cause	Resolution
200	Success	Spider ran the request.	Nothing to do.
204	No content returned	The crawl finished but found no content.	Check that the URL returns content. Try another return_format.
400	Bad request, check request parameters	Invalid parameters or a malformed JSON body.	Check the body for a missing required field such as url, or a bad value.
401	Authentication error	Missing, invalid, or expired API key.	Check the Authorization header against the authentication page.
402	Payment required	No credits left or no active subscription.	Add credits or upgrade from the dashboard.
413	Payload too large	The request body is over the size limit.	Send fewer URLs or parameters per request.
429	Rate limit exceeded	Too many requests in the current window.	Back off exponentially. The rate limits page lists your plan's limits.
500	Server error	An unexpected error on Spider's side.	Retry. If it keeps failing, contact support.
503	Service unavailable	Spider is temporarily unavailable.	Retry after a short delay. Check the status page for an incident.