Crawlzodocs
Guides

Errors and retries

What each failure means, which ones are worth retrying, and which are billed.

Two fields decide what your code should do. success says whether you got an answer. status.retryable says whether asking again could change that.

result = response.json()

if result["success"]:
    handle(result["data"])
elif result["status"]["retryable"]:
    schedule_retry_with_backoff()
else:
    log_and_stop(result["request_id"], result["status"]["code"])

Target not found

When a profile, post or video cannot be fetched because it does not exist, was deleted, or is restricted, you get a 404. The reason is in status.code.

{
  "request_id": "req_8f2c1d94e7b0",
  "success": false,
  "status": {
    "code": "PROFILE_DOESNT_EXIST",
    "message": "Profile does not exist.",
    "retryable": false
  },
  "data": null,
  "meta": { "scraper": "instagram-profile-v4", "billable": true, "duration_ms": 1205 }
}

Common codes are PROFILE_DOESNT_EXIST, PROFILE_NOT_FOUND, PROFILE_AGE_RESTRICTED, PROFILE_RESTRICTED, POST_DELETED, GEO_BLOCKED, and NOT_FOUND as the general fallback. The exact code describes why the target could not be read, so it is worth switching on rather than treating every 404 the same way.

On a 404 the response also carries error.detail, holding the reason as the scrape reported it. Log it; do not parse it.

A confirmed 404 is billed. Establishing that something is genuinely gone takes the same work as reading it, and we only return one after confirming it rather than on the first empty page.

Client errors

status.codeHTTPRetryableMeaning
INVALID_REQUEST400NoThe body could not be parsed, or a query parameter is not valid.
UNAUTHORIZED401NoThe key is missing or not recognised.
SCRAPER_NOT_ALLOWED403NoYour account is not enabled for this endpoint.
SCRAPER_NOT_FOUND404NoNo endpoint by that name. Check the spelling.
CLIENT_CLOSED_REQUEST499NoYou disconnected before the call finished. Usually a timeout set too low.

Rate limits and quota

status.codeHTTPRetryableMeaning
RATE_LIMITED429YesToo many requests too quickly. Back off.
QUOTA_EXCEEDED429YesYour plan's allowance is used up for this period.

Server errors

status.codeHTTPRetryableMeaning
REQUEST_FAILED5xxYesThe call did not complete.

REQUEST_FAILED is the single code you get for everything that goes wrong after your request was accepted. Several different situations collapse into it: the platform blocked or challenged the scrape, the page timed out, capacity was exhausted, or the request could never have worked because a parameter was misspelled or of the wrong type.

status.retryable is what separates them, and it is accurate. The original outcome sets it, and it survives the collapse.

  • retryable: true means the failure was transient. Back off and try again.
  • retryable: false on a REQUEST_FAILED means repeating the request cannot change the result. Stop, and check the request against the endpoint's page.

Cap your retries either way. Three attempts with exponential backoff will ride out a genuine transient problem. A request that fails all three is worth surfacing to a human rather than retrying forever: sending it once by hand and checking the parameter names against the endpoint's page usually settles it.

The message on a REQUEST_FAILED is always the same text, and error.detail is omitted, so there is nothing extra to parse. Keep the request_id: it is what lets us find the underlying cause if you need to ask.

What you are charged for

Calls that produce an answer are billed, and so are confirmed 404s. Calls that fail with REQUEST_FAILED, RATE_LIMITED or QUOTA_EXCEEDED are not.

meta.billable tells you which happened on every response, so you never have to infer it.

On this page