Skip to main content

Storage

Copy for LLM

The Apify platform provides three types of storage, accessible from Apify Console and externally through the REST API, API clients, and SDKs.

Storage types

Dataset

Stores results from web scraping and data processing; each Actor run gets a unique dataset. Includes table-like visualization and export formats like JSON and Excel.

Key-value store

Stores data of any type: JSON, HTML, images, strings. Accessible via Apify Console or API.

Request queue

Manages URL processing for web crawling and similar tasks. Supports different crawling orders and lets you query and update URLs, accessible via Apify Console or API.

Access your storage

Access your storage through Apify Console, the API, the API clients, or the SDKs.

Apify Console

To view your storages in Apify Console:

  1. Open the Storage section in the left-side menu.
  2. Select a tab to view your key-value stores, datasets, or request queues.
  3. Select a storage's ID to open its detail page.

To view the related API endpoints, select API in the top right corner.

Storage types in Apify Console

Toggle unnamed storages

To show or hide unnamed storages, check Include unnamed storages. By default, Apify Console shows them.

To rename a store, open the Actions menu and select Rename.

To share a storage, select Share in the Actions menu and provide an email, username, or user ID.

Apify API

The Apify API lets you access your storages programmatically using HTTP requests and share your crawling results.

When accessing storages via API, provide a store ID in one of these formats:

  • WkzbQMuFYuamGv3YF - the store's alphanumerical ID if the store is unnamed.
  • ~store-name - the store's name prefixed with tilde (~) character if the store is named (e.g. ~ecommerce-scraping-results)
  • username~store-name - username and the store's name separated by a tilde (~) character if the store is named and belongs to a different account (e.g. janedoe~ecommerce-scraping-results). Note that in this case, the store's owner needs to grant you access first.

For read (GET) requests, the alphanumerical ID alone is enough, since it's hard to guess and acts as an authentication key.

For other request types, and when using username~store-name, provide your secret API token in the request's Authorization header or as a query parameter. Find your token on the API & Integrations page of your Apify account.

Token security

Never share a URL containing your authentication token. It can compromise your account's security. If the data you want to share requires a token, download the data first and share it as a file.

For a breakdown of each storage endpoint, see the API documentation.

Apify API clients

The Apify API clients let you access your storages from any Node.js or Python application, whether it runs on the Apify platform or externally.

For more details, see the API client docs.

Apify SDKs

The Apify SDKs are JavaScript and Python libraries for building your own Actors.

  • JavaScript SDK requires Node.js 16 or later.
  • Python SDK requires Python 3.10 or above.

Named and unnamed storages

The default storages for an Actor run are unnamed, identified only by an ID. Naming a storage ensures indefinite retention regardless of plan; unnamed storages follow the data retention rules below.

Named and unnamed storages are identical except for their retention period. Named storages are easier to identify. The names janedoe~my-storage-1 and janedoe~web-scrape-results are easier to tell apart than the IDs cAbcYOfuXemTPwnIB and CAbcsuZbp7JHzkw1B. Storage names can be up to 63 characters long.

Name a storage

You can name a storage via Apify Console or through the API.

In Apify Console:

  1. Open your run's details and select the Dataset, Key-value store, or Request queue tab as appropriate.
  2. Find the store's ID.
  3. Click the ID to open the storage details.
  4. Click on the Actions menu and choose Rename.
  5. Enter a new name. Your storage is now preserved indefinitely.

Via API: get the storage's ID from the run that generated it using the Get run endpoint, then rename it using the Update [storage] endpoint (for example, Update dataset).

The SDKs and clients each have their own naming conventions. See:

Data retention

How long Apify keeps your data depends on your plan. Unnamed storages and runs beyond the 10 most recent are deleted automatically; named storages are retained indefinitely.

How retention periods are enforced
  • Free plan: Your 10 most recent runs are retained for 4 months.
  • Paid plans: All data (including your 10 most recent runs) follows your plan's retention period, which you can configure in your billing settings.
  • Named storages: Always exempt from deletion regardless of retention periods.

Unnamed storages beyond the 10 most recent runs are deleted when the retention period expires.

Estimate your costs

Estimate your storage costs
  1. Select a storage type.
  2. Choose a plan.
  3. Enter storage, duration, and operation counts.
  4. Review the estimated total and breakdown.

Storage Pricing

Estimate costs for your storage usage

This is an estimate
This is an estimate based on current pricing. Actual costs may vary.

Storage Type

Plan

Usage

Estimated Costs

Storage (1 GB × 24 hours)$0.0240
Reads (1,000 operations)$0.0004
Writes (1,000 operations)$0.0050
Total Estimated Cost$0.0294

Rate limiting

All API endpoints limit their request rate to protect Apify servers from overload. The default rate limit for storage objects is 60 requests per second per storage object, and some endpoints have higher or lower limits. The tiers differ per storage type:

If a client exceeds its limit, the API endpoints respond with the HTTP status code 429 Too Many Requests and the following body:

{
    "error": {
        "type": "rate-limit-exceeded",
        "message": "You have exceeded the rate limit of ... requests per second"
    }
}

Go to the API documentation for details and to learn what to do if you exceed the rate limit.

Share

You can grant access rights to other Apify users, share a storage by link, or generate a time-limited pre-signed URL for one-off access to a restricted resource. For details, see Share storage.

Concurrent access

If you have the storage's name or ID, you can access it from any Actor or task run. Datasets and key-value stores support concurrent reads and writes, while a request queue accepts new data from multiple runs but can only be processed by one run at a time, unless you use request locking. For details, see Use storage from another run.

Delete storages

Named storages are only removed upon your request. You can delete storages in the following ways: