Apify-Haystack integration

The official integration of the Apify platform with Haystack.

Apify is a platform for web scraping, data extraction, and web automation tasks. It provides serverless applications called Actors for tasks like crawling websites and scraping Facebook, Instagram, and Google results.

Haystack is an open-source framework for building LLM applications, RAG pipelines, and agents.

For a step-by-step guide, see the Haystack integration page in the Apify documentation.

The integration is available as the apify-haystack package on PyPI. It requires Python 3.11 or newer and Haystack 3.

pip install apify-haystack

Upgrading from 0.1.x? See UPGRADING.md.

• ApifyDatasetFromActorCall runs an Actor, waits for it to finish, and loads its default dataset.
• ApifyDatasetFromTaskCall does the same for a saved Actor task.
• ApifyDatasetLoader loads an existing dataset by its ID.
• ApifyWebSearch searches the web with the RAG Web Browser Actor and returns the content of the top result pages.
• ApifyActorTool is a tool for Haystack Agents that runs any Actor and returns its dataset items. It takes its parameters from the Actor's input schema, and it marks the items it hands to the LLM as untrusted external content to guard against prompt injection.

The dataset components turn dataset items into Haystack Document objects with a mapping function. Without one, they use default_dataset_mapping_function. For website and search Actors, it takes the item's markdown or text as the content and the page url, title, and description as meta. For other Actors, the content is the whole item as JSON.

Every component also has run_async, which Pipeline.run_async uses to run it without blocking the event loop.

When an Actor or task run finishes with any status other than SUCCEEDED, the component raises a RuntimeError.

The components read your Apify API token from the APIFY_TOKEN environment variable, or from APIFY_API_TOKEN if APIFY_TOKEN isn't set. To pass the token differently, use a Haystack Secret, for example apify_token=Secret.from_env_var('MY_APIFY_TOKEN').

The examples need an Apify account and API token. You can start with a free account at Apify and get your API token, then set it as the APIFY_TOKEN environment variable.

ApifyDatasetFromActorCall runs Website Content Crawler and loads the crawled pages as Document objects with the default mapping function:

from apify_haystack import ApifyDatasetFromActorCall

crawler = ApifyDatasetFromActorCall(
    actor_id='apify/website-content-crawler',
    run_input={
        'maxCrawlPages': 3,
        'startUrls': [{'url': 'https://haystack.deepset.ai/'}],
    },
)

for document in crawler.run()['documents']:
    print(document.meta['url'], document.content[:100])

ApifyWebSearch returns the top result pages as Markdown Document objects, with the page url, title, and description in meta, and their URLs in links:

from apify_haystack import ApifyWebSearch

search = ApifyWebSearch(top_k=3)
result = search.run(query='What is Haystack?')

for document in result['documents']:
    print(document.meta['url'], document.meta['title'])

ApifyActorTool lets a Haystack Agent run an Actor. Limit dataset_items_limit to keep the tool output within the model's context window. For pay-per-event Actors, max_total_charge_usd caps what each run the Agent starts can cost:

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

from apify_haystack import ApifyActorTool

tool = ApifyActorTool('apify/rag-web-browser', dataset_items_limit=3, max_total_charge_usd=0.5)
agent = Agent(chat_generator=OpenAIChatGenerator(), tools=[tool])

result = agent.run([ChatMessage.from_user('What is new in Haystack 3?')])
print(result['last_message'].text)

The components and the tool support Haystack serialization. The token is saved as the name of its environment variable, never as its value. Haystack 3 only loads classes and functions from trusted modules, so pass apify_haystack in allowed_modules, along with the module of your mapping function if you use a custom one:

from haystack import Pipeline

from apify_haystack import ApifyDatasetLoader

pipeline = Pipeline()
pipeline.add_component('loader', ApifyDatasetLoader('YOUR-DATASET-ID'))

yaml = pipeline.dumps()
restored = Pipeline.loads(yaml, allowed_modules=['apify_haystack'])

The examples directory has complete scripts:

• Load an existing dataset and convert its items to Documents.
• Call Website Content Crawler with a custom mapping function.
• Crawl a website, store the pages in InMemoryDocumentStore, and retrieve them.
• Answer questions about a crawled website with RAG.
• Analyze the sentiment of Instagram comments.

If you find a bug, please submit an issue on GitHub. For questions, ask on Stack Overflow or join our Discord server.

Code contributions are welcome. If you have ideas for improvements, submit an issue or create a pull request. For contribution guidelines, see CONTRIBUTING.md.

This project is licensed under the Apache License 2.0.