I am a developer of , a solution for testing LLM applications. We have a FastAPI backend and a Celery worker. The system performs many external API calls, followed by numerous LLM API queries (e.g., OpenAI) to evaluate the outputs. This is a typical I/O-bound workload.
The question is: how can we parallelize this more effectively?
Currently, we use a Celery task to execute a set of tests. Within a single task, we use asyncio—for example, if a test set contains 50 tests, we send all 50 requests concurrently and then perform all 50 evaluation queries concurrently as well.
The issue is that the number of test sets processed concurrently is limited by the number of prefork worker processes in Celery. Increasing the number of processes increases RAM usage, which we want to avoid.
What we're looking for is a way to fully leverage async—i.e., a system where tasks can be continuously scheduled onto an event loop without being constrained by a fixed number of worker processes or threads. While the code inside a task is asynchronous, the tasks themselves are still effectively executed sequentially at the worker level.
FastAPI demonstrates the model we're aiming for—handling many concurrent requests on an event loop and scaling with multiple processes (e.g., via Gunicorn). However, it does not provide task queuing capabilities.
So what would you recommend? Ideally, we're looking for a library or architecture where each process runs an event loop, and incoming tasks are scheduled onto it and executed concurrently, without waiting for previously submitted tasks to complete (unlike Celery's current model).
We also considered Dramatiq, but it appears to have a similar limitation—tasks can use async internally, but are still executed sequentially at the worker level.
Finally, we'd prefer a solution that is stable and mature — something with a proven track record in production environments, active maintenance, and a reliable community behind it. We're not looking to adopt an experimental or early-stage library as a core part of our infrastructure.
