Python `apply()` vs. `apply_async()`: Which Should You Use?
Multiprocessing lets a program run multiple tasks at the same time by creating separate processes. Each process has its own memory and Python interpreter, which allows full use of multiple CPU cores. async_apply and apply are two methods provided by Python’s multiprocessing.Pool for running parallel tasks.
Before we dive into the technical explanations of these methods, let’s talk shop. A mechanic’s shop. Imagine a mechanic’s shop with four repair bays. Just like CPUs, each bay has its own assigned mechanic team. A manager (the main Global Interpreter Lock [GIL] thread) assigns cars to each bay based on varying needs.
With apply, the manager sends a car to a specific bay. This car is a rare collector’s car, and it is very important that it gets repaired to pristine condition as quickly as possible. The repair happens in its own bay, working independently and in parallel with the main office, even though the manager waits and does not assign other cars during this time. Instead of assigning other cars to the remaining bays, the manager watches over the collector’s car while everyone else in the shop is on standby. Once the collector’s car is repaired and checked, work returns to its normal flow.
With apply_async, everyone works at the same time. As cars arrive, they get assigned to bays immediately. Multiple mechanics work on different cars simultaneously, finishing and inspecting each car before returning it to its owner shortly after completion. All bays operate in parallel.
Does apply seem like just plain old synchronous code to you?
Why Use the apply Method?
Looks can be deceiving. apply is actually a way to run synchronous code in a separate process. While the main program waits for the task to finish, the actual work happens independently on a separate CPU. Just like a rare collector’s car being repaired in its own bay while the manager pauses other duties, apply runs a task in a separate process even though the main program waits until it’s done.
So why use apply instead of running the code directly in the main thread if it’s going to be synchronous anyway? There are clear benefits. Running a task on a separate CPU lets the heavy work happen isolated from the main program’s memory and environment. This prevents crashes or slowdowns in the main program, even though the main program doesn’t continue until the task finishes.
apply is a powerful tool when you have a single, important task that must be completed before moving on and you want to run it in isolation. Debugging, image and video processing, file compression or encryption and data analysis are all good examples of this kind of work.
Syntax of apply
func: The target function to execute.args: A tuple of positional arguments.kwds: A dictionary of keyword arguments.
Example Using apply
This code creates a pool with two worker processes. The apply call runs the square function on a separate process with input 4. The main program waits for the result, which is 16, and then prints it. Only one task runs, and the program blocks until it finishes.
Output: Result with apply: 16.
What Is the apply_async Method?
apply_async is what most people think of when they picture processes running in parallel, like multiple cars being worked on at the same time.
apply_async runs the function asynchronously, meaning it immediately returns an AsyncResult object, and the main program can continue running other code while the process executes in the background. These tasks run on separate CPUs in parallel and don’t block the main program while they complete. Typically, they also don’t require results from other processes to start.
Syntax of apply_async
func: The function to execute.args: Tuple of arguments.kwds: Dictionary of keyword arguments.callback: Optional function to call when the result is ready.
Example of apply_async
This code creates a pool with two worker processes. The apply_async call sends the square function and input 4 to a separate process, but it doesn’t wait for the result right away. Instead, it returns an AsyncResult object immediately.
The program prints a message and continues running other code. When it calls .get(), it waits for and retrieves the result, which is 16. The task runs in parallel, and the main program stays responsive while it runs.
Output: Doing other work while process runs…
Result with apply_async: 16.
Comparing apply_async vs. apply: Performance
Think back to the mechanic’s shop: Each bay is its own dedicated workspace, and teams working side by side don’t get in each other’s way. However, when the shop needs to get multiple cars finished at the same time, assigning them to different bays is far more efficient than tackling them one by one.
As we covered above, apply_async and apply are two very different types of multiprocessing used for different use cases and perform differently. To recap the main differences, apply is synchronous and blocks the main program until the function returns. apply_async is asynchronous and does not block the main program.
Practical Speed Comparison
When running many tasks, apply_async can be faster overall because it allows tasks to execute in parallel. For individual tasks, the performance is basically the same, since both methods run the work in a separate process.
Performance Example: Single Task
Here, apply() and apply_async() are basically equivalent in total time because you only run one task.
Both should take approximately two seconds.
Performance Example: Multiple Tasks
In the code below,apply_async() is faster because tasks run in parallel.
Outputs will differ depending on your system, but you should see the apply completion time is significantly longer than the apply_async time when running multiple tasks.
Comparing apply_async vs. apply: Use Case Best Practices and Recommendations
The performance examples confirm the importance of selecting async_apply instead of apply really depends on the use case rather than the performance characteristics of the methods themselves.
When To Use apply
- You’re processing a single large CSV file and need the cleaned data before moving on to the next step.
- You want to encrypt a file and can’t proceed until it’s securely written to disk.
- You’re running a scientific calculation that must finish before the next calculation can start.
- You’re debugging a complex function and want to run it in isolation, but still step through the rest of your code in order.
When To Use apply_async
- You have a directory of images to resize, and you want to process them all at the same time.
- You need to scrape dozens of web pages in parallel and collect the results when they’re ready.
- You’re training multiple machine learning (ML) models with different parameters and want to use all your CPU cores to speed things up.
- You’re running several simulations that don’t depend on each other, and you want to kick them all off at once.
If your workflow is sequential and results must be handled immediately, apply is simple and clear. If your tasks are independent and you care about finishing faster, apply_async helps you get the most out of your hardware.
Understanding Results Handling
Results handling refers to how your program receives, manages and uses the outputs from the tasks you run in multiprocessing.
Managing Results With apply
Handling results from apply() is simpler. With apply(), results are returned immediately because the main program waits for the task to finish. This means you can use or assign the result right away in the very next line of code.
In the code example below, the func function runs in a separate process with input 4. The apply() call waits for the function to complete, then returns the result directly. This result is assigned to the variable result and printed immediately.
Managing Results With async_apply
Results handling with async_apply() is a bit more complicated, as many tasks are running simultaneously. With apply_async(), you get an AsyncResult object and must call .get() when you’re ready to retrieve the result.
You can also pass a callback to process results as soon as they are available.
In the example below, apply_async() starts the function in a separate process and immediately returns an AsyncResult object. The program prints a message and continues working. When you call .get() on the AsyncResult, the program waits (if needed) and retrieves the result, which is then printed.
Error Handling and Debugging
Error handling works differently between apply() and apply_async(), and that impacts how you debug your multiprocessing code.
Debugging With apply
Debugging with apply is simpler because exceptions raised inside the function are immediately propagated back to the main program. You can catch and handle them right where you call apply().
This immediate feedback makes apply() easier to debug since your program pauses and tells you about errors as soon as they happen.
Debugging With async_apply
Debugging with apply_async is more complicated because errors happen asynchronously in the background process. The exception won’t appear until you call .get() on the AsyncResult, which re-raises the exception in the main program.
Since exceptions only show up when you request the result, debugging asynchronous code can be trickier. You might not know something went wrong until later, and you have to remember to catch errors when calling .get().
General Tips for Debugging Multiprocessing Code
Catching exceptions with try-except around .get() in apply_async() is one common and effective way to debug asynchronous tasks, but it’s not the only way. It’s generally the best starting point for catching errors coming from background processes.
Other tips include:
- Adding logging inside worker functions to trace progress and spot issues.
- Using smaller test inputs to isolate problems.
- Running your function without multiprocessing first to ensure it works correctly.
- Carefully checking that arguments and data passed between processes are serializable.
- Watching out for deadlocks or hangs caused by waiting on results improperly.
Common Mistakes and Pitfalls
Mistakes are easy to make when working with multiprocessing. Multiprocessing, whether synchronous or asynchronous, immediately adds another layer of complexity to your code. One of the most common mistakes is forgetting to call .get() when you use apply_async(). If you skip it, you’ll never retrieve your result, and your program may finish without ever processing your data.
Another mistake that happens often when using async_apply() is not handling exceptions properly. Always wrap your .get() calls in a try-except block so you can catch and understand any errors that occur inside your worker processes.
Another frequent pitfall that applies to both apply() and async_apply() is not closing the pool properly. Always use a with Pool() block to manage your processes safely, or make sure you call pool.close() followed by pool.join() to clean up.
Conclusion
Key Takeaways
apply()is synchronous and blocks until done.apply_async()is asynchronous, allowing concurrent execution.- Use
apply_asyncfor better performance in parallel workloads. - The use case will guide you when deciding to use
async_applyvs.apply. - Always manage results and exceptions carefully.
Next Steps for Optimizing Multiprocessing
It’s time to take a closer look at your system, since the real power of multiprocessing comes down to how your CPU cores are used. To improve performance, tune the number of processes based on how many cores your machine has so you can get the most out of your hardware without overloading it. It’s also a good idea to profile your workloads to spot any bottlenecks and adjust your approach as needed.
