Last Updated on September 12, 2022
You can cancel tasks in the ProcessPoolExecutor by calling the cancel() function on the Future task.
In this tutorial you will discover how to cancel tasks in a Python process pool.
Let’s get started.
Need to Cancel Tasks in the Process Pool
The ProcessPoolExecutor in Python provides a pool of reusable processes for executing ad hoc tasks.
You can submit tasks to the process pool by calling the submit() function and passing in the name of the function you wish to execute on another process.
Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.
You may submit tasks that you may later decide you no longer need to complete.
Letting the tasks run may use up unwanted resources.
Additionally, you may have to wait for all tasks in the process pool to complete before you can shut it down.
The solution is to cancel unwanted tasks.
Run loops using all CPUs, download your FREE book to learn how.
How to Cancel Submitted Tasks
You can cancel tasks submitted to the ProcessPoolExecutor by calling the cancel() function on the Future object.
Recall that you will receive a Future object when you submit your task to the process pool when you call the submit() function.
|
1 2 3 |
... # submit a task to the process pool future = executor.submit(work) |
The Future object has a function called cancel() that will cancel the task if it has not yet started running and is not done, e.g. is pending.
|
1 2 3 |
... # cancel a task submitted to the process pool was_canceled = future.cancel() |
The cancel() function returns True if the task was canceled, otherwise False if the task could not be canceled.
If you successfully cancel the task, it will not execute.
If you try to retrieve a result from a cancelled task by calling the result() function, a CancelledError will be raised. Similarly, if you try to get any exception raised while executing the task that was cancelled by calling the exception() function, a CancelledError will be raised.
For example:
|
1 2 3 4 5 6 |
... # get the result try: result = future.result() except CancelledError: print('No result, task was cancelled') |
You may fail to cancel a task submitted to the process pool for one of three reasons:
- The task was already cancelled.
- The task is currently running.
- The task is done.
There is often some confusion with the difference between a task that has been submitted and a task that is running.
A ProcessPoolExecutor has a fixed number of processes, limiting the number of tasks that can be executed at once.
A task that is submitted to the ProcessPoolExecutor may not start running immediately if all processes in the pool are currently occupied. Once a process in the pool becomes available, it will take a task from the internal queue of tasks and begin executing it, changing its status to “running“.
We can check if a task is running by calling the running() function on the Future object for the task.
Only tasks that are scheduled in the process pool but have not yet started running can be cancelled.
Now that we know how to cancel tasks in a ProcessPoolExecutor, let’s look at a worked example.
Example of Cancelling a Task
Let’s demonstrate how to cancel a task submitted to the ProcessPoolExecutor.
First, we can define a simple task that will block for a given number of seconds.
|
1 2 3 |
# task that will block for a moment def work(sleep_time): sleep(sleep_time) |
We can then create a process pool with one process and submit a long running task to occupy the single process in the pool.
|
1 2 3 4 5 6 |
... # create a process pool with ProcessPoolExecutor(1) as executor: # start a long running task future1 = executor.submit(work, 2) print(f'First task running={future1.running()}') |
Next, we can submit a second shorter task and confirm that it is not yet running.
|
1 2 3 4 |
... # start a second future2 = executor.submit(work, 0.1) print(f'Second task running={future2.running()}') |
We can then cancel the second task, and confirm that it indeed was canceled and will not run.
|
1 2 3 4 |
... # cancel the second task was_cancelled = future2.cancel() print(f'Second task was cancelled: {was_cancelled}') |
Tying this together, the complete example of cancelling a task in the ProcessPoolExecutor is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of cancelling a task in a process pool from time import sleep from concurrent.futures import ProcessPoolExecutor # task that will block for a moment def work(sleep_time): sleep(sleep_time) # entry point if __name__ == '__main__': # create a process pool with ProcessPoolExecutor(1) as executor: # start a long running task future1 = executor.submit(work, 2) print(f'First task running={future1.running()}') # start a second future2 = executor.submit(work, 0.1) print(f'Second task running={future2.running()}') # cancel the second task was_cancelled = future2.cancel() print(f'Second task was cancelled: {was_cancelled}') |
Running the example first starts the long running task and confirms that it is running.
The second short duration task is submitted and is not yet running.
We then cancel the second task and confirm that it will not run.
|
1 2 3 |
First task running=True Second task running=False Second task was cancelled: True |
Free Python ProcessPoolExecutor Course
Download your FREE ProcessPoolExecutor PDF cheat sheet and get BONUS access to my free 7-day crash course on the ProcessPoolExecutor API.
Discover how to use the ProcessPoolExecutor class including how to configure the number of workers and how to execute tasks asynchronously.
No Results from Cancelled Tasks
You cannot get results or exceptions from cancelled tasks.
If we try to get a result from a cancelled task by calling the result() function, a CancelledError will be raised. The same would be the case if we tried to get the exception by calling the exception() function.
We can demonstrate this by updating the example from the previous section.
First, let’s update the target task function to return a value so we have a result to retrieve from the task. In this case, the argument that was passed in.
|
1 2 3 4 |
# task that will block for a moment def work(sleep_time): sleep(sleep_time) return sleep_time |
Next, we can attempt to get the result from the cancelled task (the short duration task) and handle the expected CancelledError that will be raised because the task was cancelled.
|
1 2 3 4 5 6 |
... # get the result try: result = future2.result() except CancelledError: print('No result, task was cancelled') |
Finally, we can try and get the exception from the cancelled task and handle the same expected CancelledError because the task was cancelled.
|
1 2 3 4 5 6 |
... # get the exception try: e = future2.exception() except CancelledError: print('No exception, task was cancelled') |
Tying this together, the complete example of attempting to get a result and an exception from a cancelled task is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of getting results and an exception from a cancelled task from time import sleep from concurrent.futures import ProcessPoolExecutor from concurrent.futures import CancelledError # task that will block for a moment def work(sleep_time): sleep(sleep_time) return sleep_time if __name__ == '__main__': # create a process pool with ProcessPoolExecutor(1) as executor: # start a long running task future1 = executor.submit(work, 2) print(f'First task running={future1.running()}') # start a second future2 = executor.submit(work, 0.1) print(f'Second task running={future2.running()}') # cancel the second task was_cancelled = future2.cancel() print(f'Second task was cancelled: {was_cancelled}') # get the result try: result = future2.result() except CancelledError: print('No result, task was cancelled') # get the exception try: e = future2.exception() except CancelledError: print('No exception, task was cancelled') |
Running the example first starts the long running task and confirms that it is running. The second short duration task is submitted, then successfully cancelled.
We then try to retrieve the result from the cancelled task and handle an expected CancelledError.
Finally, we try to retrieve any exception that occurred during the execution of the short duration task and a CancelledError is raised and handled, again because the task was cancelled.
|
1 2 3 4 5 |
First task running=True Second task running=False Second task was cancelled: True No result, task was cancelled No exception, task was cancelled |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Books
- ProcessPoolExecutor Jump-Start, Jason Brownlee (my book!)
- Concurrent Futures API Interview Questions
- ProcessPoolExecutor PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Effective Python, Brett Slatkin, 2019.
- See Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ProcessPoolExecutor: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Multiprocessing: The Complete Guide
- Python Pool: The Complete Guide
APIs
References
- Thread (computing), Wikipedia.
- Process (computing), Wikipedia.
- Thread Pool, Wikipedia.
- Futures and promises, Wikipedia.
Takeaways
You now know how to cancel tasks that have been submitted to the Python ProcessPoolExecutor.
Do you have any questions?
Ask your question in the comments below and I will do my best to answer.
Photo by Margo Brodowicz on Unsplash
Leave a Reply