You can develop a helper function to cancel an asyncio task and wait for it to be cancelled.
In this tutorial, you will discover how to cancel a task and wait for it to be cancelled.
Let’s get started.
What is Asyncio Task Cancellation?
Asyncio tasks can be canceled.
This can be achieved by calling the cancel() method on the asyncio.Task. This will request that the task cancel as soon as possible and returns True if the request was successful or False if it was not, e.g. the task is already done.
Request the Task to be cancelled. This arranges for a CancelledError exception to be thrown into the wrapped coroutine on the next cycle of the event loop.
— Coroutines and Tasks
For example:
|
1 2 3 4 |
... # cancel a task if task.cancel(): # request successful |
The next time that the target task that was cancelled resumes, a CancelledError exception will be raised.
This exception will bubble up to the top level of the task and cause it to stop running.
You can learn more about the mechanics of cancellation a task in the tutorial:
Run loops using all CPUs, download your FREE book to learn how.
Need to Cancel Task And Wait
We may need to cancel a task and then wait for the target task to be done.
The caller that cancels the target task needs to wait and know that the target task is cancelled before resuming.
It assumes that cancellation of the target task is a blocking call, and that the request for cancellation and the completion of cancellation is synchronous.
This may be for many reasons, such as:
- The target task is sensitive and must be cancelled for safety reasons.
- The application is unable to progress until the target task is in the done state.
- The target task has resources required by the calling task.
Therefore, we need a consistent way to cancel a target task and then await that task to cancel.
This is a common pattern, typically referred to as “cancel and wait” or “cancel and await“.
Note, if behavior is required and does not need to be synchronous, then a done callback function can be used to execute some behavior when the target task is indeed cancelled. This should generally be preferred if possible.
How to Cancel Task and Wait
Canceling a task and waiting is relatively straightforward.
Nevertheless, the closer you look at this pattern, the more important details surface.
Let’s look at how we might work through this pattern.
Naive Cancel And Wait
A naive way to implement this pattern is to call the cancel() method on the target task and then await the task directly.
This would require handling an asyncio.CancelledError that is expected to bubble up from the target task.
For example, we can implement this in a simple helper function:
|
1 2 3 4 5 6 7 8 9 10 |
# cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # the target was canceled, perhaps log pass |
This is straightforward and is probably how most asyncio developers implement the cancel and await pattern.
It is even suggested in the asyncio API documentation for the Task.cancel() method.
One problem with this approach is that it may consume requests of the cancel task to cancel.
Better Cancel And Wait
In the above example, the helper function handles the asyncio.CancelledError expected to bubble up from the cancelled target task.
As is, this could be a problem if the caller, the task calling cancel(), is itself cancelled while waiting on the target task to cancel.
If the caller is cancelled while waiting, it will resume after the await, and raise an asyncio.CancelledError, which will be consumed and handled by the try-except.
This may have the undesirable effect of consuming or shielding the caller from cancellation.
A solution is to have the helper function check for this case when handling the asyncio.CancelledError.
For example, it can check if the current task was canceled, and if so re-raise the exception to allow the caller of the cancel_and_wait() to handle the cancellation request normally.
This can be achieved by first calling the asyncio.current_task() function to get access to the current task object, then calling the cancelling() method which will return the number of cancel requests the task has received.
If this value is greater than zero, it can re-raise the CancelledError.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # check if the current task was cancelled if asyncio.current_task().cancelling() > 0: raise else: # the target was canceled, perhaps log pass |
The cancel_and_wait() helper will return normally if the target is done and raise a CancelledError only if the caller has a request to cancel.
This is better, but we can change it further.
The issue comes down to the contact with the Task.cancel() method and our expectations.
Even Better Cancel And Wait
The cancel() method is that it will request the target task to cancel.
The expectation of a cancelled task is that it will cancel at some point by raising a CancelledError. Awaiting a cancelled task is therefore expected to raise a CancelledError.
Put another way, a canceled task not raising a CancelledError is unexpected behavior.
… suppressing cancellation completely is not common and is actively discouraged.
— Coroutines and Tasks
We can make our helper function better by enforcing this expectation, and flip the outcomes of the previous helper function.
Note, I did not come to this solution myself, instead, I learned about it from Evgeny Osipenko in his discussion and sample code in the GitHub Issue titled “Safe synchronous cancellation in asyncio“, April 2023.
I propose adding a function to asyncio to cancel a task and then safely wait for the cancellation to complete.
— Safe synchronous cancellation in asyncio.
We can update our helper function to enforce this expectation in a few ways:
- If the target task does not raise a CancelledError, we can treat this as a RuntimeError (our code is broken).
- If the target task raises a CancelledError, we can let it bubble up to be handled by the caller.
- If the calling task is canceled, do nothing and just return.
Tying this together, the updated helper with these changes is listed below (it’s Evgeny’s code snippet, I just added comments).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # check that the current task is not cancelled if asyncio.current_task().cancelling() == 0: # bubble the exception up raise else: # the current task is canceling return else: # an expected CancelledError was not seen raise RuntimeError("Cancelled task did not end with an exception") |
Not bad.
This means that the normal case is that the target is cancelled and the helper awaits until the target is done, after which the CancelledError is bubbled up and can be handled normally, e.g. by logging and getting on with things.
For example:
|
1 2 3 4 5 6 |
... # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: # task was canceled as expected |
The unexpected case that the target task does not raise a CancelledError when canceled, causes a program fault in the case of a RuntimeError.
The unusual case that the caller is cancelled, such as while awaiting the target task, results in a normal return, which can be checked.
For example:
|
1 2 3 4 5 6 7 8 |
... # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: # task was canceled as expected else: # current task was cancelled |
We can go even further of course, but not now.
See the cancel() function developed by Jason Fried and used at Meta, and the associated discussion. It’s pretty cool.
Now that we know how to reasonably safely cancel a target task, let’s look at some worked examples.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Cancel and Wait
We can explore a simple case of canceling a long-running task and waiting for it to cancel, using our new shiny helper function developed above.
In this case, we will develop a long-running task that sleeps for 5 seconds and then reports a message.
|
1 2 3 4 5 6 |
# task that take a long time async def work(): # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') |
We can then schedule this task and allow it to begin, then cancel and wait, and handle the expected CancelledError.
|
1 2 3 4 5 6 7 8 9 10 11 |
# main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to run a moment await asyncio.sleep(1) # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: print('Target is closed') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
# SuperFastPython.com # example of cancel and wait helper import asyncio # cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # check that the current task is not cancelled if asyncio.current_task().cancelling() == 0: # bubble the exception up raise else: # the current task is canceling return else: # an expected CancelledError was not seen raise RuntimeError("Cancelled task did not end with an exception") # task that take a long time async def work(): # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to run a moment await asyncio.sleep(1) # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: print('Target is cancelled') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The work() task is created and scheduled and the main() coroutine suspends and allows it to begin.
The work() task runs and suspends on a call to sleep.
The main() coroutine runs and awaits the cancel_and_wait() helper coroutine, suspending until the target task is cancelled.
The cancel_and_wait() helper coroutine runs and calls cancel() on the target task. It then awaits the target task, suspending until it is done.
The work() task resumes and raises a CancelledError. This bubbles up and causes the work() task to terminate.
The CancelledError bubbles up through the cancel_and_wait() coroutine and is handled. The caller has not been requested to cancel, therefore the CancelledError is re-raised. It bubbles up and causes the cancel_and_wait() to terminate.
The main() coroutine resumes and the CancelledError is handled, reporting a message.
This highlights how we can cancel a target task and wait for it to be canceled, in the normal case.
|
1 |
Target is cancelled |
Next, let’s explore an example where the target task consumes the CancelledError.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Cancel and Wait, Target Suppresses CancelledError
We can explore an example where the target task unexpectedly consumes the CancelledError when cancelled.
This does not meet the expected contact for canceled tasks enforced by our helper coroutine and will raise a RuntimeError exception.
In this case, we can update our work() coroutine from the previous section to handle the CancelledError and not re-raise it, effectively suppressing it.
|
1 2 3 4 5 6 7 8 9 |
# task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') except asyncio.CancelledError: pass |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# SuperFastPython.com # example of cancel and wait where target suppresses CancelledError import asyncio # cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # check that the current task is not cancelled if asyncio.current_task().cancelling() == 0: # bubble the exception up raise else: # the current task is canceling return else: # an expected CancelledError was not seen raise RuntimeError("Cancelled task did not end with an exception") # task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') except asyncio.CancelledError: pass # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to run a moment await asyncio.sleep(1) # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: print('Target is cancelled') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The work() task is created and scheduled and the main() coroutine suspends and allows it to begin.
The work() task runs and suspends on a call to sleep.
The main() coroutine runs and awaits the cancel_and_wait() helper coroutine, suspending until the target task is cancelled.
The cancel_and_wait() helper coroutine runs and calls cancel() on the target task. It then awaits the target task, suspending until it is done.
The work() task resumes and raises a CancelledError, which is handled and suppressed.
The cancel_and_wait() helper coroutine resumes and the CancelledError is not observed as it expects. Therefore a RuntimeError is raised, terminating the coroutine.
The RuntimeError bubbles up and terminates the asyncio event loop.
This highlights how we can enforce the expectation on tasks to raise a CancelledError when cancelled.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Traceback (most recent call last): File "...", line 47, in <module> asyncio.run(main()) File ".../asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File ".../asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "...", line 42, in main await cancel_and_wait(task) File "...", line 22, in cancel_and_wait raise RuntimeError("Cancelled task did not end with an exception") RuntimeError: Cancelled task did not end with an exception |
Next, let’s look at an example of the caller task being cancelled while awaiting the canceled target task.
Example of Cancel and Wait, Caller is Cancelled
We can explore the case where the current calling task was cancelled while waiting for the target task to cancel.
In this case, we can update the work() task to take a reference to the current task.
When the work() task is cancelled, it can handle the CancelledError, cancel the provided task, then re-raised the CancelledError.
The updated work() coroutine is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# task that take a long time async def work(other): try: # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') except asyncio.CancelledError: # request that the other task cancels other.cancel() # re-raise the cancellation exception raise |
We can then create the work() coroutine and pass the current coroutine (the caller) as an argument.
|
1 2 3 4 5 |
... # create the work coroutine coro = work(asyncio.current_task()) # create and schedule work as a task task = asyncio.create_task(coro) |
Finally, we can handle the case that the current task was cancelled while waiting for the target task to cancel.
|
1 2 3 4 5 6 7 8 |
... # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: print('Target is cancelled') else: print('Current task was cancelled') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
# SuperFastPython.com # example of cancel and wait where the caller is cancelled import asyncio # cancel task and wait for it to complete async def cancel_and_wait(task, msg=None): # cancel the task task.cancel(msg) try: # wait for the task to be done await task except asyncio.CancelledError: # check that the current task is not cancelled if asyncio.current_task().cancelling() == 0: # bubble the exception up raise else: # the current task is canceling return else: # an expected CancelledError was not seen raise RuntimeError("Cancelled task did not end with an exception") # task that take a long time async def work(other): try: # sleep a long time await asyncio.sleep(5) # report a message print('Task sleep completed normally') except asyncio.CancelledError: # request that the other task cancels other.cancel() # re-raise the cancellation exception raise # main coroutine async def main(): # create the work coroutine coro = work(asyncio.current_task()) # create and schedule work as a task task = asyncio.create_task(coro) # allow the task to run a moment await asyncio.sleep(1) # cancel and wait for task to be done try: await cancel_and_wait(task) except asyncio.CancelledError: print('Target is cancelled') else: print('Current task was cancelled') # start the event loop asyncio.run(main()) |
Running the example starts the event loop and runs the main() coroutine.
The work() task is created and passed as a reference to the current task. It is scheduled and the main() coroutine suspends and allows it to begin.
The work() task runs and suspends on a call to sleep.
The main() coroutine runs and awaits the cancel_and_wait() helper coroutine, suspending until the target task is cancelled.
The cancel_and_wait() helper coroutine runs and calls cancel() on the target task. It then awaits the target task, suspending until it is done.
The work() task resumes and raises a CancelledError which is handled and calls cancel on the calling task. It then allows the CancelledError to bubble up.
The CancelledError bubbles up through the cancel_and_wait() coroutine and is handled, or it raises its own CancelledError. Either way, both CancelledError exceptions are handled.
The caller has been requested to cancel, therefore it returns directly.
- The bubbled-up CancelledError is consumed.
- The CancelledError raised by itself (or potentially raised) is consumed.
The main() coroutine resumes and a CancelledError exception was not raised, executing the else clause and reporting a message.
|
1 |
Current task was cancelled |
This highlights how we can detect and handle the case of the caller being canceled while awaiting a target task being cancelled.
At the very least, this could give ideas for writing unit tests on our own code.
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to cancel and wait for a task in asyncio.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Thuận Minh on Unsplash
Leave a Reply