We typically need to perform some activity when an asyncio task is done.
This may be to collect and store results or to clean up some resource that is no longer needed.
Performing these actions can be challenging given the asynchronous nature of tasks. The solution is to use a callback function, triggered when a task is done.
In this tutorial, you will discover how to use asyncio task done callback functions.
After completing this tutorial, you will know:
- How to add a done callback function to an asyncio task.
- How to add multiple callback functions and how to remove functions that are no longer needed.
- What happens if a callback function fails with an exception or is added after a task is already done.
Let’s get started.
What is an Asyncio Task
An asyncio Task is an object that schedules and independently runs an asyncio coroutine.
It provides a handle on a scheduled coroutine that an asyncio program can query and use to interact with the coroutine.
A Task is an object that manages an independently running coroutine.
— PEP 3156 – Asynchronous IO Support Rebooted: the “asyncio” Module
An asyncio task is represented via an instance of the asyncio.Task class.
A task is created from a coroutine. It requires a coroutine object, wraps the coroutine, schedules it for execution, and provides ways to interact with it.
A task is executed independently. This means it is scheduled in the asyncio event loop and will execute regardless of what else happens in the coroutine that created it. This is different from executing a coroutine directly, where the caller must wait for it to complete.
Tasks are used to schedule coroutines concurrently. When a coroutine is wrapped into a Task with functions like asyncio.create_task() the coroutine is automatically scheduled to run soon
— Coroutines and Tasks
We can create a task using the asyncio.create_task() function.
This function takes a coroutine instance and an optional name for the task and returns an asyncio.Task instance.
Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.
— Coroutines and Tasks
For example:
|
1 2 3 |
... # create and schedule a task task = asyncio.create_task(coro) |
You can learn more about asyncio tasks in the tutorial:
Now that we know about asyncio tasks, let’s look at how we might use done callback functions.
Run loops using all CPUs, download your FREE book to learn how.
How to Use Callback With a Task
We can add a done callback function to a task via the add_done_callback() method.
This method takes the name of a function to call when the task is done.
The callback function must take the Task instance as an argument.
For example:
|
1 2 3 4 5 6 7 |
# done callback function def handle(task): print(task) ... # register a done callback function task.add_done_callback(handle) |
Recall that a task may be done when the wrapped coroutine finishes normally when it returns, when an unhandled exception is raised or when the task is canceled.
The add_done_callback() method can be used to add or register as many done callback functions as we like.
We can also remove or de-register a callback function via the remove_done_callback() function.
For example:
|
1 2 3 |
... # remove a done callback function task.remove_done_callback(handle) |
Now that we know how to add and remove done callback functions, let’s look at some worked examples.
Example of Adding a Task Done Callback Function
We can explore how to add a done callback function to a task.
In this example, we will define a task coroutine that reports a message and sleeps for a moment.
We will then define a main coroutine that we will use as the entry point to the program. It will report a message, create and schedule the task, then add a done callback function to the task that reports a simple message.
The main coroutine then waits for the task to be completed.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of adding a done callback function to a task import asyncio # custom done callback function def callback(task): # report a message print('Task is done') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds the done callback function to the task to be executed when the task is finished.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the done callback function is called by the event loop, reporting a message.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights how a done callback can be added to a task and that it is called automatically when the task is done.
|
1 2 3 4 |
main coroutine started executing the task Task is done main coroutine done |
Next, let’s look at the case where we may want to add more than one done callback function.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Adding More Than One Task Done Callback Function
A task may have more than one done callback function registered to be called when it completes.
This can be achieved by calling the add_done_callback() function for each function to be registered.
We can explore how to register multiple callback functions to be called when a task finishes.
The example below updates the above example to define two callback functions. The first reports a generic message, and the second reports the details of the task itself.
Both callback functions are added to the task after it is created and scheduled, then called automatically once the task is complete.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
# SuperFastPython.com # example of adding more than one done callback function to a task import asyncio # custom done callback function def callback1(task): # report a message print('Task is done') # another custom done callback function def callback2(task): # report a message print(f'Task: {task}') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback1) # add another done callback function task.add_done_callback(callback2) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds both done callback functions to the task. We expect the callbacks to be executed by the asyncio event loop in the order that they are added.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the first done callback function is called by the event loop, reporting a message. Then the second callback function is called, reporting the details of the task itself.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights how we can add multiple done callback functions to a task.
|
1 2 3 4 5 |
main coroutine started executing the task Task is done Task: <Task finished name='Task-2' coro=<task_coroutine() done, defined at ...> result=None> main coroutine done |
Next, let’s look at an example of adding, then removing a done callback function.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Adding and Removing a Task Done Callback Function
A done callback function can be removed from a task.
This may be required if callbacks need to be executed conditionally and in some cases not executed at all.
The example below adds a callback function to a task after it is created but before it is running.
Then, after the task has been running for a moment, the done callback function is removed to ensure it is not executed when the task finishes.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example of adding and removing a task done callback function import asyncio # custom done callback function def callback(task): # report a message print('Task is done') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback) # wait a moment await asyncio.sleep(0.1) # remove the done callback function task.remove_done_callback(callback) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds the done callback function to the task to be executed when the task is finished.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment. This gives the task time to start running.
It then resumes and removes the done callback function from the running task.
The main() coroutine then awaits the task to be completed.
The task completes and the done callback function is not executed, as we intended.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights how a done callback function can be removed from a running task before it is done.
|
1 2 3 |
main coroutine started executing the task main coroutine done |
Next, let’s look at what happens if we attempt to remove a callback that was not added.
Example of Removing a Nonexistent Done Callback Function
It is possible to remove a callback function that was not registered with the task.
This does not cause an error, instead, it has no effect on the task and no indication is provided to the caller that the callback was never registered.
The example below demonstrates this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of removing a nonexistent task done callback function import asyncio # custom done callback function def callback(task): # report a message print('Task is done') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # wait a moment await asyncio.sleep(0.1) # remove a nonexistent done callback function task.remove_done_callback(callback) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then removes a callback from the task that was never added in the first place. This has no effect on the task or on the calling coroutine.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, no done callback function is executed, as we expect.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights that we can attempt to remove a done callback that was never registered with no effect.
|
1 2 3 |
main coroutine started executing the task main coroutine done |
Next, let’s explore what happens if we add a done callback function after a task is done.
Example of Adding a Done Callback Function After the Task is Done
A done callback function can be added after the task has already been completed.
The effect is that the done callback function will execute as soon as it is able.
The example below demonstrates this. The main coroutine creates and schedules a task, then waits for it to complete. Once the task is done, the main coroutine then adds a done callback function.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of adding a done callback function after the task is done import asyncio # custom done callback function def callback(task): # report a message print('Task is done') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # wait for the task to complete await task # add a done callback function task.add_done_callback(callback) # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine. It then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the main() coroutine adds the done callback and then terminates.
Before the asyncio event loop terminates, it executes the done callback function added to the task, reporting its message.
This example highlights that we can add a done callback function to a task after it has been completed.
|
1 2 3 4 |
main coroutine started executing the task main coroutine done Task is done |
Next, let’s take a look at what happens if an exception is raised in a done callback function.
Example of Exception in Task Done Callback Function
A done callback function can raise an exception.
Importantly, an exception raised in a done callback function will not impact the asyncio event loop. It also will not impact the task on which it was added nor any other done callback functions that need to be executed.
We can demonstrate this with a worked example.
In this example, two done callback functions are added to a scheduled task. The first callback function fails with an exception. The second operates normally and is unaffected by the failure of the first callback function.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
# SuperFastPython.com # example of an exception in a done callback function import asyncio # custom done callback function def callback1(task): # report a message print('Task is done') # raise an exception raise Exception('Something bad happened') # another custom done callback function def callback2(task): # report a message print(f'Task: {task}') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback1) # add another done callback function task.add_done_callback(callback2) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds the two done callback functions to the task to be executed when the task is finished.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the first done callback function is called by the event loop, reporting a message and then raising an exception.
The exception is caught by the asyncio event loop and logged using a default logger, in this case to standard output.
The second done callback function is then executed as per normal.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights how a done callback can fail with an exception, and how the exception will be logged by the event loop and does not impact other done callback functions.
|
1 2 3 4 5 6 7 8 9 10 |
main coroutine started executing the task Task is done Exception in callback callback1(<Task finishe...> result=None>) at ... handle: <Handle callback1(<Task finishe...> result=None>) at ...> Traceback (most recent call last): ... Exception: Something bad happened Task: <Task finished name='Task-2' coro=<task_coroutine() done, defined at ...> result=None> main coroutine done |
Next, let’s look at what happens if we register a badly defined done callback function.
Example of Adding a Badly Defined Task Done Callback Function
We can add a done callback function to a task that is badly defined.
Recall that a done callback function must take one argument, which is the task object on which the callback was added.
If a callback function is added to a task that does not have this expected argument, an error will be raised when the asyncio event loop attempts to call it after the task has been completed.
Importantly, this does not break the event loop.
The example below demonstrates this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of adding a badly defined done callback function to a task import asyncio # custom done callback function def callback(): # report a message print('Task is done') # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds the badly defined done callback function to the task to be executed when the task is finished.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the done callback function is called by the event loop.
This fails and an exception is raised and caught by the event loop. The exception is logged using the default logger to standard output.
The exception does not break the event loop.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights how an invalid callback function can be added to a task which will cause a failure that does not impact the asyncio program.
|
1 2 3 4 5 6 7 8 |
main coroutine started executing the task Exception in callback callback(<Task finishe...> result=None>) at ... handle: <Handle callback(<Task finishe...> result=None>) at ...> Traceback (most recent call last): ... TypeError: callback() takes 0 positional arguments but 1 was given main coroutine done |
Next, let’s look at what happens if we attempt to add a coroutine as a done callback function.
Example of Adding a Coroutine as a Task Done Callback Function
A done callback function must be a function.
We cannot add a coroutine as a done callback function. This is because a coroutine cannot be called as a function, instead, it is an object that must be executed by the asyncio event loop.
As such, if we attempt to add a coroutine as a done callback function, then an error will be raised.
We can demonstrate this with a worked example.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
# SuperFastPython.com # example of adding a coroutine as a done callback function import asyncio # custom done callback function async def callback(task): # report a message print('Task is done') # block for a moment await asyncio.sleep(1) # define a coroutine for a task async def task_coroutine(): # report a message print('executing the task') # block for a moment await asyncio.sleep(1) # custom coroutine async def main(): # report a message print('main coroutine started') # create and schedule the task task = asyncio.create_task(task_coroutine()) # add a done callback function task.add_done_callback(callback) # wait for the task to complete await task # report a final message print('main coroutine done') # start the asyncio program asyncio.run(main()) |
Running the example starts the asyncio event loop and executes the main() coroutine.
The main() coroutine reports a message, then creates and schedules the task coroutine.
It then adds the coroutine done callback to the task to be executed when the task is finished.
The main() coroutine then suspends and awaits the task to be completed.
The task runs, reports a message, and sleeps for a moment before terminating normally.
After the task completes, the done callback coroutine is called by the event loop.
The coroutine is not executed.
Additionally, a RuntimeWarning is raised indicating that a coroutine was created but was not executed.
The main() coroutine resumes, reports its own final message and the program ends.
This example highlights that we cannot use a coroutine as a done callback function.
|
1 2 3 4 5 6 |
main coroutine started executing the task ...: RuntimeWarning: coroutine 'callback' was never awaited self._context.run(self._callback, *self._args) RuntimeWarning: Enable tracemalloc to get the object allocation traceback main coroutine done |
Note that if we wanted to execute a coroutine after a task finishes, a done callback function could be added that in turn creates and schedules a task.
For example:
|
1 2 3 4 5 6 |
# custom done callback function def callback(task): # report a message print('Task is done') # add a task asyncio.create_task(coro_callback()) |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to use asyncio task done callback functions in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Focal Point on Unsplash
Leave a Reply