You can traverse an asynchronous generator or asynchronous iterator using an asynchronous comprehension via the “async for” expression.
In this tutorial, you will discover how to use asynchronous comprehensions and await comprehensions in Python.
Let’s get started.
What are Asynchronous Comprehensions
An async comprehension is an asynchronous version of a classical comprehension.
Asyncio supports two types of asynchronous comprehensions, they are the “async for” comprehension and the “await” comprehension.
PEP 530 adds support for using async for in list, set, dict comprehensions and generator expressions
— PEP 530: Asynchronous Comprehensions, What’s New In Python 3.6
Before we look at each, let’s first recall classical comprehensions.
Comprehensions
Comprehensions allow data collections like lists, dicts, and sets to be created in a concise way.
List comprehensions provide a concise way to create lists.
— List Comprehensions
A list comprehension allows a list to be created from a for expression within the new list expression.
For example:
|
1 2 3 |
... # create a list using a list comprehension result = [a*2 for a in range(100)] |
Comprehensions are also supported for creating dicts and sets.
For example:
|
1 2 3 4 5 |
... # create a dict using a comprehension result = {a:i for a,i in zip(['a','b','c'],range(3))} # create a set using a comprehension result = {a for a in [1, 2, 3, 2, 3, 1, 5, 4]} |
Asynchronous Comprehensions
An asynchronous comprehension allows a list, set, or dict to be created using the “async for” expression with an asynchronous iterable.
We propose to allow using async for inside list, set and dict comprehensions.
— PEP 530 – Asynchronous Comprehensions
For example:
|
1 2 3 |
... # async list comprehension with an async iterator result = [a async for a in aiterable] |
This will create and schedule coroutines or tasks as needed and yield their results into a list.
Recall that the “async for” expression may only be used within coroutines and tasks.
You can learn more about the async for expression in the tutorial:
Also, recall that an asynchronous iterator is an iterator that yields awaitables.
asynchronous iterator: An object that implements the __aiter__() and __anext__() methods. __anext__ must return an awaitable object. async for resolves the awaitables returned by an asynchronous iterator’s __anext__() method until it raises a StopAsyncIteration exception.
— PYTHON GLOSSARY
The “async for” expression allows the caller to traverse an asynchronous iterator of awaitables and retrieve the result from each.
Internally, the async for loop will automatically resolve or await each awaitable, scheduling coroutines as needed.
An async generator automatically implements the methods for the async iterator and may also be used in an asynchronous comprehension.
asynchronous generator: A function which returns an asynchronous generator iterator.
— PYTHON GLOSSARY
For example:
|
1 2 3 |
... # async list comprehension with an async generator result = [a async for a in agenerator] |
Await Comprehensions
The “await” expression may also be used within a list, set, or dict comprehension, referred to as an await comprehension.
We propose to allow the use of await expressions in both asynchronous and synchronous comprehensions
— PEP 530 – Asynchronous Comprehensions
Like an async comprehension, it may only be used within an asyncio coroutine or task.
This allows a data structure, like a list, to be created by suspending and awaiting a series of awaitables.
For example:
|
1 2 3 |
... # await list compression with a collection of awaitables results = [await a for a in awaitables] |
This will create a list of results by awaiting each awaitable in turn.
The current coroutine will be suspended to execute awaitables sequentially, which is different and perhaps slower than executing them concurrently using asyncio.gather().
Now that we know what async and await comprehensions are, let’s look at some examples.
Run loops using all CPUs, download your FREE book to learn how.
Example of Async List Comprehension With Generator
We can explore how to use an asynchronous list comprehension with an asynchronous generator.
In this example, we will define an asynchronous generator that will sleep for one second each iteration to simulate some blocking work and yield an integer.
You may recall that an asynchronous generator is a coroutine that yields a value when iterated.
We will then populate a list using the asynchronous generator with an asynchronous list comprehension that makes use of the “async for” expression.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of an asynchronous list comprehension with an generator import asyncio # define an asynchronous generator async def async_generator(): # normal loop for i in range(10): # block to simulate doing work await asyncio.sleep(1) # yield the result yield i # main coroutine async def main(): # asynchronous list comprehension results = [item async for item in async_generator()] # report results print(results) # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.
The main() coroutine runs and iterates the asynchronous generator in an asynchronous list comprehension using the “async for” expression, intended for asynchronous iterators, of which the generator is one example.
A lot is happening on this one compact line.
The generator is created and the “async for” expression schedules the first iteration and then suspends the main() coroutine. The generator executes the first iteration of the loop, sleeps, then yields a single value. The main() coroutine resumes, retrieves one value, and appends it to the list.
The main() coroutine then iterates the generator and is suspended. The generator resumes from where it yielded and performs another iteration of the loop and yields a value. The process continues until no further iterations are possible.
Each iteration of the asynchronous generator schedules a coroutine to resume the iterator where it left off last time.
The result list is populated and is then reported.
This highlights how we can use an asynchronous list comprehension with an asynchronous generator via the “async for” expression.
|
1 |
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
Next, we will look at how to use an asynchronous list comprehension with an asynchronous iterator.
Example of Asynchronous List Comprehension With Iterator
We can explore how to use an asynchronous list comprehension with an asynchronous iterator.
In this example, we will define an asynchronous iterator object. The iterator will maintain a counter and will sleep for a second and return the value of the counter each time it is iterated.
You may recall that an asynchronous iterator is an object that implements the __aiter__() and __anext__() methods.
We will then populate a list using the asynchronous iterator with an asynchronous list comprehension that makes use of the “async for” expression.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# SuperFastPython.com # example of an asynchronous list comprehension with an iterator import asyncio # define an asynchronous iterator class AsyncIterator(): # constructor, define some state def __init__(self): self.counter = 0 # create an instance of the iterator def __aiter__(self): return self # return the next awaitable async def __anext__(self): # check for no further items if self.counter >= 10: raise StopAsyncIteration # block to simulate work await asyncio.sleep(1) # increment the counter self.counter += 1 # return the counter value return self.counter # main coroutine async def main(): # asynchronous list comprehension results = [item async for item in AsyncIterator()] # report results print(results) # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.
The main() coroutine runs and iterates the asynchronous iterator in an asynchronous list comprehension using the “async for” expression.
As with the previous example, a lot is happening on this one compact line.
The iterator object is first created. It is then used in the “async for” expression which schedules the first iteration and then suspends the main() coroutine. The iterator executes the __anext__() function and the first iteration of the loop, sleeps, increments the counter, then returns a single value. The main() coroutine resumes, retrieves one value, and appends it to the list.
The main() coroutine then steps the iterator again and is suspended. The iterator executes the __anext__() function again and performs another iteration of the loop and returns a value. The process continues until the counter is equal to 10 and a StopAsyncIteration exception is raised, terminating the iteration.
The result list is populated and is then reported.
This highlights how we can use an asynchronous list comprehension with an asynchronous iterator object via the “async for” expression.
|
1 |
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10] |
Next, we will look at how to use an await list comprehension.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Await List Comprehension
We can explore how to use an await list comprehension with coroutines.
In this example, we will define a coroutine that takes a value as an argument, sleeps, and returns a multiple of the argument.
We will then create a list of these coroutines and then populate a list with their return values, awaiting each, in turn, using an await comprehension.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of an await list comprehension import asyncio # task coroutine async def task_coro(value): # block a moment to simulate work await asyncio.sleep(1) # calculate and return return value * 10 # main coroutine async def main(): # create a list of awaitables awaitables = [task_coro(i) for i in range(10)] # await list comprehension to collect results results = [await a for a in awaitables] # report results print(results) # run the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.
The main() coroutine runs and first creates a list of coroutine objects using a classical list comprehension.
It then executes the list of coroutines using an await list comprehension. In each iteration of this comprehension, the main() coroutine is suspended and the next coroutine is scheduled and executed, rerunning a value that is appended to the result list.
This process continues until all coroutines in the list of awaitables have been awaited.
This example highlights how we can use an await list comprehension to collect results from multiple awaitables, like coroutines.
We can see that the coroutines are executed one by one sequentially, rather than concurrently. This suggests that using an await comprehension may only be appropriate when there is a linear dependency between the tasks.
An alternative would be to execute all awaitables concurrently, then collect their results.
|
1 |
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90] |
Next, we will look at a faster alternative to using an await comprehension.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of an Alternative to an Await Comprehension: asyncio.gather()
We can explore how to use an alternative to an await comprehension.
In this example, we will update the above example that populates a list with the return values from multiple coroutines by awaiting each.
Instead of using an await comprehension, we will use the asyncio.gather() function that will schedule and execute all coroutines in the list concurrently, rather than one at a time sequentially.
You can learn more about the asyncio.gather() function in the tutorial:
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of an alternative to an await list comprehension import asyncio # task coroutine async def task_coro(value): # block a moment to simulate work await asyncio.sleep(1) # calculate and return return value * 10 # main coroutine async def main(): # create a list of awaitables awaitables = [task_coro(i) for i in range(10)] # gather the awaitables concurrently results = await asyncio.gather(*awaitables) # report results print(results) # run the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and runs it as the entry point into the asyncio program.
The main() coroutine runs and first creates a list of coroutine objects using a classical list comprehension.
It then executes the list of coroutines concurrently using the asyncio.gather() function.
This function takes awaitables as positional arguments. We can unpack the list of coroutines into separate expressions using the star expression (*).
In this case, all ten coroutines execute concurrently and their results are collected into the result list, which is then reported.
This differs from the previous example that used an await comprehension and executed each awaitable one at a time, sequentially.
This highlights the asyncio.gather() function as an alternative to an await comprehension that can be used when there is no linear dependence between the awaitables being executed.
|
1 |
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90] |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to use async comprehensions and await comprehensions in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Wes Tindel on Unsplash
Leave a Reply