You can benchmark asyncio programs using the loop.time() method on the asyncio event loop.
The asyncio event loop maintains an internal monotonic timer. It is mostly used for managing timeouts and delays. We can retrieve the current time maintained by asyncio via the loop.time() method.
In this tutorial, you will discover how to benchmark asyncio programs using the loop.time() method.
Let’s get started.
Need To Benchmark Asyncio Programs
Benchmarking Python code refers to comparing the performance of one program to variations of the program.
Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
— Benchmarking, Wikipedia.
Typically, we make changes to the programs, such as adding concurrency, in order to improve the performance of a program on a given system.
Improving performance typically means reducing the run time of the program.
Therefore, when we benchmark programs in Python after adding concurrency, we typically are interested in recording how long a program takes to run.
It is critical to be systematic when benchmarking code.
The first step is to record how long an unmodified version of the program takes to run. This provides a baseline in performance to which all other versions of the program must be compared. If we are adding concurrency, then the unmodified version of the program will typically perform tasks sequentially, e.g. one-by-one.
We can then make modifications to the program, such as adding thread pools, process pools, or asyncio. The goal is to perform tasks concurrently (out of order), even in parallel (simultaneously). The performance of the program can be benchmarked and compared to the performance of the unmodified version.
The performance of the modified versions of the program must have better performance than the unmodified version of the program. If they do not, they are not improvements and should not be adopted.
How can we benchmark the performance of asyncio programs in Python?
Run loops using all CPUs, download your FREE book to learn how.
What is loop.time()
The asyncio event loop maintains an internal timer.
This timer is monotonic, meaning that whenever the time is retrieved it is always equal to or greater than the last time that was retrieved. The underlying clock cannot be adjusted backward in time via time synchronization or daylight savings.
We can access the time maintained by the event loop via the time() method on the event loop object.
Return the current time, as a float value, according to the event loop’s internal monotonic clock.
— Asyncio Event Loop
This means that we must first retrieve the event loop object, and then call the time() method to get the time.
We can get the object for the current event loop via the asyncio.get_running_loop().
For example:
|
1 2 3 4 5 |
... # get the event loop loop = asyncio.get_running_loop() # get the current event loop time time = loop.time() |
If we want to retrieve the time many times in our program, such as during benchmarking, we could reduce this to a single compound statement.
For example:
|
1 2 3 |
... # get the current event loop time time = asyncio.get_running_loop().time() |
The loop.time() method is implemented using the time.monotonic() function.
We can confirm this if we review the source code for the BaseEventLoop class and specifically the code for the time() method, listed below.
|
1 2 3 4 5 6 7 8 |
def time(self): """Return the time according to the event loop's clock. This is a float expressed in seconds since an epoch, but the epoch, precision, accuracy and drift are unspecified and may differ per event loop. """ return time.monotonic() |
Recall that the time.monotonic() function provides a timer that uses a system-wide monotonic clock that is not adjustable.
It is used in the Python standard library generally for timeouts and delays, much as the loop.time() method is used in the asyncio event loop.
You can learn more about the time.monotonic() function for benchmarking in the tutorial:
The comment for the default implementation of the time() method is telling.
It reminds us that the implementation may differ from event loop to event loop, and we should not rely on the default properties of time.monotonic(), e.g. the precision, the epoch used, and the fact that it’s not adjusted.
Now that we know about the loop.time() method, let’s look at how we can use it to benchmark our asyncio programs.
How to Benchmark with loop.time()
We can use the loop.time() method to benchmark asyncio programs.
There are perhaps 3 case studies we may want to consider, they are:
- Benchmarking a statement.
- Benchmarking a coroutine.
- Benchmarking an asyncio program.
Let’s look at how we can benchmark using the loop.time() method.
How to Benchmark a Python Statement
We can use the loop.time() method to benchmark arbitrary Python statements.
The procedure is as follows:
- Record loop.time() before the statement.
- Execute the statement.
- Record loop.time() after the statement.
- Subtract start time from after time to give duration.
- Report the duration using print().
For example:
|
1 2 3 4 5 6 7 8 9 10 |
# record start time time_start = asyncio.get_running_loop().time() # execute the statement ... # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') |
How to Benchmark an Asyncio Coroutine
We can use the loop.time() method to benchmark arbitrary asyncio coroutines.
The procedure is as follows:
- Record loop.time() before the coroutine.
- Await the coroutine.
- Record loop.time() after the coroutine.
- Subtract start time from after time to give duration.
- Report the duration using print().
For example:
|
1 2 3 4 5 6 7 8 9 10 |
# record start time time_start =asyncio.get_running_loop().time() # await the coroutine await custom_coro() # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') |
How to Benchmark an Asyncio Program
We can use the loop.time() method to benchmark arbitrary asyncio programs.
Typically, the entry point to an asyncio program is a coroutine called main(). We can add the benchmarking code to the beginning and end of this coroutine.
This coroutine can then be created and passed to the asyncio.run() to start the event loop and run the asyncio program.
The procedure is as follows:
- Record loop.time() at the beginning of the main() coroutine
- Run the body of the main() coroutine
- Record loop.time() at the end of the main() coroutine
- Subtract start time from after time to give duration.
- Report the duration using print().
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
... # define main coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # execute the program ... # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') # start the event loop asyncio.run(main()) |
Now that we know how to benchmark Python code using loop.time(), let’s look at some worked examples.
Free Python Benchmarking Course
Get FREE access to my 7-day email course on Python Benchmarking.
Discover benchmarking with the time.perf_counter() function, how to develop a benchmarking helper function and context manager and how to use the timeit API and command line.
Example of Benchmarking a Statement with loop.time()
We can explore how to benchmark a Python statement using loop.time() in an asyncio program with a worked example.
In this example, we will define a statement that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.
|
1 2 3 |
... # execute the statement data = [i*i for i in range(100000000)] |
We will then surround this statement with benchmarking code.
Firstly, we will record the start time using the loop.time() function.
|
1 2 3 |
... # record start time time_start = asyncio.get_running_loop().time() |
Afterward, we will record the end time, calculate the overall execution duration, and report the result.
|
1 2 3 4 5 6 7 |
... # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# SuperFastPython.com # example of benchmarking a statement with loop.time() import asyncio # define coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # execute the statement data = [i*i for i in range(100000000)] # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') # run the coroutine asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and records the start time using the event loop time.
Next, the Python statement is executed, in this case creating a list of 100 million squared integers.
The end time is then recorded, using the event loop time.
The difference between the two recorded times is calculated, providing the statement execution duration in seconds.
Finally, the result is reported, truncated to three decimal places (milliseconds).
In this case, we can see that the statement took about 5.139 seconds to complete.
Note, that the results on your system may vary.
This highlights how we can benchmark a Python statement in an asyncio program using the loop.time() method.
|
1 |
Took 5.139 seconds |
Next, let’s explore an example of benchmarking a coroutine using the loop.time() function.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Benchmarking a Coroutine with loop.time()
We can explore how to benchmark an asyncio coroutine using loop.time() with a worked example.
In this example, we will define a coroutine that creates a list of 100 million squared integers in a list comprehension, which should take a number of seconds.
|
1 2 3 4 |
# our work coroutine async def work(): # do some CPU bound work data = [i*i for i in range(100000000)] |
We will then await this coroutine, and surround the call with benchmarking code.
Firstly, we will record the start time using the loop.time() function.
|
1 2 3 |
... # record start time time_start = asyncio.get_running_loop().time() |
Afterward, we will record the end time, calculate the overall execution duration, and report the result.
|
1 2 3 4 5 6 7 |
... # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of benchmarking a coroutine import time import asyncio # our work coroutine async def work(): # do some CPU bound work data = [i*i for i in range(100000000)] # define coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # suspend and await the coroutine await work() # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') # run the coroutine asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and records the start time using the event loop time.
Next, the work() is awaited. This suspends the main() coroutine and executes the work() coroutine, in this case creating a list of 100 million squared integers.
The main() coroutine resumes and the end time is then recorded using the event loop time.
The difference between the two recorded times is calculated, providing the coroutine execution duration in seconds.
Finally, the result is reported, truncated to three decimal places (milliseconds).
In this case, we can see that the function took about 6.273 seconds to complete.
Note, that the results on your system may vary.
This highlights how we can benchmark an asyncio coroutine using the loop.time() function.
|
1 |
Took 6.273 seconds |
Next, let’s confirm that benchmarking with loop.time() includes await times.
Confirm loop.time() Includes Await Time
A concern when using the loop.time() method for benchmarking is whether the recorded interval includes await times.
Generally, the loop.time() method internally uses the time.monotonic() function which uses an internal clock that is not suspended when the calling coroutine, thread, and process is blocked.
Nevertheless, this behavior is not documented and we can confirm it with a worked example.
In this case, we will define a coroutine that suspends by awaiting asyncio.sleep() for one second.
|
1 2 3 4 |
# our work coroutine async def work(): # sleep a moment await asyncio.sleep(1) |
We will then benchmark this coroutine for one second and expect that the duration is about one second.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# define coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # await the target coroutine await work() # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of benchmarking a coroutine that awaits import asyncio # our work coroutine async def work(): # sleep a moment await asyncio.sleep(1) # define coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # await the target coroutine await work() # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') # run the coroutine asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and records the start time using the event loop time.
Next, the work() coroutine is awaited. This suspends the main() coroutine and executes the work() coroutine, in this case suspending and awaiting asyncio.sleep() for one second. The work() coroutine then resumes and terminates.
The main() coroutine resumes and the end time is then recorded using the event loop time.
The difference between the two recorded times is calculated, providing the coroutine execution duration in seconds.
Finally, the result is reported, truncated to three decimal places (milliseconds).
In this case, we can see that the function took about 1.005 seconds to complete. This matches the expectation of about one second.
Note, that the results on your system may vary.
This highlights that using loop.time() for benchmarking in asyncio programs includes await times, and that the clock used by the event loop is not suspended when the program sleeps.
|
1 |
Took 1.005 seconds |
Next, let’s confirm that when the loop.time() method is used for benchmarking, and it includes time when the thread that is managing the event loop is suspended.
Confirm loop.time() Includes Thread Blocking Time
Another concern with using loop.time() is that we are not sure if the underlying clock will be suspended if the thread in which the event loop is running is suspended.
For example, if the thread is paused with a call to time.sleep(), this will block both the thread and the event loop.
|
1 2 3 4 |
# our work coroutine async def work(): # sleep a moment time.sleep(1) |
We suspect that the underlying event loop clock will not be suspended, but we can confirm this with a worked example.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of benchmarking a blocking coroutine import time import asyncio # our work coroutine async def work(): # sleep a moment time.sleep(1) # define coroutine async def main(): # record start time time_start = asyncio.get_running_loop().time() # await the target coroutine await work() # record end time time_end = asyncio.get_running_loop().time() # calculate the duration time_duration = time_end - time_start # report the duration print(f'Took {time_duration:.3f} seconds') # run the coroutine asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and records the start time using the event loop time.
Next, the work() coroutine is awaited. This suspends the main() coroutine and executes the work() coroutine, in this case blocking the event loop and the current thread with a call to time.sleep(). The work() coroutine then resumes and terminates.
The main() coroutine resumes and the end time is then recorded using the event loop time.
The difference between the two recorded times is calculated, providing the coroutine execution duration in seconds.
Finally, the result is reported, truncated to three decimal places (milliseconds).
In this case, we can see that the function took about 1.005 seconds to complete. This matches the expectation of about one second.
Note, that the results on your system may vary.
This highlights that using loop.time() for benchmarking in asyncio programs includes calls that block the event loop and the current thread.
|
1 |
Took 1.005 seconds |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python Benchmarking, Jason Brownlee (my book!)
Also, the following Python books have chapters on benchmarking that may be helpful:
- Python Cookbook, 2013. (sections 9.1, 9.10, 9.22, 13.13, and 14.13)
- High Performance Python, 2020. (chapter 2)
Guides
- 4 Ways to Benchmark Python Code
- 5 Ways to Measure Execution Time in Python
- Python Benchmark Comparison Metrics
Benchmarking APIs
- time — Time access and conversions
- timeit — Measure execution time of small code snippets
- The Python Profilers
References
Takeaways
You now know how to benchmark asyncio programs using the loop.time() method.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Andhika Soreng on Unsplash
Leave a Reply