Last Updated on October 29, 2022
You can show the progress of tasks in the ThreadPool using a callback function.
In this tutorial, you will discover how to show the progress of tasks in the ThreadPool in Python.
Let’s get started.
Need To Show Progress of Tasks in the ThreadPool
The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.
A thread pool object which controls a pool of worker threads to which jobs can be submitted.
— multiprocessing — Process-based parallelism
The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.
Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.
A ThreadPool can be configured when it is created, which will prepare the new threads.
We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().
When issuing many tasks to the ThreadPool, we may need to keep track of how much work remains to be completed.
How can we show the progress of completed tasks in the ThreadPool?
Run loops using all CPUs, download your FREE book to learn how.
How to Show Progress in the ThreadPool
We can show the progress of tasks in the ThreadPool using the callback function.
This can be achieved by issuing tasks asynchronously to the ThreadPool, such as via the apply_async() function and specifying a callback function via the “callback” argument.
For example:
|
1 2 3 |
... # issue a task to the thread pool pool.apply_async(task, callback=progress) |
Our custom callback function will then be called after each task in the ThreadPool is completed.
We can then perform some action to show the progress of completed tasks in the callback function, such as printing a dot to standard out.
For example:
|
1 2 3 |
# progress indicator for tasks in the thread pool def progress(results): print('.', end='') |
Now that we know how to show the progress of tasks in the ThreadPool in a standard way, let’s look at a worked example.
Example of Showing Progress in the ThreadPool
We can explore how to show the progress of tasks issued to the ThreadPool.
In this example, we will define a task that will block for a fraction of a second. We will then issue many of these tasks to the ThreadPool. A callback will be called as each task is finished and will print a message to show the progress of tasks as they are completed.
Firstly, we can define a callback function to call each time a task is completed in the ThreadPool.
The function takes the return value of a custom task function, if present, and prints a single dot to standard out without a newline.
The progress() function below implements this.
|
1 2 3 |
# progress indicator for tasks in the thread pool def progress(results): print('.', end='') |
Next, we can define a custom task function.
The task will first generate a random floating point value between 0 and 1. It will then sleep for that many seconds, which will be less than one second. This sleeping will simulate computational work that takes a variable amount of time.
The task() function below implements this.
|
1 2 3 4 5 6 |
# task executed in a worker thread def task(): # generate a random value value = random() # block for a moment sleep(value) |
Next, in the main thread, we can create the ThreadPool.
We will create the ThreadPool with the default configuration and use the context manager interface.
|
1 2 3 4 |
... # create and configure the thread pool with ThreadPool() as pool: # ... |
You can learn more about the context manager interface in the tutorial:
We can then issue 20 tasks to the ThreadPool, each calling our custom task() function and using the custom progress() function as a callback to show progress.
We can issue each task one by one using the apply_asyc() function which returns an AsyncResult object.
This can be achieved in a list comprehension, providing a list of AsyncResult objects, one for each task.
|
1 2 3 |
... # issue many tasks asynchronously to the thread pool results = [pool.apply_async(task, callback=progress) for _ in range(20)] |
You can learn more about the apply_async() function in the tutorial:
Finally, the main thread will close the ThreadPool and block, waiting for the issued task to complete.
|
1 2 3 4 5 |
... # close the pool pool.close() # wait for all issued tasks to complete pool.join() |
You can learn more about joining the ThreadPool in the tutorial:
After the ThreadPool is closed, a final message can be reported.
|
1 2 3 |
... # report all done print('\nDone!') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of showing progress in the thread pool with separate tasks from time import sleep from random import random from multiprocessing.pool import ThreadPool # progress indicator for tasks in the thread pool def progress(results): print('.', end='') # task executed in a worker thread def task(): # generate a random value value = random() # block for a moment sleep(value) # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool() as pool: # issue many tasks asynchronously to the thread pool results = [pool.apply_async(task, callback=progress) for _ in range(20)] # close the pool pool.close() # wait for all issued tasks to complete pool.join() # report all done print('\nDone!') |
Running the example first creates the ThreadPool.
Then, 20 tasks are issued to the pool, one at a time.
The main thread then closes the ThreadPool and blocks waiting for all issued tasks to be completed.
Each task blocks for a fraction of a second and finishes.
As each task is finished, the callback function is called, printing a dot.
The dots accumulate on standard output, showing the progress of all issued tasks.
Once all tasks are finished, the main thread continues and reports a final message.
|
1 2 |
.................... Done! |
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to show the progress of tasks in the ThreadPool in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Nick Fewings on Unsplash
Leave a Reply