You can get the details of the thread used to initialize worker threads and to execute tasks in the ThreadPool.
Additionally, the ThreadPool uses helper threads internally to help with the management of the ThreadPool. We can get the details of these threads in the main thread that creates the ThreadPool.
In this tutorial, you will discover how to get the details of threads used by the ThreadPool in Python.
Let’s get started.
Need Details of Threads in the ThreadPool
The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.
A thread pool object which controls a pool of worker threads to which jobs can be submitted.
— multiprocessing — Process-based parallelism
The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.
Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.
A ThreadPool can be configured when it is created, which will prepare the new threads.
We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().
When using the ThreadPool, we may need the names of the thread used to initialize and execute tasks in worker threads.
This may be for many reasons, although it is likely to aid in designing the thread-safe handling of data or program state in worker threads or to help with debugging.
How can we get the worker thread’s names in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Get Threads Used in ThreadPool
We can get the current thread via the threading.current_thread() function.
This will return a threading.Thread instance that can be used to get the details of the current thread in situations such as:
- A worker threads in the ThreadPool executing a task.
- Initializing a worker thread in the ThreadPool.
For example, we may have questions such as:
- Are worker threads initialized using the main thread or in a worker thread?
You can learn more about utility functions for getting threads in the tutorial:
Additionally, the ThreadPool itself uses worker threads. We can get the details of these threads, to help differentiate them from any other threads we may have in our program.
Let’s take a closer look at each one of these aspects in turn.
Thread Used to Execute Task
We can get the details of the current thread executing a task in the ThreadPool.
This can be achieved by calling the threading.current_thread() function to get the threading.Thread instance for the thread executing the task in the worker thread.
|
1 2 3 |
... # get the current thread thread = current_thread() |
For example, we can call this function in a custom function executed by the ThreadPool:
|
1 2 3 4 5 |
# task executed in a worker thread def task(identifier): # get the current thread thread = current_thread() print(thread) |
Thread Used to Initialize Worker
We can get the details of the current thread initializing a task in the ThreadPool.
This can be achieved by calling the threading.current_thread() function to get the threading.Thread instance for the thread executing the function used to initialize in the worker thread.
|
1 2 3 |
... # get the current thread thread = current_thread() |
For example, we can call this function in a custom function executed used to initialize workers in the ThreadPool:
|
1 2 3 4 5 |
# initialize the worker thread def init_worker(): # get the current thread thread = current_thread() print(thread) |
We can then configure the ThreadPool to call the custom function when initializing worker threads.
|
1 2 3 |
... # configure the thread pool to initialize worker threads pool = ThreadPool(initializer=init_worker) |
You can learn more about how to initialize worker threads in the ThreadPool in the tutorial:
Threads Used to Manage ThreadPool
The multiprocessing.pool.Pool class (that the ThreadPool extends) uses worker threads internally to help manage the pool.
Looking at the source code for the Pool class, we can see that at least three threads are created with responsibilities such as:
- Handling the creation of workers.
- Handling the dispatching of tasks to workers.
- Handing the task results from workers.
In the main thread that creates the ThreadPool, we can get a list of all running threads and inspect the details of the threads associated with the ThreadPool.
We can call the threading.enumerate() function to get a list of all actively running threads. This will return a list of threading.Thread instances, one for each running thread.
For example:
|
1 2 3 |
... # get a list of all running threads threads = threading.enumerate() |
We can get this list of threads after the ThreadPool has been created, and ideally after tasks have already been issued so that all workers and maintenance threads have been started.
Now that we know how to get the details of threads associated with the ThreadPool, let’s look at some worked examples.
Example of Getting Thread Used to Execute Task
We can explore how to get the details of the thread used to execute tasks in the ThreadPool.
In this example, we will define a custom function to be executed in the ThreadPool. The task will get the current thread and report its details. The main thread will create the ThreadPool and issue the task into the ThreadPool.
Firstly, we can define a custom function to execute as a task in the ThreadPool.
The function will take an argument and then get the threading.Thread instance for the current thread and report the thread name. We could report other details of the thread as well.
The task() function below implements this.
|
1 2 3 4 5 6 |
# task executed in a worker thread def task(identifier): # get the current thread thread = current_thread() # report the name of the current thread print(f'Worker thread: {thread.name}') |
You can learn more about the details of a thread in the tutorial:
Next, in the main thread, we will create the ThreadPool with the default configuration.
We will use the context manager interface to ensure that the ThreadPool is closed automatically once we are finished with it.
|
1 2 3 4 |
... # create and configure the thread pool with ThreadPool(initializer=init_worker) as pool: # ... |
You can learn more about the context manager interface in the tutorial:
We will issue a single call to the custom function as a task in the ThreadPool.
This can be achieved via the apply_async() function that will issue the task asynchronously and return an AsyncResult as a handle on the task.
|
1 2 3 |
... # issues task to the thread pool result = pool.apply_async(task, (0,)) |
You can learn more about issuing tasks via the apply_async() function in the tutorial:
The main thread will then wait for the task to finish via the returned AsyncResult object.
|
1 2 3 |
... # wait for tasks to complete result.wait() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of getting the details of the thread used to execute the task from multiprocessing.pool import ThreadPool from threading import current_thread # task executed in a worker thread def task(identifier): # get the current thread thread = current_thread() # report the name of the current thread print(f'Worker thread: {thread.name}') # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool() as pool: # issues task to the thread pool result = pool.apply_async(task, (0,)) # wait for tasks to complete result.wait() # thread pool is closed automatically |
Running the example first creates the ThreadPool.
The main thread then issues a task to the ThreadPool and then blocks until the task completes.
The custom function runs in the ThreadPool. It gets the current thread and reports its name.
In this case, we can see, as we might expect, that a worker thread is used in the pool to execute tasks, in this case with the name “Thread-1“, a default name assigned to threads
|
1 |
Worker thread: Thread-1 |
Next, let’s explore how we might check which thread is used to initialize the worker thread.
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
Example of Getting Thread Used to Initialize Workers
We can explore how to get the details of the thread used to initialize the worker threads.
In this example, we will define a custom function to initialize the worker threads. The initialization function will get the current thread and report its details. To ensure that the worker threads are initialized, we will issue one task into the ThreadPool that will just block for a fraction of a second.
Firstly, we can define the custom function that will initialize the worker threads.
The function does not take any arguments or return any values. It will get the threading.Thread instance for the current thread, then report its name.
The init_worker() function below implements this.
|
1 2 3 4 5 6 |
# initialize the worker thread def init_worker(): # get the current thread thread = current_thread() # report the name of the current thread print(f'Initialize worker thread: {thread.name}') |
Next, we can define a custom function to execute a task in the ThreadPool.
The task will take an argument and block for a fraction of a second.
The task() function below implements this.
|
1 2 3 4 |
# task executed in a worker thread def task(identifier): # block for a moment sleep(0.5) |
Finally, in the main thread, we can create the ThreadPool with the default configuration using the context manager interface.
We can issue the task using the apply_async() function, get an AsyncResult in return immediately, then wait on the object for the task to complete.
|
1 2 3 4 5 6 7 |
... # create and configure the thread pool with ThreadPool(initializer=init_worker) as pool: # issues task to the thread pool result = pool.apply_async(task, (0,)) # wait for tasks to complete result.wait() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of getting the details of the thread used to initialize worker thread from time import sleep from multiprocessing.pool import ThreadPool from threading import current_thread # initialize the worker thread def init_worker(): # get the current thread thread = current_thread() # report the name of the current thread print(f'Initialize worker thread: {thread.name}') # task executed in a worker thread def task(identifier): # block for a moment sleep(0.5) # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool(initializer=init_worker) as pool: # issues task to the thread pool result = pool.apply_async(task, (0,)) # wait for tasks to complete result.wait() # thread pool is closed automatically |
Running the example first creates the ThreadPool.
The main thread then issues the task into the ThreadPool and waits for it to complete.
The ThreadPool is created and starts a default number of worker threads, eight in this case, although the number of threads will differ depending on the number of CPU cores in your system.
Each worker thread is then initialized using the custom initialization function that gets the current thread and reports its name.
In this case, we can see that each worker thread is initialized using the main thread of the worker thread.
This is not surprising.
|
1 2 3 4 5 6 7 8 |
Initialize worker thread: Thread-1 Initialize worker thread: Thread-2 Initialize worker thread: Thread-3 Initialize worker thread: Thread-4 Initialize worker thread: Thread-5 Initialize worker thread: Thread-6 Initialize worker thread: Thread-7 Initialize worker thread: Thread-8 |
Next, let’s investigate the worker threads created to help manage the ThreadPool.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Getting Threads Used To Manage the ThreadPool
We can explore the threads created automatically by the ThreadPool.
Looking at the source code for the multiprocessing.pool.Pool class (that is the parent of the ThreadPool class), we can see that helper threads are created by the process pool.
A brief review of the code suggests that the roles of these helper functions might be to:
- Help to create workers.
- Help with dispatching work to workers.
- Help handling results sent back from workers.
In this example, we will create a ThreadPool and issue tasks and wait for them to complete. We will then get a list of all active threads in the main thread and report their details.
Firstly, we can define a custom function to execute in the ThreadPool.
The task does not need to do anything other than help start up the ThreadPool completely.
In this case, the task will take an argument and block for a fraction of a second.
The task() function below implements this.
|
1 2 3 4 |
# task executed in a worker thread def task(identifier): # block for a moment sleep(0.5) |
Next in the main thread, we will create the ThreadPool using the default configuration. We will use the context manager interface to ensure the pool is closed automatically once we are finished with it.
|
1 2 3 4 |
... # create and configure the thread pool with ThreadPool() as pool: # ... |
Next, we will issue 10 tasks to the ThreadPool, calling our custom task() function with different integer values from 0 to 9.
This can be achieved with the map_async() function that will issue the tasks asynchronously and return immediately with an AsyncResult object. We can then wait on the object until the issued tasks are completed.
|
1 2 3 4 5 |
... # issues tasks to thread pool result = pool.map_async(task, range(10)) # wait for tasks to complete result.wait() |
You can learn more about the map_async() function in the tutorial:
Next, we can get a list of all active threads in the main thread and report the details of each.
|
1 2 3 4 |
... # report all threads in this process for thread in threading.enumerate(): print(f'Thread: {thread}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of getting the details of threads used to maintain the thread pool from time import sleep from multiprocessing.pool import ThreadPool from threading import current_thread import threading # task executed in a worker thread def task(identifier): # block for a moment sleep(0.5) # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool() as pool: # issues tasks to thread pool result = pool.map_async(task, range(10)) # wait for tasks to complete result.wait() # report all threads in this process for thread in threading.enumerate(): print(f'Thread: {thread}') # thread pool is closed automatically |
Running the example first creates the ThreadPool
The 10 tasks are then issued to the ThreadPool and the main thread blocks until they are complete.
Each task runs in the ThreadPool, blocks for a moment, then returns.
The main thread continues on and gets a list of all active threads and reports their details.
We can see the main thread, used to execute the program with the type _MainThread and the name of MainThread.
In this case, we can see eight worker threads with the type DummyProcess and names Thread-1 to Thread-8.
We can then see the three helper threads that have the type Thread and names Thread-9 to Thread-11.
All worker threads and helper threads are daemon threads, meaning they will not prevent the main program from exiting.
Unfortunately, the helper threads and workers have ambiguous default, e.g. “Thread-1” to “Thread-11“. Hopefully, they are renamed in a future version of Python to indicate their association with the ThreadPool.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
Thread: <_MainThread(MainThread, started 4607856128)> Thread: <DummyProcess(Thread-1, started daemon 123145506635776)> Thread: <DummyProcess(Thread-2, started daemon 123145523425280)> Thread: <DummyProcess(Thread-3, started daemon 123145540214784)> Thread: <DummyProcess(Thread-4, started daemon 123145557004288)> Thread: <DummyProcess(Thread-5, started daemon 123145573793792)> Thread: <DummyProcess(Thread-6, started daemon 123145590583296)> Thread: <DummyProcess(Thread-7, started daemon 123145607372800)> Thread: <DummyProcess(Thread-8, started daemon 123145624162304)> Thread: <Thread(Thread-9, started daemon 123145640951808)> Thread: <Thread(Thread-10, started daemon 123145657741312)> Thread: <Thread(Thread-11, started daemon 123145674530816)> |
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Takeaways
You now know how to get the details of threads used by the ThreadPool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by David M. Chambers on Unsplash
Leave a Reply