Last Updated on October 29, 2022
Child worker threads are started automatically after creating an instance of the ThreadPool class.
In this tutorial, you will discover when the worker threads are created in the ThreadPool in Python.
Let’s get started.
ThreadPool Workers
The ThreadPool provides a pool of reusable workers for executing ad hoc tasks with thread-based concurrency.
An instance of the ThreadPool class can be created, specifying the number of workers to create, otherwise, a default number of workers will be created to match the number of logical CPUs in the system.
Once created, ad hoc tasks can be issued to the pool for execution via the apply() method. Multiple tasks may be executed in the book by calling the same function with different arguments via the map() method.
Tasks may be executed asynchronously in the pool via apply_async() and map_async() methods, allowing the caller to carry on with other tasks.
One concern with the ThreadPool is when exactly the worker threads used internally within the pool are created.
There are two main options for when child workers could be created:
- Immediately after an instance of the class is created.
- On-demand as tasks are issued to the pool.
We may need to know this for many reasons, such as:
- Will there be lag when the first task is issued if worker threads are created on demand?
- Will the pool need to be closed correctly if no tasks have been issued to the pool?
When are worker threads created in the ThreadPool?
Run loops using all CPUs, download your FREE book to learn how.
When Are ThreadPool Workers Created
Workers in the ThreadPool are started immediately after the pool is created.
For example:
|
1 2 3 |
... # create a pool pool = multiprocessing.pool.ThreadPool(4) |
Moments after the class is created, it will internally create enough workers to populate the pool. In this case four.
This also applies when using the pool via the context manager interface.
For example:
|
1 2 3 4 |
... # create a pool with the context manager interface with multiprocessing.pool.ThreadPool(4) as pool: # ... |
Again, moments after the Pool instance is defined, it is populated with worker threads.
Now that we know when the worker threads are created in the ThreadPool, let’s look at a worked example.
Example of When Workers Are Created in the Pool
We can explore when the worker threads in the ThreadPool are created.
In this example, we will create a ThreadPool, wait a moment, then check if the caller thread has any running threads.
If it does have workers and the number of active threads matches the number of workers configured in the pool, it means that the child workers are created after the class is created.
Otherwise, it means that worker threads are created on demand.
This is an experimental approach to the question, an alternate approach would be to review the source code of the multiprocessing.pool.Pool class (the base class for the ThreadPool class).
Firstly, we can create a ThreadPool instance with four child worker threads. Don’t worry if you have more or fewer CPU cores, as we will not be running any tasks.
|
1 2 3 |
... # create a pool pool = ThreadPool(4) |
We will then wait a moment to allow any internal initialization by the pool to be completed.
|
1 2 3 |
... # wait a moment sleep(0.5) |
Next, we can get access to all active threads for the current process and report their details.
This can be achieved via the threading.enumerate() module function.
|
1 2 3 4 |
... # report all active child threads for thread in threading.enumerate(): print(thread) |
This could be a list with one item, the main thread running the program. It could be a list with 5 or more threads, one for each worker, the main thread, and any other threads running.
Finally, we can close the pool now that we are finished with it.
|
1 2 3 |
... # close the pool pool.close() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# SuperFastPython.com # when are worker threads started from time import sleep from multiprocessing.pool import ThreadPool import threading # protect the entry point if __name__ == '__main__': # create a pool pool = ThreadPool(4) # wait a moment sleep(0.5) # report all active threads for thread in threading.enumerate(): print(thread) # close the pool pool.close() |
Running the example first creates an instance of the ThreadPool.
The caller then blocks to allow the pool to start up completely. This is probably not necessary.
Next, a list of all running threads is retrieved and traversed, reporting the details of each.
In this case, we can see that the list of running threads contains many threads, including:
- the main thread, with the type _MainThread and name MainThread
- the four worker threads in the ThreadPool with the type DummyProcess and names Thread-1 to Thread-4.
- the three helper threads in the ThreadPool with the type Thread and names Thread-5 to Thread-7.
This confirms that worker threads are created after the class is created, and not on demand.
Finally, the ThreadPool is closed.
|
1 2 3 4 5 6 7 8 |
<_MainThread(MainThread, started 4505034240)> <DummyProcess(Thread-1, started daemon 123145323581440)> <DummyProcess(Thread-2, started daemon 123145340370944)> <DummyProcess(Thread-3, started daemon 123145357160448)> <DummyProcess(Thread-4, started daemon 123145373949952)> <Thread(Thread-5, started daemon 123145390739456)> <Thread(Thread-6, started daemon 123145407528960)> <Thread(Thread-7, started daemon 123145424318464)> |
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know when worker threads are created in the ThreadPool.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Leave a Reply