Last Updated on September 12, 2022
You can configure and start new threading.Thread instances within new child multiprocessing.Process instances.
In this tutorial you will discover how to start threads in child processes in Python.
Let’s get started.
Need To Start Threads in Child Processes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In multiprocessing, we may need to start new threads in child processes.
For example, this is a common requirement when we need to perform many IO-bound tasks within each child process.
How can we start threads within processes in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Start Threads in a Child Process
We can start new threads in child processes directly.
A new thread can be created via the threading.Thread class.
We can specify the function to execute in a new thread via the “target” argument.
For example:
|
1 2 3 |
... # create a new thread thread = threading.Thread(target=thread_task) |
Where, thread_task refers to a custom function we might define to run in a new thread.
|
1 2 3 |
# function to run in a new thread def thread_task(): # ... |
Once configured we can start the thread by calling the start() method.
For example:
|
1 2 3 |
... # start the thread thread.start() |
You can learn more about running functions in new threads in the tutorial:
We can then start new child processes to start new threads.
This can be achieved by defining a custom function that starts as many threads that are required to execute a given second custom function.
For example:
|
1 2 3 4 5 6 7 |
# function to run in a child process def child_task(): # create a new thread thread = threading.Thread(target=thread_task) # start the thread thread.start() # ... |
A new multiprocessing.Process instance can then be created and configured to execute the custom function to start threads and then start itself.
|
1 2 3 |
... # configure a new child process child = multiprocessing.Process(target=child_task) |
If you are new to running functions in a new child process, see the tutorial:
Now that we know how to start threads in child processes, let’s look at a worked example.
Example of Starting Threads in Processes
We can explore how to start threads in child processes.
In this example we will start five child processes. Each child process will execute a task that involves starting three threads. Each thread will block for a random fraction of three seconds, then report a message. This means that there will be a total of five new child processes and 15 new threads running in the application, plus the main process and the six main threads running in each process.
First, we can define a function to execute in each thread.
The function will block for a random fraction of three seconds, then report that it is done including both the process name and the thread name.
The multiprocessing.current_process() function can be used to get the current process, and the “name” attribute can be used to access the name of the current process.
Similarly, the threading.current_thread() function can be used to get the current thread and the “name” attribute accessed to get the name of the current thread.
The thread_task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task executed by new threads def thread_task(): # block for a moment sleep(3 * random()) # report a message process_name = current_process().name thread_name = current_thread().name print(f'>thread {thread_name} in process {process_name} done.', flush=True) |
Next, we can define a function to be executed by each child process.
The function will first create a list of three threads. This can be achieved in a list comprehension.
The threads are then started and the child process blocks until all threads have finished.
Finally, the child process gets the name of the current process and reports that it is done.
The process_task() function below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# task executed by new processes def process_task(): # configure new threads threads = [Thread(target=thread_task) for _ in range(3)] # start new threads for thread in threads: thread.start() # wait for threads to finish for thread in threads: thread.join() # report message process_name = current_process().name print(f'Process {process_name} done.', flush=True) |
Finally, in the main process, we can create a list of five child processes. This can be achieved using a list comprehension.
The child processes are then started and the main process blocks until all child processes are finished.
The main process then reports that it is complete.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# protect the entry point if __name__ == '__main__': # configure child processes processes = [Process(target=process_task) for _ in range(5)] # start new processes for child in processes: child.start() # wait for processes to finish for child in processes: child.join() # report message print('Main process done.', flush=True) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
# SuperFastPython.com # example of starting threads in child processes from random import random from time import sleep from threading import Thread from threading import current_thread from multiprocessing import Process from multiprocessing import current_process # task executed by new threads def thread_task(): # block for a moment sleep(3 * random()) # report a message process_name = current_process().name thread_name = current_thread().name print(f'>thread {thread_name} in process {process_name} done.', flush=True) # task executed by new processes def process_task(): # configure new threads threads = [Thread(target=thread_task) for _ in range(3)] # start new threads for thread in threads: thread.start() # wait for threads to finish for thread in threads: thread.join() # report message process_name = current_process().name print(f'Process {process_name} done.', flush=True) # protect the entry point if __name__ == '__main__': # configure child processes processes = [Process(target=process_task) for _ in range(5)] # start new processes for child in processes: child.start() # wait for processes to finish for child in processes: child.join() # report message print('Main process done.', flush=True) |
Running the example first configures then starts five child processes.
The main process then blocks until all child processes finish.
Each child process then configures and starts three worker threads. The child processes then block until their own worker thread finishes.
Each worker thread blocks for a random fraction of three seconds, then reports a message.
The threads for a child process finish and the child process continues on and reports a message.
The child processes all finish, and the main process continues on and reports its final message.
This highlights how to start and manage new threads with new processes in Python.
Note, results will differ each time the code is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
>thread Thread-2 in process Process-1 done. >thread Thread-1 in process Process-4 done. >thread Thread-3 in process Process-4 done. >thread Thread-3 in process Process-3 done. >thread Thread-3 in process Process-5 done. >thread Thread-2 in process Process-2 done. >thread Thread-2 in process Process-5 done. >thread Thread-2 in process Process-3 done. >thread Thread-1 in process Process-2 done. >thread Thread-1 in process Process-5 done. Process Process-5 done. >thread Thread-3 in process Process-2 done. Process Process-2 done. >thread Thread-1 in process Process-1 done. >thread Thread-1 in process Process-3 done. Process Process-3 done. >thread Thread-2 in process Process-4 done. Process Process-4 done. >thread Thread-3 in process Process-1 done. Process Process-1 done. Main process done. |
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to start threads in child processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Andre Ouellet on Unsplash
Leave a Reply