You can get a multiprocessing context via the multiprocessing.get_context() function in order to use multiple different process start methods within one program.
In this tutorial you will discover how to get and use multiprocessing contexts in Python.
Let’s get started.
Need Multiple Start Methods
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In multiprocessing, we may need multiple start methods within the one program.
This may be for many reasons, although the most common reason is the use of third-party libraries within the program that require the use of a specific start method that may differ from the rest of the program.
For example, we may use a library that requires the use of the ‘spawn’ start method so that it is consistent across all platforms, whereas in our program we may require the use of the ‘fork’ start method.
How can we use multiple start methods within the one application?
Run loops using all CPUs, download your FREE book to learn how.
What is a Start Method
A start method is the technique used to start child processes in Python.
There are three start methods, they are:
- spawn: start a new Python process.
- fork: copy a Python process from an existing process.
- forkserver: new process from which future forked processes will be copied.
Each platform has a default start method.
The following lists the major platforms and the default start methods.
- Windows (win32): spawn
- macOS (darwin): spawn
- Linux (unix): fork
We can use the current start method using the multiprocessing.get_start_method() module function.
We can change the start method by passing the preferred method as an argument to the multiprocessing.set_start_method() module function.
Finally, we can get a list of supported start methods via the multiprocessing.get_all_start_methods() module function.
You can learn more about start methods in the tutorial:
Next, let’s take a look at an alternate way of managing start methods called a multiprocessing context.
What is a Multiprocessing Context
A multiprocessing context provides an alternate way of managing start methods within a Python program.
It is an object that is configured to use a specific start method and provides the entire multiprocessing module API for that start method.
Context objects have the same API as the multiprocessing module, and allow one to use multiple start methods in the same program.
— multiprocessing — Process-based parallelism
We can configure a multiprocessing context for a given start method, then use that context for our multiprocessing needs within a part of our program. We can then configure a different multiprocessing context with an alternate start method for use in a different part of our application.
In fact, we might think of the multiprocessing module itself as a default multiprocessing context, where the module functions multiprocessing.get_start_method() and multiprocessing.set_start_method() are used to change the start method.
Multiprocessing contexts provide a more flexible way to manage process start methods directly within a program, and may be a preferred approach to changing start methods in general, especially within a Python library.
A library which wants to use a particular start method should probably use get_context() to avoid interfering with the choice of the library user.
— multiprocessing — Process-based parallelism
A limitation of using a multiprocessing context is that objects created in one context may not be comfortable with another context.
For example, we may create synchronization primitives in one context, such as a multiprocessing.Lock or a multiprocessing.Semaphore, then attempt to share them with processes created with different contexts. This may result in an error.
Note that objects related to one context may not be compatible with processes for a different context. In particular, locks created using the fork context cannot be passed to processes started using the spawn or forkserver start methods.
— multiprocessing — Process-based parallelism
Now that we know what multiprocessing contexts are, let’s look at how we might use them.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
How to Use a Multiprocessing Context
We can configure and retrieve a multiprocessing context using the multiprocessing.get_context() function.
The function takes an argument named “method” that specifies the start method, which defaults to None.
The function returns a multiprocessing context object that provides the full multiprocessing module API for working with processes for concurrency.
If the “method” argument is not specified, the default value of None is used and a context is returned that uses the currently configured start method in the multiprocessing module.
For example:
|
1 2 3 |
... # get a multiprocessing context with the default start method context = multiprocessing.get_context() |
A start method can be provided as an argument but must be a start method supported by the current platform, e.g one of the methods returned from the multiprocessing.get_all_start_methods() function.
For example, we might get a multiprocessing context that uses the ‘spawn‘ start method, supported on all platforms.
For example:
|
1 2 3 |
... # get a multiprocessing context with the spawn start method context = multiprocessing.get_context('spawn') |
Once retrieved, we can use the context directly to create and configure new child processes.
For example:
|
1 2 3 |
... # create and configure a child process using the context child = context.Process(target=task) |
We may also use the multiprocessing context to create synchronization primitives, such as barriers, semaphores, locks, conditions and more.
For example:
|
1 2 3 |
... # create a mutex lock using the context lock = context.Lock() |
The multiprocessing context can be used for all needs normally handled by the multiprocessing module.
Objects created with one multiprocessing context should not be used with the default context or with other contexts as it may result in an error.
For example, a lock created with one multiprocessing context should not be shared and used with child processes created with another child process.
Now that we know how to get a multiprocessing context, let’s look at some worked examples.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Use Context To Change Start Methods
We can use a multiprocessing context to change the start method used within a program.
In this example we will get a multiprocessing context that uses the fork start method then start a child process. We will then get another different multiprocessing context that uses the spawn start method and start a second child process.
This example will highlight how we can easily use multiple different start methods within the same program via multiprocessing contexts.
Note, this example assumes that your system supports fork and spawn start methods, e.g. it may not run on Windows.
Firstly, we can define a function that runs in a child process.
The function will first get the start method that is being used in the current context and then report it. The process will then sleep for a second.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# function run in a child process def task(): # get the start method being used method = get_start_method() # report the start method print(f'Child process using start method: {method}', flush=True) # block for a moment sleep(1) |
Next, in the main process, we will get a multiprocessing context configured to use the fork start method.
|
1 2 3 |
... # get a context that uses the 'fork' start method context1 = get_context('fork') |
We will then use this start method to create a new child process and configure it to run our task() function.
|
1 2 3 |
... # configure a child process child1 = context1.Process(target=task) |
We can then start the new child process and wait for it to terminate.
|
1 2 3 4 5 |
... # start the process child1.start() # wait for the child to finish child1.join() |
Next, we can get a second multiprocessing context configured to use a different start method, the spawn method in this case.
|
1 2 3 |
... # get a context that uses the 'spawn' start method context2 = get_context('spawn') |
We can then use this second context to create a child process and configure it to run our same task() function. We can then start the process and wait for it to terminate.
|
1 2 3 4 5 6 7 |
... # configure a child process child2 = context2.Process(target=task) # start the process child2.start() # wait for the child to finish child2.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# SuperFastPython.com # example of changing the start method in a program from time import sleep from multiprocessing import get_start_method from multiprocessing import get_context from multiprocessing import Process # function run in a child process def task(): # get the start method being used method = get_start_method() # report the start method print(f'Child process using start method: {method}', flush=True) # block for a moment sleep(1) # protect the entry point if __name__ == '__main__': # get a context that uses the 'fork' start method context1 = get_context('fork') # configure a child process child1 = context1.Process(target=task) # start the process child1.start() # wait for the child to finish child1.join() # get a context that uses the 'spawn' start method context2 = get_context('spawn') # configure a child process child2 = context2.Process(target=task) # start the process child2.start() # wait for the child to finish child2.join() |
Running the example first gets a multiprocessing context configured to use the ‘fork‘ start method.
A new child process is then created using the context, started and run. The main process then blocks until the child process terminates.
The child process runs and reports its start method, which is fork, as expected given that the process was created using the new context. The child process then blocks.
The child process terminates and the main process continues on.
Next, we get a second multiprocessing context, this time configured to use the ‘spawn‘ start method.
A second child process is created using the second context and is started. The main process waits for the child process to terminate.
The second child process runs, gets the start method and reports it as ‘spawn‘. This too is as expected given that we created this second process using a second context using the spawn start method. The child process blocks for a moment then terminates.
This demonstrates how to use two different multiprocessing contexts in one application, each configured to use different start methods.
|
1 2 |
Child process using start method: fork Child process using start method: spawn |
Next, let’s look at what happens if we try to share objects created with one context with objects created with a second context.
Error When Sharing Objects From Different Contexts
We can demonstrate that errors will occur when sharing objects created with different multiprocessing contexts.
In this example we will first get a context configured to use the fork start method. We will use this context to create a mutex lock, designed to protect a critical section from race conditions. Next, we will get a second context configured to use the spawn start method. We will then create a child process using this second context and run a custom function, passing in the lock and using the lock to protect a critical section. This will result in an error.
If you are new to using mutex locks to protect critical sections, see the tutorial:
Note, this example assumes that your platform supports multiple process start methods, it may not run on Windows.
Firstly, we can define the function to run in the child process.
The function will take a mutex lock as an argument. It will then acquire the lock using the context manager interface. Within the critical section, the function will report the start method being used and block for a moment.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 9 10 |
# function run in a child process def task(lock): # acquire the lock with lock: # get the start method being used method = get_start_method() # report the start method print(f'Child process using start method: {method}', flush=True) # block for a moment sleep(1) |
Next, in the main process we will first get a multiprocessing context configured to use the ‘fork‘ start method.
|
1 2 3 |
... # get a context that uses the 'fork' start method context1 = get_context('fork') |
Next, we will use the context to create a mutex lock.
|
1 2 3 |
... # create a mutex lock lock = context1.Lock() |
Next, we will get a second multiprocessing context, this time configured to use a spawn start method.
|
1 2 3 |
... # get a context that uses the 'spawn' start method context2 = get_context('spawn') |
We will then use this context to create a child process configured to execute our task() function and pass the lock variable as an argument. The child process is then started and the main process blocks until the child process terminates.
|
1 2 3 4 5 6 7 |
... # configure a child process and pass it the lock child2 = context2.Process(target=task, args=(lock,)) # start the process child2.start() # wait for the child to finish child2.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# SuperFastPython.com # example of sharing process objects created with different contexts from time import sleep from multiprocessing import get_start_method from multiprocessing import get_context from multiprocessing import Process # function run in a child process def task(lock): # acquire the lock with lock: # get the start method being used method = get_start_method() # report the start method print(f'Child process using start method: {method}', flush=True) # block for a moment sleep(1) # protect the entry point if __name__ == '__main__': # get a context that uses the 'fork' start method context1 = get_context('fork') # create a mutex lock lock = context1.Lock() # get a context that uses the 'spawn' start method context2 = get_context('spawn') # configure a child process and pass it the lock child2 = context2.Process(target=task, args=(lock,)) # start the process child2.start() # wait for the child to finish child2.join() |
Running the example first gets a multiprocessing context using the fork start method.
The context is then used to create a mutex lock.
A second multiprocessing context is then created and configured to use the spawn start method. The second context is then used to create a child process and the child process is started. The main process then blocks until the child process finishes.
The child process runs and attempts to acquire the lock passed in as an argument.
An error is raised as the lock that was passed in was created using a fork start method and was attempted to be acquired in a process created using the spawn start method. This combination is not comfortable.
|
1 2 3 4 |
Process SpawnProcess-1: Traceback (most recent call last): ... OSError: [Errno 9] Bad file descriptor |
You may not always get an error when sharing objects created with different multiprocessing contexts.
For example, we can change the example so that the lock is created using a context configured to use the ‘spawn‘ start method and shared with a process created with a context using the ‘fork‘ start method.
The complete example with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# SuperFastPython.com # example of sharing process objects created with different contexts from time import sleep from multiprocessing import get_start_method from multiprocessing import get_context from multiprocessing import Process # function run in a child process def task(lock): # acquire the lock with lock: # get the start method being used method = get_start_method() # report the start method print(f'Child process using start method: {method}', flush=True) # block for a moment sleep(1) # protect the entry point if __name__ == '__main__': # get a context that uses the 'spawn' start method context1 = get_context('spawn') # create a mutex lock lock = context1.Lock() # get a context that uses the 'fork' start method context2 = get_context('fork') # configure a child process and pass it the lock child2 = context2.Process(target=task, args=(lock,)) # start the process child2.start() # wait for the child to finish child2.join() |
Running the example creates the lock then creates the child process and shares the lock with it.
The child process acquires the lock and reports its message as per normal.
No error is raised as in the previous example, even though we are sharing objects across different multiprocessing contexts.
It just so happens that this specific combination is compatible.
|
1 |
Child process using start method: fork |
Nevertheless, the recommendation and best practice is to never mix multiprocessing objects created using different contexts.
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to get and use multiprocessing contexts in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Leave a Reply