You can create a hosted centralized version of Python objects using a Manager and share proxy objects that allow child processes to interact with the hosted object in a process-safe manager.

In this tutorial you will discover the multiprocessing Manager in Python.

Let’s get started.

What is a Multiprocessing Manager

Multiprocessing Manager provides a way of creating centralized Python objects that can be shared safely among processes.

Managers provide a way to create data which can be shared between different processes, including sharing over a network between processes running on different machines.

— multiprocessing – Process-based parallelism

Manager objects create a server process which is used to host Python objects. Managers then return proxy objects used to interact with the hosted objects.

The proxy objects automatically ensure process-safety and serialize data to and from the centralized objects.

A proxy is an object which refers to a shared object which lives (presumably) in a different process. The shared object is said to be the referent of the proxy. Multiple proxy objects may have the same referent. A proxy object has methods which invoke corresponding methods of its referent […]. In this way, a proxy can be used just like its referent can

— multiprocessing – Process-based parallelism

This means that the proxy objects can be shared among multiple processes allowing the centralized object to be used in parallel seamlessly in a process-safe manner.

A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies.

— multiprocessing – Process-based parallelism

What are the Benefits of a Manager

Managers provide three key capabilities for process-based concurrency, they are:

• Centralized: A single instance of a shared object is maintained in a separate server process.
• Process-safety: Proxy objects ensure that access to the centralized object is process-safe in order to avoid race conditions.
• Pickability: Proxy objects can be pickled and shared with child processes such as arguments in process pools and items in queues.

In addition to providing safe access to a shared object across processes on one system, managers allow the same object to be accessed safely across processes on other systems via network access.

How to Use A Manager?

Using a manager involves four steps, they are:

1. Create the manager instance.
2. Start the manager.
3. Create one or more hosted objects to share.
4. Shutdown the manager.

1. Create a Manager

A manager can be created by creating an instance of the multiprocessing.Manager class.

For example:

2. Start the Manager

After the manager is created, it must be started.

This can be achieved by calling the start() method.

For example:

This will start the server process used to host centralized versions of Python objects.

3. Create Hosted Objects

Once the manager is started, it can be used to create hosted objects and return proxy objects for those hosted objects.

Only objects that have been registered with the Manager, e.g. those objects that the manager knows about, can be created.

For example:

This creates a centralized version of an object in the Manager‘s server process, in this case a mutex Lock class, and returns a proxy for interacting with the hosted object.

4. Shutdown the Manager

Once we are finished with the shared objects, we can shutdown the manager.

This will release all resources used by the manager, including the server process.

This can be achieved by calling the close() function.

For example:

Context Manager Interface

An alternate usage pattern for the Manager is to use the context manager interface.

This ensures that the Manager instance is always closed, in case we forget to call close() or in case the close() method is not reached due to an error.

For example:

The manager is created and started at the beginning of the context manager block, all usage of the manager is restricted to the block, then the manager is closed as soon as the block is exited, normally or otherwise.

This is the preferred way to use the Manager in your program, if possible.

What Objects Does a Manager Provide

Managers are provided via the multiprocessing.Manager class, which creates and returns a multiprocessing.managers.SyncManager instance.

The SyncManager allows a suite of Python objects to be created and managed by default, including:

1. Data structures
2. Shared ctypes
3. Concurrency primitives
4. Queues
5. Other Objects

Let’s take a closer look at each group:

Managed Data Structures

This includes Python objects we may want to share, such as:

• dict
• list

You can see an example of sharing a managed collection among processes in the tutorial:

• Multiprocessing Manager Example in Python

Managed Shared Ctypes

It includes shared ctypes for primitive values, such as:

• Value
• Array

If you are new to shared ctypes, I recommend the tutorial:

• Multiprocessing Shared ctypes in Python

Managed Concurrency Primitives

It includes concurrency primitives for synchronizing and coordinating processes, such as:

• Lock
• Event
• Condition
• Semaphore
• BoundedSemaphore
• Barrier

You can see an example of sharing a managed concurrency primitive in the tutorial:

• Multiprocessing Manager Share Concurrency Primitives in Python

Managed Queues

It includes process-safe queues for sharing data between processes, such as:

• Queue
• JoinableQueue

You can see an example of sharing a managed queue among processes in the tutorial:

• Multiprocessing Manager Share Queue in Python

Managed Other Objects

Other objects are provided by the SyncManager class, some of which are not documented, such as:

• Namespace
• Pool (along with Iterator and AsyncResult)

You can see an example of sharing a managed Namespace in the tutorial:

• How to Use a Manager Namespace to Share Data with Processes

You can see an example of using a managed Pool in the tutorial:

• Share a Multiprocessing Pool With Workers

You can see an example of sharing a managed custom class in the tutorial:

• Multiprocessing Manager With a Custom Class

More Details

In fact, if we look at the source code for the multiprocessing.managers.SyncManager class, we can see that the actual classes that are shared are from the “threading” module, not the “multiprocessing” module.

Likely this is because the hosted objects are never used outside of the Manager‘s server process. They are only interacted with indirectly via proxy objects which manage the inter-process communication and serialization for process-safety.

Care must be taken when nesting objects hosted by a manager.

You can see an example of using nested objects with a manager in the tutorial:

• Multiprocessing Manager Nested Proxy Objects

When to Use a Manager

A manager should be used when an object needs to be shared among processes.

Sometimes the object can be shared directly without problem, other times it cannot and a manager is required for those cases, which include:

1. When the shared object cannot be pickled and needs to be pickled in order to be shared.
2. When the shared object is not process-safe and needs to be used in multiple processes simultaneously.

If the object to be shared meets either or both of these requirements, then it is a good candidate for being hosted in a manger server process and shared via proxy objects.

What Are Some Examples Where a Manager is Required

There are a number of common use cases where you may need to use a manager.

They include:

1. When sharing a concurrency primitive (e.g. lock, semaphore, event, etc.) or queue with a multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor. This is because concurrency primitives cannot be pickled and all arguments to process pools must be pickled.
2. When sharing an object that cannot be pickled with processes via a multiprocessing.Queue. This is because the multiprocessing.Queue requires that all objects put on the queue be pickled.

Further Reading

This section provides additional resources that you may find helpful.

Python Multiprocessing Books

• Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
• Multiprocessing API Interview Questions
• Multiprocessing API Cheat Sheet

I would also recommend specific chapters in the books:

• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
  • See: Chapter 9: The multiprocessing Module
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Multiprocessing: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide

APIs

• multiprocessing — Process-based parallelism
• PEP 371 - Addition of the multiprocessing package

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know about the multiprocessing Manager in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Krzysztof Kowalik on Unsplash