What is the multiprocessing.dummy Module

The multiprocessing.dummy module module provides a wrapper for the multiprocessing module, except implemented using thread-based concurrency.

It provides a drop-in replacement for multiprocessing, allowing a program that uses the multiprocessing API to switch to threads with a single change to import statements.

In this tutorial you will discover the multiprocessing.dummy module in Python.

Let’s get started.

What is multiprocessing.dummy?

The multiprocessing.dummy module provides the multiprocessing module API except implemented using threads.

The API documentation reports that dummy is a wrapper around the threading module that implements the multiprocessing module API.

multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.

— multiprocessing — Process-based parallelism

Next, let’s consider why the module exists.

Why does multiprocessing.dummy exist?

The multiprocessing.dummy module is not explicitly mentioned in the PEP 371 that describes the introduction of the multiprocessing module.

Nevertheless, the multiprocessing.dummy does appear in Python 2.6 which saw the introduction of the multiprocessing module and the dummy module is described in the API documentation in this version.

For example:

multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module

— multiprocessing — Process-based parallelism

It is stated in the PEP that the multiprocessing module seeks to provide the same API as the threading module, except implemented using processes instead of threads.

The processing package mimics the standard library threading module functionality to provide a process-based approach to threaded programming allowing end-users to dispatch multiple tasks that effectively side-step the global interpreter lock.

— PEP 371 – Addition of the multiprocessing package to the standard library

The multiprocessing module introduces additional capabilities that do not exist in the threading API, such as:

• Manager
• Shared ctypes (Value and Array)
• ThreadPool

The best guess is that the multiprocessing.dummy module is provided to be a drop-in replacement for the multiprocessing.

This would allow a program to be implemented using the multiprocessing module API with process-based concurrency and switch to thread-based concurrency with a single line change, e.g. switching the import of multiprocessing to multiprocessing.dummy.

Next, let’s explore what classes are provided in the dummy module.

What Classes does multiprocessing.dummy Provide?

The multiprocessing.dummy module provides many classes and module functions that match the multiprocessing module.

For example, we can report the contents of the module directly via the __all__ list.

Running the example report the contents of the module, as follows:

We can see 3 module methods and the rest are multiprocessing class names and mostly concurrency primitives.

Reviewing the source code for the dummy module, we can see that most of the API involves importing classes from the threading module, such as queues and concurrency primitives such as locks and semaphores.

There a few classes defined in the module, they are:

• DummyProcess, which is aliased as Process and used in the ThreadPool
• Namespace, used in managers
• Array and Value,
• Manager
• Pool

Next, let’s take a closer look at the multiprocessing.dummy.Pool class.

multiprocessing.dummy.Pool vs multiprocessing.pool.ThreadPool

Perhaps the most significant class in multiprocessing.dummy is Pool.

This is because it returns a ThreadPool class instance that is a wrapper for Pool, except that it uses threads instead of processes for concurrency.

In particular, the Pool function provided by multiprocessing.dummy returns an instance of ThreadPool, which is a subclass of Pool that supports all the same method calls but uses a pool of worker threads rather than worker processes.

— multiprocessing — Process-based parallelism

It is significant because the ThreadPool is a useful, although often neglected class.

It is the only class that is part of multiprocessing.dummy (technically multiprocessing.pool) that warrants mention in the multiprocessing API documentation.

Note A ThreadPool shares the same interface as Pool, which is designed around a pool of processes and predates the introduction of the concurrent.futures module. As such, it inherits some operations that don’t make sense for a pool backed by threads, and it has its own type for representing the status of asynchronous jobs, AsyncResult, that is not understood by any other libraries.

— multiprocessing — Process-based parallelism

ThreadPool is defined in the multiprocessing.pool module, although probably should sit in the multiprocessing.dummy module.

Importantly, it imports the Process class from the dummy module for worker threads, which is an alias for the DummyProcess class that extends threading.Thread.

We can see this in the source code for the dummy module where Pool is defined.

For example:

Creating an instance of multiprocessing.dummy.Pool will return an instance of the multiprocessing.pool.ThreadPool.

We can demonstrate this with a worked example.

Running the example imports the multiprocessing.dummy.Pool class.

An instance of the Pool is created, and the details of the object are reported.

We can see that the Pool is an instance of the multiprocessing.pool.ThreadPool class.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know about the multiprocessing.dummy module in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Kamil Pietrzak on Unsplash