Does the ThreadPool Stop Main From Exiting

Last Updated on October 29, 2022

The ThreadPool will be closed automatically by the Python garbage collector if required.

It will not prevent the program from exiting and the worker threads will not keep running if the main thread exits without closing the pool.

In this tutorial, we will explore what happens if you forget or are unable to close the ThreadPool in your program.

Let’s get started.

Will The ThreadPool Stop The Program From Exiting?

The ThreadPool provides a pool of reusable workers for executing ad hoc tasks with thread-based concurrency.

An instance of the ThreadPool class can be created, specifying the number of workers to create, otherwise, a default number of workers will be created to match the number of logical CPUs in the system.

Once created, ad hoc tasks can be issued to the pool for execution via the apply() method. Multiple tasks may be executed in the book by calling the same function with different arguments via the map() method.

Tasks may be executed asynchronously in the pool via apply_async() and map_async(), allowing the caller to carry on with other tasks.

One concern with the ThreadPool is whether it will prevent the program from exiting if it is not explicitly closed.

For example, we may not close the ThreadPool correctly for many reasons, such as:

• We forget to call the close() or terminate() function.
• An error is raised that prevents a normal exit.

Sometimes in Python concurrency, a running thread or running child process can prevent the main process, and therefore the program from exiting normally.

Additionally, we may be concerned that if the pool is not closed properly, the worker threads will keep running if the main thread of the main process exits.

Will a running ThreadPool prevent the program from exiting normally (or will the program hang)?

ThreadPool Do Not Stop The Program From Exiting

A running ThreadPool does not stop the program from exiting.

This means that we can forget to close a ThreadPool and the main thread will exit normally.

For example:

It also means that if an exception or error is raised and we are unable to close the pool normally, it will not prevent the program from exiting.

For example:

In both cases, the pool will be closed automatically by the garbage collector in the Python interpreter (parent process) and the worker threads will be closed and released.

Although this occurs, it should not be relied upon.

Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called

— multiprocessing — Process-based parallelism

Additionally, the worker threads are created as daemon threads.

This means that they are intended to run in the background and do not prevent the program from exiting.

You can learn more about daemon threads in the tutorial:

• How to Use Daemon Threads in Python

Now that we know that the ThreadPool will not stop the program from exiting and that the python garbage collector will close the pool for us if needed, let’s look at a worked example.

Example of Not Closing a ThreadPool

We can explore the effect of not closing the ThreadPool.

In this example, we will create a ThreadPool with a fixed number of workers. We will then confirm that all worker threads are running and then attempt to exit the program.

Firstly, we can create a ThreadPool with four workers.

Don’t worry if you have more or fewer CPU cores, as we will not be running any tasks.

Next, we can wait a moment for the pool to initialize completely. This is probably not required, but will likely avoid any possible race conditions in checking if the workers are running.

We can then get a list of all active threads for the current process. This will include all workers in the ThreadPool.

This can be achieved via the threading.enumerate() function.

We will then attempt to exit the program normally, with the ThreadPool still running.

Tying this together, the complete example is listed below.

Running the example first creates a ThreadPool with four worker threads.

The main thread then blocks a moment to wait for the pool to initialize completely.

Next, a list of all running threads is retrieved and their details are reported.

We can see the four worker threads and three helper threads in the running ThreadPool.

We can notice that the worker threads and helper threads in the ThreadPool are daemon threads, meaning that they are intended to run in the background.

The main thread then ends normally, and the program exits.

The Python interpreter garbage collector in the process terminates the ThreadPool automatically.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know what happens if you forget or are unable to close the ThreadPool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Jonatan Pie on Unsplash