We can log from tasks directly in the ThreadPool by calling a function on the logging module.

In this tutorial, you will discover how to log from tasks in the Python ThreadPool.

Let’s get started.

Need to Log From Tasks in the ThreadPool

The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.

A thread pool object which controls a pool of worker threads to which jobs can be submitted.

— multiprocessing — Process-based parallelism

The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.

Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.

A ThreadPool can be configured when it is created, which will prepare the new threads.

We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().

Tasks often involve interacting with resources and can raise exceptions.

As such, we may wish to log events like warnings, errors, debug, or info messages from within tasks executed by the ThreadPool.

If we are logging to a file, database, or to the console, we may be concerned as to whether the Python logging supports calls from multiple threads within an application.

That is, is logging from tasks thread-safe?

Call Logging Directly From Tasks

You can log directly from target task functions when using the ThreadPool.

This is because the Python logging module is thread-safe.

It is a common question as to whether logging from multiple threads in Python is threads-safe. As such, it is addressed directly in the logging module API documentation; for example:

The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.

— Thread Safety, logging – Logging facility for Python.

We can log directly from target task functions by calling functions on the logger module at the desired logging level.

For example, we can log an exception as an error:

We might also want to log debug messages, such as whether tasks were completed successfully and the time they were completed.

Now that we know how to log from tasks executed by the ThreadPool, let’s look at a worked example.

How to Log From Tasks in the ThreadPool

Let’s develop an example that logs directly from target task functions in the ThreadPool.

We can define a task that has a unique name received as an argument to the target task function.

The main work for the function is to sleep for a fraction of a second. Once the task is completed, we can log a debug message to indicate that the task has been completed. Then return the value for the task.

Tying this together, the task() function below implements our task with debug logging.

By default, Python logging will report log messages to stdout (standard out, i.e. the console), which is fine for our purposes.

In our program, we can set the logging level to debug before we get started.

We can then configure and start a ThreadPool with the default number of worker threads using the context manager.

You can learn more about the ThreadPool context manager interface in the tutorial:

• How to Use the ThreadPool Context Manager

We will then issue ten tasks asynchronously via the map_async() function which returns immediately with an AsyncResult object. We then wait on this object for all issued tasks to be completed.

You can learn more about the map_async() method in the tutorial:

• How to Use ThreadPool map_async() in Python

Tying this together, the complete example of logging directly from tasks in the ThreadPool is listed below.

Running the example starts the ThreadPool and issues all ten tasks.

We can then see that as the tasks are completed, they log debug messages that are then reported on the command line.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to log directly from tasks executed by the ThreadPool.

Do you have any questions about this tutorial?
Ask your question in the comments below and I will do my best to answer.

Photo by Andrew Palmer on Unsplash