You can log from tasks in the ThreadPoolExecutor by calling a function on the logging module.

In this tutorial, you will discover how to log from tasks in the Python thread pool.

Let’s get started.

Log From Tasks in the ThreadPoolExecutor

The ThreadPoolExecutor in Python provides a pool of reusable threads for executing ad hoc tasks.

You can submit tasks to the thread pool by calling the submit() function and passing in the name of the function you wish to execute on another thread.

Calling the submit() function will return a Future object that allows you to check on the status of the task and get the result from the task once it completes.

You can also submit tasks by calling the map() function and specifying the name of the function to execute and the iterable of items to which your function will be applied.

Tasks often involve interacting with resources and can raise exceptions.

As such, we may wish to log events like warnings, errors, debug, or info messages from within tasks executed by the ThreadPoolExecutor.

If we are logging to a file, database or to the console, we may be concerned as to whether the Python logging supports calls from multiple threads within an application.

That is, is logging from tasks thread safe?

How to Log Directly From Tasks

You can log directly from target task functions when using the ThreadPoolExecutor.

This is because the Python logging module is thread safe.

It is a common question as to whether logging from multiple threads in Python is threads-safe. As such, it is addressed directly in the logging module API documentation; for example:

The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O.

— Thread Safety, logging – Logging facility for Python.

We can log directly from target task functions by calling functions on the logging module at the desired logging level.

For example, we can log an exception as an error:

We might also want to log debug messages, such as whether tasks completed successfully and the time they completed.

Now that we know how to log from tasks executed by the ThreadPoolExecutor, let’s look at a worked example.

Example of Logging From Tasks in the ThreadPoolExecutor

Let’s develop an example that logs directly from target task functions in the ThreadPoolExecutor.

We can define a task that has a unique name received as an argument to the target task function.

The main work for the function is to sleep for a fraction of a second. Once the task is completed, we can log a debug message to indicate that the task has completed. Then return the value for the task.

Tying this together, the task() function below implements our task with debug logging.

By default, Python logging will report log messages to stdout (standard out, i.e. the console), which is fine for our purposes.

In our program, we can set the logging level to debug before we get started.

We can then start a thread pool with ten worker threads using the context manager and submit ten tasks to the pool for execution.

We can then wait for all tasks to complete.

Tying this together, the complete example of logging directly from tasks in the ThreadPoolExecutor is listed below.

Running the example starts the thread pool and submits all ten tasks.

We can then see that as the tasks complete, they log debug messages that are then reported on the command line.

Further Reading

This section provides additional resources that you may find helpful.

Books

• ThreadPoolExecutor Jump-Start, Jason Brownlee, (my book!)
• Concurrent Futures API Interview Questions
• ThreadPoolExecutor Class API Cheat Sheet

I also recommend specific chapters from the following books:

• Effective Python, Brett Slatkin, 2019.
  • See Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPoolExecutor: The Complete Guide
• Python ProcessPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• concurrent.futures - Launching parallel tasks
• threading — Thread-based parallelism

References

• Thread (computing), Wikipedia.
• Futures and promises, Wikipedia.
• Thread pool, Wikipedia.

Takeaways

You now know how to log directly from tasks executed by the ThreadPoolExecutor.

Do you have any questions?
Ask your question below and I will do my best to answer.

Photo by Viktor Bystrov on Unsplash