You may encounter one among a number of common errors when using the ThreadPool in Python.
These errors are often easy to identify and often involve a quick fix.
In this tutorial, you will discover the common errors when using the ThreadPool in Python and how to fix each in turn.
Let’s get started.
Common Errors When Using ThreadPool
There are a number of common errors when using the ThreadPool.
These errors are typically made because of bugs introduced by copy-and-pasting code, or from a slight misunderstanding of how the ThreadPool works.
We will take a closer look at some of the more common errors made when using the ThreadPool, such as:
- Using a Function Call in submit()
- Using a Function Call in map()
- Incorrect Function Signature for map()
- Incorrect Function Signature for Future Callbacks
- Tasks Fail Silently
- Joining Pool While Running
- Issuing Tasks to a Closed Pool
Do you have an error using the ThreadPool?
Let me know in the comments so I can recommend a fix and add the case to this tutorial.
Run loops using all CPUs, download your FREE book to learn how.
Error 1: Using a Function Call in apply_async()
A common error is to call your function when using the apply_async() function.
For example:
|
1 2 3 |
... # issue the task result = pool.apply_async(task()) |
A complete example of this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# SuperFastPython.com # example of calling submit with a function call from time import sleep from multiprocessing.pool import ThreadPool # custom function executed in another thread def task(): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # start the thread pool with ThreadPool() as pool: # issue the task result = pool.apply_async(task()) # get the result value = result.get() print(value) |
Running this example will fail with an error.
|
1 2 3 |
Traceback (most recent call last): ... TypeError: 'str' object is not callable |
You can fix the error by updating the call to apply_async() to take the name of your function and any arguments, instead of calling the function in the call to execute.
For example:
|
1 2 3 |
... # issue the task result = pool.apply_async(task) |
Error 2: Using a Function Call in map()
A common error is to call your function when using the map() function.
For example:
|
1 2 3 4 |
... # issue all tasks for result in pool.map(task(), range(5)): print(result) |
A complete example of this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of calling map with a function call from time import sleep from multiprocessing.pool import ThreadPool # custom function executed in another thread def task(value): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # start the thread pool with ThreadPool() as pool: # issue all tasks for result in pool.map(task(), range(5)): print(result) |
Running the example results in a TypeError.
|
1 2 3 |
Traceback (most recent call last): ... TypeError: task() missing 1 required positional argument: 'value' |
This error can be fixed by changing the call to map() to pass the name of the target task function instead of a call to the function.
|
1 2 3 4 |
... # issue all tasks for result in pool.map(task, range(5)): print(result) |
Free Python ThreadPool Course
Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.
Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously
Error 3: Incorrect Function Signature for map()
Another common error when using map() is to provide no second argument to the function, e.g. the iterable.
For example:
|
1 2 3 4 |
... # issue all tasks for result in pool.map(task): print(result) |
A complete example of this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of calling map without an iterable from time import sleep from multiprocessing.pool import ThreadPool # custom function executed in another thread def task(value): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # start the thread pool with ThreadPool() as pool: # issue all tasks for result in pool.map(task): print(result) |
Running the example does not issue any tasks to the ThreadPool as there was no iterable for the map() function to iterate over.
Running the example results in a TypeError.
|
1 2 3 |
Traceback (most recent call last): ... TypeError: map() missing 1 required positional argument: 'iterable' |
The fix involves providing an iterable in the call to map() along with your function name.
|
1 2 3 4 |
... # issue all tasks for result in pool.map(task, range(5)): print(result) |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Error 4: Incorrect Function Signature for Callbacks
Another common error is forgetting to include the result in the signature for the callback function when issuing tasks asynchronously.
For example:
|
1 2 3 |
# result callback function def handler(): print(f'Callback got: {result}') |
A complete example of this error is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# SuperFastPython.com # example of a callback function for apply_async() from time import sleep from multiprocessing.pool import ThreadPool # result callback function def handler(): print(f'Callback got: {result}') # custom function executed in another thread def task(): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool() as pool: # issue tasks to the thread pool result = pool.apply_async(task, callback=handler) # get the result value = result.get() print(value) |
Running this example will result in an error when the callback is called by the ThreadPool.
This will break the ThreadPool and the program will have to be killed manually with a Control-C.
|
1 2 3 4 |
Exception in thread Thread-11: Traceback (most recent call last): ... TypeError: handler() takes 0 positional arguments but 1 was given |
Fixing this error involves updating the signature of your callback function to include the result from the task.
|
1 2 3 |
# result callback function def handler(result): print(f'Callback got: {result}') |
You can learn more about using callback functions with asynchronous tasks in the tutorial:
This error can also happen with the error callback and forgetting to add the error as an argument in the error callback function.
Error 5: Tasks Fail Silently
A common error is when tasks are issued to the ThreadPool but fail silently.
The expected result or output does not occur and no message is provided by the ThreadPool.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# SuperFastPython.com # example of asynchronous tasks failing silently in the thread pool from time import sleep from multiprocessing.pool import ThreadPool # task executed in a worker thread def task(): # block for a moment sleep(1) # fail raise Exception('Something bad happened') # report a message print(f'Task done') # protect the entry point if __name__ == '__main__': # create and configure the thread pool with ThreadPool() as pool: # issue an asynchronous task into the thread pool result = pool.apply_async(task) # wait for all tasks to finish result.wait() |
Running the example results in no message from the task or the ThreadPool itself.
In order to trigger the error, we must attempt to retrieve the result from the asynchronous task.
For example:
|
1 2 3 |
... # get the result result.get() |
Alternatively, we can register an error callback function with the task.
For example:
|
1 2 3 4 5 6 7 |
# error callback function def callback(error): print(f'Error: {error}') ... # issue task and register an error callback result = pool.apply_async(task, error_callback=callback) |
You can learn more about tasks failing silently in the ThreadPool in the tutorial:
Error 6: Joining Pool While Running
Another common error occurs when attempting to join the ThreadPool in order to wait for all running tasks to complete.
This is achieved by calling the join() method.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# SuperFastPython.com # example of an error while joining the pool from time import sleep from multiprocessing.pool import ThreadPool # custom function executed in another thread def task(): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # start the thread pool with ThreadPool() as pool: # issue the task result = pool.apply_async(task) # wait for all tasks to finish pool.join() |
Running the example results in an exception.
|
1 2 3 |
Traceback (most recent call last): ... ValueError: Pool is still running |
This error occurs because you attempt to join the ThreadPool while it is still running.
You can fix this error by first closing the pool by calling close() or terminate().
For example:
|
1 2 3 4 5 |
... # close the pool pool.close() # wait for all tasks to finish pool.join() |
You can learn more about joining the ThreadPool in the tutorial:
Error 7: Issuing Tasks to a Closed Pool
A common error occurs when attempting to issue tasks to the ThreadPool.
This can happen if the pool was inadvertently closed before the task was issued.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of issuing tasks to a pool that is closed from time import sleep from multiprocessing.pool import ThreadPool # custom function executed in another thread def task(): # block for a moment sleep(1) return 'all done' # protect the entry point if __name__ == '__main__': # start the thread pool with ThreadPool() as pool: # issue the task result = pool.apply_async(task) # close the pool pool.close() # wait for all tasks to finish pool.join() # issue another task result = pool.apply_async(task) |
Running the example results in an exception.
|
1 2 3 |
Traceback (most recent call last): ... ValueError: Pool not running |
This error occurs because you have closed the ThreadPool and then attempted to issue tasks to execute.
The pool cannot execute tasks if it is not running.
You must start a new pool or issue tasks before closing the pool.
You can learn more about correctly shutting down the ThreadPool in the tutorial:
Further Reading
This section provides additional resources that you may find helpful.
Books
- Python ThreadPool Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- ThreadPool PDF Cheat Sheet
I also recommend specific chapters from the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python ThreadPool: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python Threading: The Complete Guide
APIs
References
Takeaways
You now know about the common errors when using the ThreadPool in Python.
Do you have any questions?
Ask your question in the comments below and I will do my best to answer.
Photo by ZHENYU LUO on Unsplash
Leave a Reply