You can run a task in a child process with a timeout by waiting for the task to complete for a fixed number of seconds by calling the join() method, then terminate the task if the child process is still running via the terminate() method.
In this tutorial, you will discover how to execute a task in a child process with a timeout.
This tutorial was inspired by questions and discussions with Derek K. Thank you deeply! If you have a question about Python concurrency, message me anytime.
Let’s get started.
Need a Task With a Timeout
We can execute target functions in a new child process using the multiprocessing.Process class.
This requires creating an instance of the class and specifying the target function to execute via the “target” argument.
For example:
|
1 2 3 |
... # execute the target function in a child process process = Process(target=function) |
Once created, the process can be started by calling the start() method.
For example:
|
1 2 3 |
... # start the child process process.start() |
This will run the target function in a new child process and leave the caller free to resume other tasks.
If the caller needs to wait for the target function to complete, the join() method on the process can be called which will block until the child process has terminated.
|
1 2 3 |
... # wait for the child process process.join() |
You can learn more about running functions in a child process in the tutorial:
We may want to run a function in a child process, but cancel it after a timeout.
This may be needed if the program requires the task to be completed with a given time limit.
For example, we may issue a task to be completed in a child process and require that it is completed within 10 seconds and if not, to stop or halt the task.
How can we expect a task in a child process with a time limit?
Run loops using all CPUs, download your FREE book to learn how.
How to Execute a Task With a Timeout
We can execute a task in a child process with a time limit by terminating it if the time limit elapses before it is complete.
A task can be executed in a child process as per normal, such as specifying the name of the target function as an argument to the multiprocessing.Process class constructor.
Another process, such as the parent process, can monitor the task in the child process.
For example, it can choose to wait for the task to complete for a fixed time limit by joining it via the join() method and specifying a timeout in seconds.
For example:
|
1 2 3 |
... # wait a fixed interval for the process to complete process.join(timeout=10) |
The caller will block until the child process is done, or the time limit elapses.
You can learn more about the join() method in processes in the tutorial:
We can then check if the child process is still running via the is_alive() method.
For example:
|
1 2 3 4 |
... # check if the process is not yet done if process.is_alive(): # ... |
If the process is still alive, we can terminate it immediately.
This can be achieved by calling the terminate() method.
For example:
|
1 2 3 |
... # terminate the process process.terminate() |
You can learn more about killing and terminating a process in the tutorial:
Now that we know how to execute a task with a timeout, let’s look at some worked examples.
Example of Executing a Task With a Timeout
We can explore an example of executing a task in a child process with a timeout.
In this example, we will define a task to complete in a child process that reports a message, sleeps, and reports a final message.
The task() function below implements this.
|
1 2 3 4 5 6 7 8 |
# task def task(): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # report final message print('Task is done', flush=True) |
Next, we can define a function that encapsulates the behavior of executing a task in a child process with a timeout.
This function takes the timeout in seconds and the name of a target function. It creates a child process to run the target function, runs it, and waits for it to complete for the given timeout. When the child process terminates or the time-out elapses the function will resume. If the child process is still running, it is terminated.
The run_task_with_timeout() function below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # configure the process process = Process(target=function) # start the child process process.start() # wait a fixed interval for the process to complete process.join(timeout=timeout) # check if the process is not yet done if process.is_alive(): # terminate the process process.terminate() |
And that’s it.
Firstly, we can test this function with the normal case where the task has enough time to run.
That is, the timeout is longer than the duration of the task.
|
1 2 3 4 |
# protect the entry point if __name__ == '__main__': # run the task with a timeout run_task_with_timeout(2, task) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFasPython.com # example of executing a task with a timeout, normal case from time import sleep from multiprocessing import Process # task def task(): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # report final message print('Task is done', flush=True) # run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # configure the process process = Process(target=function) # start the child process process.start() # wait a fixed interval for the process to complete process.join(timeout=timeout) # check if the process is not yet done if process.is_alive(): # terminate the process process.terminate() # protect the entry point if __name__ == '__main__': # run the task with a timeout run_task_with_timeout(2, task) |
Running the example calls the custom run_task_with_timeout() function with a timeout of 2 seconds to execute our custom task() function in a child process.
A child process is created to run our custom function, then started.
The main process blocks for 2 seconds while waiting for the task to complete.
The task runs in the child process, reporting a message, sleeping for one second, then reporting a second message.
The child process completes and the main process resumes.
The child process is no longer running, and the run_task_with_timeout() function returns normally.
|
1 2 |
Task is running... Task is done |
Next, we can test the same function, this time with a timeout that is too short.
This will force the child process to be terminated before it has been completed.
|
1 2 3 4 |
# protect the entry point if __name__ == '__main__': # run the task with a timeout run_task_with_timeout(0.5, task) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFasPython.com # example of executing a task with a timeout, timeout case from time import sleep from multiprocessing import Process # task def task(): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # report final message print('Task is done', flush=True) # run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # configure the process process = Process(target=function) # start the child process process.start() # wait a fixed interval for the process to complete process.join(timeout=timeout) # check if the process is not yet done if process.is_alive(): # terminate the process process.terminate() # protect the entry point if __name__ == '__main__': # run the task with a timeout run_task_with_timeout(0.5, task) |
Running the example calls the custom run_task_with_timeout() function with a timeout of 0.5 seconds to execute our custom task() function in a child process.
A child process is created to run our custom function, then started.
The main process blocks for 0.5 seconds waiting for the task to complete.
The task runs in the child process, reporting a message, and sleeping for one second.
The main process resumes and checks the status of the child process.
The child process is still running, so it is terminated.
This highlights how a task can be executed in a child process and terminated automatically if it does not completed within a time limit.
|
1 |
Task is running... |
Next, let’s explore the case where we might want to return a value from the target function.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Executing a Task With a Result With a Timeout
We can explore an example of executing a task in a child process with a timeout that returns a result.
There are a number of ways to return a value from a task executed in a child process.
One approach is to use a shared queue and have the task put the return value on the queue and the caller retrieves the value from the queue.
You can learn more about multiprocessing queues in the tutorial:
Firstly, we can update the task() function to take the shared queue as an argument and share the result on the queue.
In this case, the result is a contrived string for demonstration purposes.
|
1 2 3 4 5 6 7 8 9 10 |
# task def task(queue): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # place a result on the queue queue.put('RESULT') # report final message print('Task is done', flush=True) |
We can then update the run_task_with_timeout() function to create the shared queue and share it with the task in the child process as an argument.
|
1 2 3 4 5 |
... # create the shared queue queue = Queue() # configure the child process process = Process(target=function, args=(queue,)) |
Instead of joining the process, we can wait for the return value on the queue by calling the get() method with a timeout.
If the timeout elapses a queue.Empty exception is raised and we can terminate the child process, otherwise, the return value is retrieved and returned.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
... # wait for the result with a timeout try: # get the result from the queue result = queue.get(timeout=timeout) # return result return result except Empty: # no result in time limit, terminate process.terminate() # return no result return None |
Tying this together, the updated run_task_with_timeout() that can return a result from the task in a child process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # create the shared queue queue = Queue() # configure the child process process = Process(target=function, args=(queue,)) # start the child process process.start() # wait for the result with a timeout try: # get the result from the queue result = queue.get(timeout=timeout) # return result return result except Empty: # no result in time limit, terminate process.terminate() # return no result return None |
We can then test this function with the normal case where the task is able to complete and return a value within the time out.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
# SuperFasPython.com # example of executing a task that returns a value with a timeout from time import sleep from multiprocessing import Process from multiprocessing import Queue from queue import Empty # task def task(queue): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # place a result on the queue queue.put('RESULT') # report final message print('Task is done', flush=True) # run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # create the shared queue queue = Queue() # configure the child process process = Process(target=function, args=(queue,)) # start the child process process.start() # wait for the result with a timeout try: # get the result from the queue result = queue.get(timeout=timeout) # return result return result except Empty: # no result in time limit, terminate process.terminate() # return no result return None # entry point if __name__ == '__main__': # run the task with a timeout result = run_task_with_timeout(2, task) # report result print(result) |
Running the example calls the custom run_task_with_timeout() function with a timeout of 2 seconds to execute our custom task() function in a child process.
A child process is created to run our custom function and pass the shared queue, then started.
The main process blocks for 2 seconds while waiting for the task to complete.
The task runs in the child process, reporting a message, sleeping for one second, putting the result on the queue, then reporting a second message.
The child process completes and the main process resumes.
A result was retrieved from the queue and returned immediately. We do not need to check the status of the child process.
|
1 2 3 |
Task is running... Task is done RESULT |
Next, we can update the example so that the task does not have enough time to complete.
The parent process will wait for the result for a limited time, then terminate the child process and return no result, e.g. a None value.
|
1 2 3 4 5 6 |
# entry point if __name__ == '__main__': # run the task with a timeout result = run_task_with_timeout(0.5, task) # report result print(result) |
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
# SuperFasPython.com # example of executing a task that returns a value with a timeout, timeout case from time import sleep from multiprocessing import Process from multiprocessing import Queue from queue import Empty # task def task(queue): # report a message print('Task is running...', flush=True) # simulate work sleep(1) # place a result on the queue queue.put('RESULT') # report final message print('Task is done', flush=True) # run a task in a new process with a timeout def run_task_with_timeout(timeout, function): # create the shared queue queue = Queue() # configure the child process process = Process(target=function, args=(queue,)) # start the child process process.start() # wait for the result with a timeout try: # get the result from the queue result = queue.get(timeout=timeout) # return result return result except Empty: # no result in time limit, terminate process.terminate() # return no result return None # entry point if __name__ == '__main__': # run the task with a timeout result = run_task_with_timeout(0.5, task) # report result print(result) |
Running the example calls the custom run_task_with_timeout() function with a timeout of 0.5 seconds to execute our custom task() function in a child process.
A child process is created to run our custom function and pass the shared queue, then started.
The main process blocks for 0.5 seconds waiting for the task to complete.
The task runs in the child process, reporting a message, and sleeping for one second.
The main process resumes and a queue.Empty exception is raised and caught.
The main process terminates the child process and returns a None value.
This highlights how to execute a task that returns a value in a child process with a timeout that is terminated before the task is completed.
|
1 2 |
Task is running... None |
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to execute a task in a child process with a timeout.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Remy Lovesy on Unsplash
Leave a Reply