You can execute a target function that returns a value in a child process without changing the target function by using a wrapper function to handle the transmission of the return value back to the caller automatically.
An alternate approach is to define a callback function and have the wrapper function call the callback function to handle the return value automatically, instead of returning it.
In this tutorial, you will discover how to execute a target function that returns a result in a child process without changing the target function.
Let’s get started.
Need to Return Value From Child Process Without Changing Target Function
We can run a task in a child process.
This can be achieved by creating an instance of the multiprocessing.Process class and specify the target function to execute via the “target” argument and any arguments to the target function via the “args” argument.
For example:
|
1 2 3 |
... # create and configure a child process process = multiprocessing.Process(target=task) |
You can learn more about running a function in a child process in the tutorial:
When executing a task in a child function, we cannot return a result directly.
Instead, we must simulate a return value, such as by using a shared queue.
You can learn more about simulating a return value from a child process in the tutorial:
The problem with this approach is that it requires that the target function be changed to put the result on the shared queue or shared pipe, rather than return it.
Sometimes we cannot change the target function because we don’t have direct access to it, for example, it may be part of a third-party library. Alternatively, the target function may have other uses, requiring that the interface does not change.
How can we return a value from a target function executed in a child process without changing the target function?
Run loops using all CPUs, download your FREE book to learn how.
How to Simulate Return Value From Child Process
We can simulate a return value from a child process without changing the target function by using a wrapper function.
We can define a new function that executes the target function in a child process, collects the return value, and then transmits it back to the calling process via a queue or pipe.
For example:
|
1 2 3 4 5 6 |
# helper function for running a target function in a child process def run_in_process_helper(function, queue): # run the function and get the result result = function() # put the result on the queue queue.put(result) |
The caller can then retrieve the return value directly from the shared queue.
Alternatively, we can hide the queue completely in a second function responsible for creating, configuring, and starting the child process, calling the wrapper function, and then returning the value retrieved from the shared queue.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# run a function with a return value in a child process def run_in_process(function): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # return the return value return result |
This encapsulates the execution of the task in the child process and the mechanism for retrieving the return value.
Another approach that could be used is to pass a callback function used to handle the result.
The callback function can be passed and executed in the child process once the result from the target function has been retrieved.
For example:
|
1 2 3 4 5 6 |
# helper function for running a target function in a child process def run_in_process_helper(function, callback): # run the function and get the result result = function() # call the callback with the result callback(result) |
Alternatively, the callback function could be executed in the calling process, once the result has been retrieved from the shared queue.
For example:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# run a function with a return value in a child process def run_in_process(function, callback): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # execute the callback callback(result) |
Now that we have explored some approaches for executing a target function that returns a result in a child process without changing the target function, let’s look at some worked examples.
Example of Simulated Return Value From Child Process
We can explore how to return a value from a target function executed in a child process using a wrapper function.
In this example, we will first define a wrapper function to execute in a child process.
The wrapper function takes the name of the function to execute and the shared queue. It executes the target function and puts the result on the shared queue.
|
1 2 3 4 5 6 |
# helper function for running a target function in a child process def run_in_process_helper(function, queue): # run the function and get the result result = function() # put the result on the queue queue.put(result) |
Next, we can define a function to drive the overall process.
This function takes the name of the target function. It creates the shared queue, then creates and configures a child process to execute the wrapper function with the name of the target function and shared queue as arguments. It starts the child process, then waits for the result on the shared queue, which is then returned.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# run a function with a return value in a child process def run_in_process(function): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # return the return value return result |
We can then define our actual target function.
In this case, our target task function does not take any arguments, blocks for one second to simulate work, then returns an integer value.
|
1 2 3 4 5 6 |
# task that returns a value def task(): # block for a moment to simulate work sleep(1) # return some data return 111 |
Importantly, the target function can return a value normally and is not concerned without the return value being transmitted from the child process back to the calling process.
Finally, we can call the helper function to run the target function in the child process, passing it the name of the target function to execute.
|
1 2 3 4 5 6 |
# protect the entry point if __name__ == '__main__': # run a target function with a return value in a child process result = run_in_process(task) # report the result print(f'got: {result}') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
# SuperFastPython.com # example of simulating a return value from a child process from time import sleep from multiprocessing import Process from multiprocessing import Queue # helper function for running a target function in a child process def run_in_process_helper(function, queue): # run the function and get the result result = function() # put the result on the queue queue.put(result) # run a function with a return value in a child process def run_in_process(function): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # return the return value return result # task that returns a value def task(): # block for a moment to simulate work sleep(1) # return some data return 111 # protect the entry point if __name__ == '__main__': # run a target function with a return value in a child process result = run_in_process(task) # report the result print(f'got: {result}') |
Running the example calls the run_in_process() function with the name of the target function.
This creates the shared queue, then creates and configures a new child process instance to execute the run_in_process_helper() wrapper function and passes it the name of the target function and the shared queue.
The run_in_process() then starts the child process and blocks, waiting for the result to be retrieved from the shared queue.
The child process executes the target function and then places the result on the shared queue.
The main process retrieves the result from the queue and returns it back to the caller, where it is reported.
This highlights how we can use wrapper functions to execute a target function that returns a result in a child process, without making any changes to the target function.
|
1 |
got: 111 |
Next, let’s explore an alternative approach that uses a callback function to handle the result of the target function, instead of returning it to the caller.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Simulated Return Value From Child Process With Callback
We can explore the case of handling the return value from the target function with a callback function, instead of returning it to the caller.
In this example, we will define a callback function that takes the return value from the target function, and in this case simply reports it.
|
1 2 3 4 |
# callback function for handling the return value def custom_callback(result): # report the result print(f'got: {result}') |
We can then update the run_in_process() function to take the name of the callback function as an argument, and then call the callback function in the calling process, once the result from the target function has been retrieved from the shared queue.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# run a function with a return value in a child process def run_in_process(function, callback): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # execute the callback callback(result) |
Finally, we can call the run_in_process() with the name of the target function and the name of the callback function.
|
1 2 3 4 |
# protect the entry point if __name__ == '__main__': # run a target function with a return value in a child process run_in_process(task, custom_callback) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
# SuperFastPython.com # example of simulating a return value from a child process with a callback from time import sleep from multiprocessing import Process from multiprocessing import Queue # helper function for running a target function in a child process def run_in_process_helper(function, queue): # run the function and get the result result = function() # put the result on the queue queue.put(result) # run a function with a return value in a child process def run_in_process(function, callback): # create the shared queue queue = Queue() # configure the child process process = Process(target=run_in_process_helper, args=(function, queue)) # start the child process process.start() # wait for the return value on the queue result = queue.get() # execute the callback callback(result) # task that returns a value def task(): # block for a moment to simulate work sleep(1) # return some data return 111 # callback function for handling the return value def custom_callback(result): # report the result print(f'got: {result}') # protect the entry point if __name__ == '__main__': # run a target function with a return value in a child process run_in_process(task, custom_callback) |
Running the example calls the run_in_process() function with the name of the target function and the name of the callback function.
This creates the shared queue, then creates and configures a new child process instance to execute the run_in_process_helper() wrapper function and passes it the name of the target function and the shared queue.
The run_in_process() then starts the child process and blocks, waiting for the result to be retrieved from the shared queue.
The child process executes the target function and then places the result on the shared queue.
The main process retrieves the result from the queue and returns and then calls the callback function with the return value.
The callback function executes and reports the return value.
This highlights how we can use a callback function to handle the return value from a target function executed in a child process, without changing the target function.
|
1 |
got: 111 |
Next, let’s explore how we might execute the target function in a child process asynchronously and handle the result using a callback function.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Simulated Return Value From Child Process Asynchronous Callback
A benefit of executing a target function in a child process is that it does not have to block the caller.
A downside of the two designs presented so far is that they block the caller until the target function executed in the child process is complete.
In this example, we can update the previous example that handles the result from the target function using a callback function, and execute the target function asynchronously. This allows the caller to resume and execute other tasks if needed.
This can be achieved by executing the callback function using the wrapper function that is executed in the child function. The callback function can be passed to the wrapper function.
Because the result is no longer needed in the calling process, we no longer need the shared queue.
|
1 2 3 4 5 6 |
# helper function for running a target function in a child process def run_in_process_helper(function, callback): # run the function and get the result result = function() # call the callback with the result callback(result) |
We can then update the run_in_process() function to pass the callback function to the wrapper function and no longer create or wait on a shared queue.
Instead, it starts the child process and returns immediately, without blocking.
|
1 2 3 4 5 6 |
# run a function with a return value in a child process def run_in_process(function, callback): # configure the child process process = Process(target=run_in_process_helper, args=(function, callback)) # start the child process process.start() |
And that’s it.
The complete example of executing the target function in a child process asynchronously and handling the result with a callback function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
# SuperFastPython.com # example of asynchronous a return value from a child process with a callback from time import sleep from multiprocessing import Process from multiprocessing import Queue # helper function for running a target function in a child process def run_in_process_helper(function, callback): # run the function and get the result result = function() # call the callback with the result callback(result) # run a function with a return value in a child process def run_in_process(function, callback): # configure the child process process = Process(target=run_in_process_helper, args=(function, callback)) # start the child process process.start() # task that returns a value def task(): # block for a moment to simulate work sleep(1) # return some data return 111 # callback function for handling the return value def custom_callback(result): # report the result print(f'got: {result}') # protect the entry point if __name__ == '__main__': # run a target function with a return value in a child process run_in_process(task, custom_callback) # report a message print('Main is done') |
Running the example calls the run_in_process() function with the name of the target function and the name of the callback function.
This creates and configures a new child process instance to execute the run_in_process_helper() wrapper function and passes it the name of the target function and the name of the callback function. The child process is started and the run_in_process() returns immediately.
The main process is then done, but the program does not exit because the child process is still executing and is not a daemon process.
The run_in_process_helper() wrapper function executes in the child process. It calls the target function, retrieves the result, then calls the callback function with the result.
The callback function executes and reports the return value.
This highlights how we can execute a target function asynchronously in a child process and handle the return value using a callback, all without having to make any changes to the target function.
|
1 2 |
Main is done got: 111 |
Takeaways
You now know how to execute a target function that returns a result in a child process without changing the target function.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Media Studio Hong Kong on Unsplash
Leave a Reply