You cannot use threading.Lock mutex locks to protect critical sections in asyncio programs.
In this tutorial, you will discover what happens if we try to use threading mutex locks in asyncio programs in Python.
Let’s get started.
Can We Use threading.Lock instead of asyncio.Lock
The threading.Lock provides a mutex that can be used to protect a critical section of code from concurrent execution by multiple threads.
You can learn more about the threading.Lock in the tutorial:
Similarly, the asyncio.Lock provides a mutex to protect critical sections from coroutines. The asyncio.Lock is not thread-safe, meaning it is only appropriate for use with coroutines.
class asyncio.Lock: Implements a mutex lock for asyncio tasks. Not thread-safe.
— Asyncio Synchronization Primitives
You can learn more about the asyncio.Lock in the tutorial:
We may consider whether we can use a threading Lock in an asyncio program.
Can a threading Lock be used to protect a critical section from concurrent execution by multiple threads and multiple coroutines?
Let’s explore this question.
In the next section, we will try to protect a critical section in an asyncio program using a threading.Lock.
Run loops using all CPUs, download your FREE book to learn how.
Example of threading.Lock in Asyncio
We can explore what happens if we use a threading.Lock in an asyncio program.
We can define a task coroutine that is passed the lock. The lock is acquired to protect a critical section that reports a message and sleeps a moment.
The main coroutine creates the lock, creates a number of coroutines that will share and make use of the lock concurrently, and then executes all coroutines and waits for them to finish.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of using a threading Lock in an asyncio program from random import random from threading import Lock import asyncio # task coroutine with a critical section async def task(lock, num, value): # acquire the lock to protect the critical section with lock: # report a message print(f'>coroutine {num} got the lock, sleeping for {value}') # block for a moment await asyncio.sleep(value) # entry point async def main(): # create a shared lock lock = Lock() # create many concurrent coroutines coros = [task(lock, i, random()) for i in range(10)] # execute and wait for tasks to complete await asyncio.gather(*coros) # run the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutines and executes it as the entry point into the asyncio program.
The main() coroutine runs, first creating the threading mutex lock.
A list comprehension is then used to create 10 coroutines, each passing the lock, a unique integer, and a random number.
The main() coroutine is then suspended and awaits the execution of all coroutines.
The coroutines executed. The first acquires the lock, reports a message, then sleeps a moment.
This allows another coroutine to resume. It attempts to acquire the lock.
The program freezes with a deadlock.
It must be killed manually using CONTROL-C.
A sample of the program output is listed below, showing the message reported by the first coroutine only.
|
1 |
>coroutine 0 got the lock, sleeping for 0.10450054983534196 |
What happened?
A thread can only acquire a mutex lock once.
Attempting to acquire a lock that itself already holds, results in a deadlock.
It is irrelevant whether the thread attempted to acquire the lock again in a coroutine or not.
We can make this clearer with a more focused example.
Example of A Thread Acquiring a Lock Twice
We can explore a simpler example of a thread attempting to acquire the same lock twice.
In this example, we define a function to execute in a new thread.
The function takes the lock as an argument, reports a message, acquires the lock, reports a message again, attempts to acquire it again, then reports a final message.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of a thread attempting to acquire the same lock more than once from threading import Thread from threading import Lock # function executed in a new thread def task(lock): # report a message print('Acquiring lock') # acquire the lock with lock: # report a message print('Acquiring lock again') # acquire the lock with lock: # report message print('All done') # create the lock lock = Lock() # create the thread thread = Thread(target=task, args=(lock,)) # start the thread thread.start() # wait for the thread to finish thread.join() |
Running the example first creates the shared lock.
A new thread is created and configured to execute the task() function and is passed the shared lock as an argument. The thread is started and the main thread blocks until the new thread is terminated.
The new thread runs.
A message is reported and the lock is acquired.
A second message is reported and the thread attempts to acquire the lock again.
This fails. The thread waits forever for the lock to become available. It does not become available because it already holds the lock.
The program must be terminated manually, e.g. via CONTROL-C.
|
1 2 |
Acquiring lock Acquiring lock again |
The threading.Lock is not reentrant, meaning a thread may not acquire the same lock again.
Doing so results in a deadlock.
You can learn more about this in the tutorial:
This is what happens in our asyncio program in the previous section.
The thread attempts to acquire the same lock twice, resulting in a deadlock.
This highlights why we cannot use a threading.Lock in an asyncio program and instead must use an asyncio.Lock.
Next, let’s consider using a reentrant threading Lock.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of threading Reentrant Lock in Asyncio
A threading.Lock is not reentrant, meaning a thread cannot acquire the lock, then acquire it again.
Nevertheless, the threading API provides a reentrant lock via the threading.RLock class.
You can learn more about the threading.RLock class in the tutorial:
This lock can be acquired by a thread, then acquired again.
Can we use this to protect critical sections in our coroutines?
The example below updates the asyncio program to use the reentrant lock instead of the non-reentrant lock.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of using a threading RLock in an asyncio program from random import random from threading import RLock import asyncio # task coroutine with a critical section async def task(lock, num, value): # acquire the lock to protect the critical section with lock: # report a message print(f'>coroutine {num} got the lock, sleeping for {value}') # block for a moment await asyncio.sleep(value) # entry point async def main(): # create a shared lock lock = RLock() # create many concurrent coroutines coros = [task(lock, i, random()) for i in range(10)] # execute and wait for tasks to complete await asyncio.gather(*coros) # run the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutines and executes it as the entry point into the asyncio program.
The main() coroutine runs, first creating the threading reentrant mutex lock.
A list comprehension is then used to create 10 coroutines, each passing the lock, a unique integer and a random number.
The main() coroutine is then suspended and awaits the execution of all coroutines.
The coroutines executed. The first acquires the lock, reports a message, then sleeps a moment.
This allows another coroutine to resume. It attempts to acquire the lock and does. It reports a message, then sleeps for a moment.
The next coroutine runs, and so on.
All coroutines execute as fast as they are able and the lock is acquired immediately each time.
|
1 2 3 4 5 6 7 8 9 10 |
>coroutine 0 got the lock, sleeping for 0.4568901431793604 >coroutine 1 got the lock, sleeping for 0.29864186932986536 >coroutine 2 got the lock, sleeping for 0.09374909488228744 >coroutine 3 got the lock, sleeping for 0.4073937933735554 >coroutine 4 got the lock, sleeping for 0.5205997886127717 >coroutine 5 got the lock, sleeping for 0.38918445371545385 >coroutine 6 got the lock, sleeping for 0.17515599302225426 >coroutine 7 got the lock, sleeping for 0.4054819258792076 >coroutine 8 got the lock, sleeping for 0.36510786246727756 >coroutine 9 got the lock, sleeping for 0.13441856138988206 |
What happened?
The lock is thread-reentrant.
Asyncio uses a single thread to execute all coroutines.
Therefore, the same thread attempts to acquire the reentrant lock each time a coroutine reaches the critical section. This is allowed, the thread reenters the lock, and the critical section is not protected from concurrent execution.
This highlights that we cannot use threading mutex locks, reentrant, or otherwise to protect critical sections from concurrent execution by coroutines.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Mixing Threads and Coroutines
We now know that we cannot use threading concurrency primitives to protect coroutines from race conditions.
This supports the need for asyncio-specific mutex locks.
What if we wanted to mix coroutines and threads?
For example, we may have a resource used by multiple coroutines across multiple threads.
That is, one asyncio event loop per thread, and multiple coroutines per event loop, all accessing the same critical section or resource.
I would strongly recommend separating the concerns of coroutines and threads.
For example, perhaps access to the resource can be limited to threads, such as in a thread pool, protected by a thread-safe mutex and coordinated by coroutines.
Or the verse, accessed only via coroutines and coordinated by threads, such as via a message queue.
If you are exploring this use case, let me know and I will develop some further examples.
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know that threading mutex locks cannot be used to protect critical sections in coroutines.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Tahlia Doyle on Unsplash
Leave a Reply