Mutual exclusion locks or mutex locks for short can be used to protect critical sections of code from concurrent execution.

It is possible to suffer race conditions if two or more coroutines operate upon the same variables, or if tasks are executed out of order.

We can use locks in asyncio via the asyncio.Lock class to define atomic blocks of code where execution is serialized, e.g. made sequential.

In this tutorial, you will discover how to use mutex locks to protect critical sections in asyncio.

After completing this tutorial, you will know:

• The importance of mutex locks in concurrent programming.
• How we may suffer race conditions in asyncio with coroutines.
• How to use mutex locks in asyncio programs.

Let’s get started.

What is a Mutual Exclusion Lock

A mutual exclusion lock or mutex lock is a synchronization primitive intended to prevent a race condition.

A race condition is a concurrency failure case when two units of concurrency (processes, threads, or coroutines) run the same code and access or update the same resource (e.g. data variables, stream, etc.) leaving the resource in an unknown and inconsistent state.

Although race conditions and mutex locks are often described in the context of threads, they are just as relevant and applicable to coroutines in asyncio.

Race conditions often result in unexpected behavior of a program and/or corrupt data.

These sensitive parts of code that can be executed by multiple threads concurrently and may result in race conditions are called critical sections. A critical section may refer to a single block of code, but it also refers to multiple accesses to the same data variable or resource from multiple functions.

The most commonly used mechanism for ensuring mutual exclusion is a mutual exclusion lock or mutex, or simply lock. A mutex is a special type of object that has support in the underlying hardware. The basic idea is that each critical section is protected by a lock.

— PAGE 53, AN INTRODUCTION TO PARALLEL PROGRAMMING, 2020.

A mutex lock can be used to ensure that only one thread at a time executes a critical section of code at a time, while all other threads trying to execute the same code must wait until the currently executing thread is finished with the critical section and releases the lock.

While one thread “owns” the lock—that is, has returned from a call to the lock function but hasn’t yet called the unlock function—any other thread attempting to execute the code in the critical section will wait in its call to the lock function.

— PAGE 53, AN INTRODUCTION TO PARALLEL PROGRAMMING, 2020.

Each thread must attempt to acquire the lock at the beginning of the critical section. If the lock has not been obtained, then a thread will acquire it and other threads must wait until the thread that acquired the lock releases it.

Before a thread can execute the code in the critical section, it must “obtain” the mutex by calling a mutex function, and when it’s done executing the code in the critical section, it should “relinquish” the mutex by calling an unlock function.

— PAGE 53, AN INTRODUCTION TO PARALLEL PROGRAMMING, 2020.

If the lock has not been acquired, we might refer to it as being in the “unlocked” state. Whereas if the lock has been acquired, we might refer to it as being in the “locked” state.

• Unlocked: The lock has not been acquired and can be acquired by the next thread that makes an attempt.
• Locked: The lock has been acquired by one thread and any thread that makes an attempt to acquire it must wait until it is released.

Locks are created in the unlocked state.

Now that we know what a mutex lock is, let’s take at why we need a lock in an asyncio program.

Why Do We Need a Lock in AsyncIO?

All coroutines for an event loop run in one thread, and a thread runs in one process.

Therefore, coroutines are both thread-safe and process-safe, as long as they are using resources that are not used by other threads or processes.

Nevertheless, coroutines may access state or resources that are not coroutine-safe.

How can this be, especially considering only one coroutine can run at a time within an event loop?

Consider a block of code that may be executed by multiple coroutines concurrently and has at least a point where the coroutine is suspended, e.g. an await expression or similar.

This block of code may manipulate program state, and data, or access an external resource.

If multiple coroutines execute this block concurrency, then the program state, data, or external resource may be left in an inconsistent or corrupt state, or data may be lost.

That is, coroutines may have critical sections that span many lines of code and must be protected from concurrent execution.

An obvious example is multiple coroutines reading and writing to/from the same socket.

A less obvious example is multiple coroutines that update program state while executing some task.

A necessary condition for coroutines to suffer a race condition in an asyncio program is that the critical section must suspend, allowing one or more other coroutines to execute the same critical section, or manipulate the same state or resources.

You can see an example of an asyncio race condition in the tutorial:

• Asyncio Race Conditions

Now that we know we need locks in asyncio programs, let’s look at how to use the asyncio.Lock class.

How to Use the asyncio.Lock Class

Python provides a mutual exclusion lock for coroutines via the asyncio.Lock class.

Critically, this lock is neither thread-safe nor process-safe. It cannot be used to protect critical sections from multiple threads or multiple processes.

Implements a mutex lock for asyncio tasks. Not thread-safe. An asyncio lock can be used to guarantee exclusive access to a shared resource.

— Asyncio Synchronization Primitives

An instance of the lock must be created and shared among coroutines in order to protect a resource or critical sections.

For example:

The lock can be acquired by calling the acquire() coroutine.

This is done at the beginning of a critical section.

This call must be awaited because it may block if the lock is currently being held by another coroutine that is currently suspended.

This will suspend the calling coroutine and potentially allow the coroutine holding the lock to progress,

For example:

Once the critical section is completed, the lock can be released.

This is not a blocking call.

The lock must always be released.

If not, other coroutines awaiting it will never be able to acquire the lock and the program may be deadlocked.

As such, it may be a good practice to enclose the critical section in a try-finally block, ensuring the release() function is always recalled regardless of how the critical section block is exited.

For example:

We can achieve the same effect using the context manager interface.

This is preferred as the lock always be released automatically as soon as the enclosed critical section block is exited.

Recall that we must use an asynchronous context manager via the “async with” expression.

For example:

You can learn more about the “async with” expression in the tutorial:

• What is Asyncio async with

You can learn more about asynchronous context managers in the tutorial:

• Asynchronous Context Managers in Python

Finally, we can also check if the lock is currently acquired by a coroutine via the locked() function.

For example:

The asyncio.Lock class is different from the threading.Lock and multiprocessing.Lock classes in that it does not support a timeout when acquiring the lock, or the ability to acquire the lock without blocking.

You can more about those classes in the tutorials:

• Threading Mutex Lock in Python
• Multiprocessing Lock in Python

Now that we know how to use the lock class, let’s look at a worked example.

Example of Using the asyncio.Lock

We can develop an example to demonstrate how to use the mutex lock.

First, we can define a target task coroutine that takes a lock as an argument and uses the lock to protect a contrived critical section.

In this case, the critical section involves reporting a message and blocking for a fraction of a second.

It is important that the critical section suspend in some way, in order to allow other coroutines to run and potentially interact with the same resources or critical sections that require the lock.

The task() coroutine below implements this.

We can then create one instance of the asyncio.Lock shared among the coroutines.

This can be achieved in the main() coroutine, used as the entry point to the program.

We can then create a large number of coroutines and pass the shared lock. Each coroutine will have a unique integer argument and a random floating point value between 0 and 1, which will be how long the coroutine will sleep while holding the lock.

The coroutines are created in a list comprehension and provided to the asyncio.gather() function. The main() coroutine will then block until all coroutines are complete.

The main() coroutine that implements this is listed below.

The asyncio.gather() function cannot take a list of coroutines directly, instead, we must unpack the expressions using the star operator. This can look confusing if this is the first time you are seeing the asyncio.gather() function.

You can learn more about asyncio.gather() in the tutorial:

• Asyncio gather() to Run Many Tasks Concurrently

Tying this together, the complete example of using a lock is listed below.

Running the example first creates the main() coroutine and uses it as the entry point into the asyncio program.

The main() coroutine runs, first creating the shared lock.

It then creates a list of coroutines, each is passed the shared lock, a unique integer, and a random floating point value.

The list of coroutines is passed to the gather() function and the main() coroutine suspends until all coroutines are completed.

A task coroutine executes, acquires the lock, reports a message, then awaits the sleep, suspending.

Another coroutine resumes. It attempts to acquire the lock and is suspended, while it waits. This process is repeated with many if not all coroutines.

The first coroutines resumes, exits the block, and releases the lock automatically via the asynchronous context manager.

The first coroutine to wait on the lock resumes, acquires the lock, reports a message, and sleeps.

This process repeats until all coroutines are given an opportunity to acquire the lock, execute the critical section and terminate.

Once all tasks terminate, the main() coroutine resumes and terminates, closing the program.

The output from the program will differ each time it is run given the use of random numbers.

Common Questions

This section lists common questions about the asyncio.Lock mutex in Python.

Is asyncio.Lock coroutine-safe?

Yes.

An asyncio.Lock object can be used to protect a resource or critical section from concurrently executing coroutines within one event loop.

Is asyncio.Lock thread-safe?

No.

An asyncio.Lock object cannot protect a resource or critical section from execution by multiple threads.

Instead, you should use a threading.Lock object.

See:

• Threading Mutex Lock in Python

Is asyncio.Lock process-safe?

No.

An asyncio.Lock object cannot protect a resource or critical section from execution by multiple processes.

Instead, you should use a multiprocessing.Lock object.

See:

• Multiprocessing Lock in Python

Can asyncio Coroutines Have Race Conditions?

Yes.

A critical section must have an await expression or similar causing the executing coroutine to suspend and for other coroutines to execute the same critical section or access the same resource.

Do We Need To Protect Critical Sections In Light of the GIL?

Yes.

The GIL only pertains to threads, not coroutines.

All coroutines for one asyncio event loop execute within a single thread, and a thread executes within a single process.

The GIL does not prevent race conditions in critical sections in coroutines, threads, or processes.

Race conditions at all three levels are possible.

See:

• How to Fix a Race Condition With Timing in Python
• Race Condition With a Shared Variable in Python

What If We Don’t Release The Lock?

Not releasing the lock is a concurrency bug.

If the lock is not released by a coroutine that acquired it, then it can not be acquired again and the code protected by the critical section cannot be executed again.

This may result in a deadlock.

See:

• How to Identify a Deadlock in Python

What If We Ignore The Lock?

Ignoring the lock is a concurrency bug.

You may write code so that some coroutines may obey the lock and some may not. This will likely result in a race condition and defeat the purpose of having a lock in the first place.

The lock only protects the critical section (data, resources, etc.) if it is enforced in all accesses to the critical section.

Is the asyncio.Lock Reentrant?

No.

A reentrant lock is a lock that if held by one coroutine can be acquired again by the same coroutine.

At the time of writing, there does not appear to be support for a reentrant lock in asyncio.

Also, see:

• Reentrant locks in asyncio

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to use mutex locks to protect critical sections from coroutines in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Tjaard Krusch on Unsplash