Python Generators Explained: Efficient Iteration with Yield

Ever dealt with a huge dataset in Python and run out of memory? Python generators might be the solution. Generators use the yield keyword to produce values one at a time, so you can handle large (even infinite) sequences efficiently. In this beginner-friendly tutorial, we’ll demystify generators – what they are, how they work, and when to use them – so you can write more efficient Python code.

What are Python Generators?

A generator in Python is a special type of function that returns an iterator object. Unlike regular functions that return value(s) and then terminate, generators can pause their execution and resume from where they left off, maintaining their state between calls.

Some key characteristics:

• Use the yield keyword instead of return
• Maintain the state between function calls
• They are very memory-efficient
• They are iterable like other Python iterators(with the use of the “next” keyword, which we will see in the following sections)

Tip💡: Learn about more Python advanced topics like this.

Generator Functions vs Regular Functions In Python

As a simple illustration, let’s compare a regular function with a generator function:

# Regular function
def get_squares(n):
    squares = []
    for i in range(n):
        squares.append(i ** 2)
    return squares

# Generator function
def squares_generator(n):
    for i in range(n):
        yield i ** 2

# Usage comparison
regular_result = get_squares(5)  # Creates and stores all values at once
generator_result = squares_generator(5)  # Creates values on-demand
regular_result = get_squares(5)  # Creates and stores all values at once
generator_result = squares_generator(5)  # Creates values on-demand
print(regular_result) # Output: [0, 1, 4, 9, 16]
print(generator_result) # Output: <generator object squares_generator at 0x1078a7780>Code language: PHP (php)

Why Use Python Generators?

Python Generators are particularly useful due to their following characteristics:

• Save Memory: Since they yield data one piece at a time, they don’t store all data in memory, making them efficient for large datasets.
• Lazy Evaluation: Generators evaluate values only when needed, which makes them more efficient for handling data that may not be required simultaneously.
• Ease of Use: They help make code cleaner and more readable when managing complex data flows.

Creating Your First Python Generator

Here’s a simple example of a generator that yields numbers from 1 to n:

def number_generator(n):
    num = 1
    while num <= n:
        yield num
        num += 1

# Using the generator
gen = number_generator(3)
print(next(gen))  # Output: 1
print(next(gen))  # Output: 2
print(next(gen))  # Output: 3
# Let's try one more time for fun
print(next(gen))  # Output: Throws StopIteration errorCode language: PHP (php)

Caution ⚠️: As you might notice, if we try to call the generator function more than the intended limit, it will throw a “StopIteration” error. So, make sure to catch such exceptions if there’s any unexpected Python method calls for these methods.

generator vs iterator

In Python, iterators and generators both let you traverse sequences of values one at a time, but they differ in subtle ways:

• Boilerplate: Generators require far less code—just a function with yield.
• State management: Iterators need manual tracking of state; generators automatically “pause” with their local variables intact.
• Memory usage: Both produce values lazily, but generators are often preferred for concise, on-the-fly iteration without extra object overhead.

generator vs list comprehension

• Evaluation: Generators use Lazy evaluation — computes each item on-the-fly as you iterate where list comprehension builds the entire list in memory.
• Memory Use: Due to above, Generators uses way less memory(constant) than list comprehension.
• Reusability: Generators are single use and needs re-creating once exhausted.

Understanding The yield Statement

The yield statement is what makes python generators special. When a generator function is called, it returns a generator object without executing the function’s body. The function only executes when next() it is called on the generator object.

def demonstrate_yield():
    print("First point")
    yield 1
    print("Second point")
    yield 2
    print("Third point")
    yield 3

gen = demonstrate_yield()
print(next(gen))
"""
Output:
First point
1
"""
print(next(gen))
"""
Output:
Second point
2
"""
print(next(gen))
"""
Output:
Third point
3
"""Code language: PHP (php)

What does the yield keyword do?

yield keyword pauses a function’s execution and returns a value, allowing the function to resume where it left off. Each time yield is called, the generator produces the next value without losing its prior state.

Other Relevant Generator Methods

• send(): Send values back to generator
• throw(): Raise exceptions inside generator
• close(): Stop generator execution

Practical Example: Random Number Generator

Here’s a practical example of using a generator to create an infinite sequence of random numbers:

import random

def random_number_generator(minimum, maximum, seed=None):
    """
    Generate an infinite sequence of random numbers between minimum and maximum.
    
    Args:
        minimum (int): Lower bound for random numbers
        maximum (int): Upper bound for random numbers
        seed (int, optional): Seed for random number generation
    
    Yields:
        int: Random number between minimum and maximum
    """
    if seed is not None:
        random.seed(seed)
    
    while True:
        yield random.randint(minimum, maximum)

# Usage example
random_gen = random_number_generator(1, 100)
print("Generated random numbers:")
for _ in range(5):
    print(next(random_gen))Code language: PHP (php)

This generator provides several benefits:

1. It generates numbers only when needed
2. It can run indefinitely without consuming extra memory
3. It’s reusable and configurable

Python generator expression:

A Python generator expression is a compact, lazy-evaluated way to create a generator in a single line—much like a list comprehension, but using parentheses instead of square brackets. Instead of building the entire list in memory, a generator expression yields one item at a time as you iterate over it.

# List comprehension (eager)
squares_list = [x*x for x in range(5)]

# Generator expression (lazy)
squares_gen = (x*x for x in range(5))

print(type(squares_gen))
# → <class 'generator'>

for val in squares_gen:
    print(val)Code language: PHP (php)

Common Use Cases For Python Generators

Some common scenarios/use cases suitable for Python generators include, but are not limited to:

1. Processing large files, reading chunks of data at a time.

def read_file_lines(filename):
    with open(filename, 'r') as file:
        for line in file:
            yield line.strip()

# Usage
for line in read_file_lines('large_file.txt'):
    process_line(line)Code language: PHP (php)

2. Processing a stream of data/data pipeline and/and other infinite sequences

def fibonacci_generator():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Get first 10 Fibonacci numbers
fib = fibonacci_generator()
for _ in range(10):
    print(next(fib))Code language: PHP (php)

3. You need memory efficiency while processing extensive data

# Memory inefficient
def get_all_numbers(n):
    return [i for i in range(n)]

# Memory efficient
def get_numbers_generator(n):
    for i in range(n):
        yield iCode language: PHP (php)

Conclusion

Python generators are a powerful tool for creating memory-efficient iterators. They’re particularly useful when working with large datasets or infinite sequences. By using the yield keyword and understanding generator expressions, you can write more efficient and cleaner code.

Remember that generators are lazy and compute values only when needed, making them perfect for scenarios where memory efficiency is crucial. Start incorporating generators into your Python projects, and you’ll soon appreciate their elegance and utility.

Also, refer to the official documentation to learn more.

Is there anything I missed, or do you have any specific questions about Python generators? Feel free to comment below. Happy Python programming 🐍!