🔄 What Are Generators in Python and How to Use Them (Properly)?

If you've come across the yield keyword in Python and wondered what it does, or you've heard that "generators save memory" but never quite understood how — this article is for you.

This is the fourth article in my Python fundamentals series, and we’ll dive deep into what generators are, how they work, when to use them, and what caveats to watch out for. All with practical examples and accessible explanations.

---

⚙️ What Is a Generator?

A generator is a special type of function that produces values on demand using the yield keyword instead of return. Instead of returning all results at once, it pauses execution and resumes from where it left off each time it's called.

This allows you to:

• Process large volumes of data without consuming much memory

• Build lazy evaluation pipelines

• Write cleaner, more efficient code for loops and data streams

---

🧪 Simple Example of a Generator:

def counter():
    yield 1
    yield 2
    yield 3

for number in counter():
    print(number)

Output:

1
2
3

Each yield call returns a value and pauses the function, which only continues when the next value is requested.

---

🧠 How Does It Work Internally?

Calling a regular function executes it all at once and returns a result. But calling a function with yield returns a generator object:

g = counter()
print(g)  # <generator object counter at 0x...>

This object implements the iterator protocol (__iter__ and __next__) and can be advanced using next():

print(next(g))  # 1
print(next(g))  # 2
print(next(g))  # 3
print(next(g))  # Error: StopIteration

---

🧵 Difference Between Lists and Generators

# List
[1, 2, 3]  # stores all elements in memory

# Generator
(x for x in range(3))  # calculates values on the fly

---

🚀 When Should You Use Generators?

Generators are ideal when:

• Working with large data sets

• Memory efficiency is critical

• Building data pipelines (e.g., read, transform, filter)

• Reading files line by line

• Consuming streams, APIs, or databases

---

📂 File Reading with Generators:

def read_lines(filepath):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

for line in read_lines("data.txt"):
    print(line)

You can process files efficiently without loading them entirely into memory.

---

🔁 Infinite Generators:

def infinite_counter():
    n = 0
    while True:
        yield n
        n += 1

for number in infinite_counter():
    if number > 5:
        break
    print(number)

Generators can be infinite, and your code remains efficient since it only generates one value at a time.

---

❗ Pitfalls and Limitations

• A generator cannot be restarted — it is consumed once.

• When exhausted, it raises StopIteration (which is normal).

• If you need to reuse the data, convert it to a list (list(gen)), but that loses memory benefits.

---

🧩 Generators with yield from

The yield from statement allows delegating to another generator — useful for composition and cleaner code:

def subgen():
    yield 1
    yield 2

def gen():
    yield 0
    yield from subgen()
    yield 3

for value in gen():
    print(value)

Output:

0
1
2
3

---

✅ Function vs Generator Comparison

# Regular function

def get_numbers():
    return [x for x in range(1000000)]  # memory intensive

# Generator

def gen_numbers():
    for x in range(1000000):
        yield x  # much lighter

---

📌 Conclusion

Generators are one of Python’s most powerful and underused tools. They let you write code that is lighter, cleaner, and more memory-efficient — especially when working with large datasets or data streams.

By understanding how they work under the hood and where to apply them, you can transform ordinary functions into performant data pipelines.

And remember: if you’re thinking "loop + lots of memory + on demand", generators are your best friend!

---

#Python #Generators #Yield #Performance #CleanCode #PythonFundamentals #DevLife #MemoryEfficiency