Day 33:Using list() instead of generator for large data

 

🐍 Python Mistakes Everyone Makes ❌

Day 33: Using list() Instead of a Generator for Large Data

When working with large datasets, how you iterate matters a lot. One small choice can cost you memory, time, and even crash your program.

---

❌ The Mistake

Creating a full list when you only need to loop once.

numbers = list(range(10_000_000))

for n in numbers: 

   process(n)

This builds all 10 million numbers in memory before doing any work.

---

❌ Why This Fails

• Uses a lot of memory

• Slower startup time

• Completely unnecessary if data is used once

• Can crash programs with very large datasets

---

✅ The Correct Way

Iterate lazily using a generator (range is already one).

def process(n):
    # simulate some work
    if n % 1_000_000 == 0:
         print(f"Processing {n}")

for n in range(10_000_000): 

    process(n)

This processes values one at a time, without storing them all.

---

🧠 Simple Rule to Remember

🐍 If data is large and used once → use a generator
🐍 Use lists only when you need all values at once

---

🔑 Key Takeaways

• Generators are memory-efficient

• range() is already lazy in Python 3

• Avoid list() unless you truly need the list

• Small choices scale into big performance wins

---

Efficient Python isn’t about fancy tricks it's about making the right default choices 🚀