Iterators and Generators Checkpoint

25 min · Intermediate

Fix the mistakes

The slips that cost marks in this module. Spot each one, then check the fix.

Real data work often means reading a stream, cleaning it, and taking a few results. This checkpoint builds one such pipeline and then tests the whole module.

Pick the simplest tool

A generator function is the shortest way to write most iterators. Use an iterator class only when you need extra methods or state that others can read. Use an iterable class with a generator __iter__ when the data must be walked many times.

Python

1class Catalog:2    def __init__(self, products):3        self.products = products4
5    def __iter__(self):6        for name, price in self.products:7            if price > 0:8                yield name

Example 1 · Reusable catalog class

Python

1class Catalog:2    def __init__(self, products):3        self.products = products4
5    def __iter__(self):6        for name, price in self.products:7            if price > 0:8                yield name9
10shop = Catalog([("pen", 10), ("gift", 0), ("book", 90)])11print(list(shop))12print(list(shop))

Output

['pen', 'book']
['pen', 'book']

What this code does

Both calls give the same result because each loop starts a new generator.
A plain generator object would give an empty list the second time.

Each for loop calls __iter__ and gets a fresh generator, so a Catalog can be read again and again.

Chain generators into a pipeline

Each stage takes an iterable and yields results. Values flow through one at a time, so no stage builds a full list.

Python

1def read(lines):2    for line in lines:3        yield line.strip()4
5def errors(lines):6    return (l for l in lines if l.startswith("ERROR"))7
8log = ["ok 1", "ERROR disk", "ERROR net "]9print(list(errors(read(log))))

Example 2 · A three-stage pipeline

Python

1from itertools import islice2
3def read(lines):4    for line in lines:5        yield line.strip()6
7def parse(lines):8    for line in lines:9        level, _, message = line.partition(" ")10        yield level, message11
12def only(level, rows):13    return (m for lv, m in rows if lv == level)14
15log = ["INFO start ", "ERROR disk full", "INFO retry", "ERROR net down"]16first = list(islice(only("ERROR", parse(read(log))), 1))17print(first)

Output

['disk full']

What this code does

Each stage is a generator, so only enough lines are read to find the first error.
The pipeline stops as soon as islice has its one item.

Because every stage is lazy, islice on the end of the pipeline stops all the stages early.

Watch for one-pass and order traps

A generator is used up after one pass. groupby needs sorted data. islice consumes the items it skips. Check each of these before trusting a result.

Python

from itertools import islice
it = iter(range(10))print(list(islice(it, 3)))print(next(it))

Example 3 · Iterator consumed by islice

Python

1from itertools import islice2
3it = iter(range(10))4print(list(islice(it, 3)))5print(next(it))6print(list(islice(it, 2)))

Output

[0, 1, 2]
3
[4, 5]

What this code does

islice takes from the same iterator, so the position moves on each time.
Use a new iter(range(10)) when you need to start again.

The second line prints 3, not 0, because islice already consumed the first three items from the same iterator.

Finish up

Mark complete & continue

Next: Closures and Captured Variables

1class Catalog:2 def __init__(self, products):3 self.products = products4 5 def __iter__(self):6 for name, price in self.products:7 if price > 0:8 yield name

1class Catalog:2 def __init__(self, products):3 self.products = products4 5 def __iter__(self):6 for name, price in self.products:7 if price > 0:8 yield name9 10shop = Catalog([("pen", 10), ("gift", 0), ("book", 90)])11print(list(shop))12print(list(shop))

1def read(lines):2 for line in lines:3 yield line.strip()4 5def errors(lines):6 return (l for l in lines if l.startswith("ERROR"))7 8log = ["ok 1", "ERROR disk", "ERROR net "]9print(list(errors(read(log))))

1from itertools import islice2 3def read(lines):4 for line in lines:5 yield line.strip()6 7def parse(lines):8 for line in lines:9 level, _, message = line.partition(" ")10 yield level, message11 12def only(level, rows):13 return (m for lv, m in rows if lv == level)14 15log = ["INFO start ", "ERROR disk full", "INFO retry", "ERROR net down"]16first = list(islice(only("ERROR", parse(read(log))), 1))17print(first)