In-depth understanding of Python generators (Generator)

We can create a list simply and directly through list generation, but due to memory constraints, the list capacity is definitely limited. Moreover, creating a list containing 1 million elements not only takes up a lot of storage space, but if we only need to access the first few elements, the space occupied by most of the subsequent elements is wasted.

So, if the list elements can be calculated according to a certain algorithm, can we continuously calculate subsequent elements during the loop? This eliminates the need to create a complete list, saving a lot of space. In Python, this mechanism of looping and calculating at the same time is called a generator.

To create a generator, there are many ways. The first method is very simple. Just change the [] of a list generation expression to () to create a generator:

>>> mylist = [ x for x in range(1, 10)]
>>> mylist
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> gen = (x for x in range(1,10))
>>> gen
<generator object <genexpr> at 0x7f1d7fd0f5a0>

The only difference between creating mylist and gen is the outermost [] and (). Mylist is A list, and gen is a generator.

We can directly print out each element of the list, but how do we print out each element of the generator?

If you want to print them out one by one, you can use the next() method of the generator:

>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
3
...
>>> gen.next()
9
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

As we said, the generator saves the algorithm. Each time next() is called, the value of the next element is calculated until it is calculated. When the last element is the last element and there are no more elements, a StopIteration error is thrown.

In fact, we can use for loop instead of next() method, which is more in line with efficient programming ideas:

>>> gen = ( x for x in range(1, 10))
>>> for num in gen:
...     print num
... 
1
2
3
4
5
6
7
8
9

generator is very powerful. If the calculation algorithm is relatively complex and cannot be implemented using a for loop similar to list generation, you can also use a function to implement it.

For example, in the famous Fibonacci sequence, except for the first and second numbers, any number can be obtained by adding the first two numbers:

1, 1, 2, 3, 5 , 8, 13, 21, 34, ...

The Fibonacci sequence cannot be written using list generation, but it is easy to print it out using a function:

def fib(max):
    n = 0 
    a, b = 0, 1
    while n < max:
        print b
        a, b = b, a + b
        n = n + 1

The above function can output Fibonacci The first N numbers of the Fibonacci sequence:

>>> fib(6)

If you look closely, you can see that the fib function actually defines the calculation rules of the Fibonacci sequence. You can start from the first element and calculate any subsequent elements. This This logic is actually very similar to generator.

In other words, the above function is only one step away from the generator. To turn the fib function into a generator, just change print b to yield b:

def fib(max):
    n = 0 
    a, b = 0, 1
    while n < max:
        yield b
        a, b = b, a + b
        n = n + 1

This is another way to define a generator. If a function definition contains the yield keyword, then the function is no longer an ordinary function, but a generator:

>>> fib(6)

Here, the most difficult thing to understand is that the execution processes of generators and functions are different. Functions are executed sequentially and return when encountering a return statement or the last line of function statements. The function that becomes a generator is executed every time next() is called, returns when encountering a yield statement, and continues execution from the yield statement returned last time when executed again.

As a simple example, define a generator that returns the numbers 1, 3, and 5 in sequence:

>>> def odd():
...     print 'step 1'
...     yield 1
...     print 'step 2'
...     yield 3
...     print 'step 3'
...     yield 5
...
>>> o = odd()
>>> o.next()
step 1
1
>>> o.next()
step 2
3
>>> o.next()
step 3
5
>>> o.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

You can see that odd is not an ordinary function, but a generator. During the execution process, it will interrupt when it encounters yield. Next time And continue to execute. After executing yield three times, there is no more yield to execute, so an error is reported when next() is called for the fourth time.

Back to the fib example, if we keep calling yield during the loop, it will continue to be interrupted. Of course, you need to set a condition for the loop to exit the loop, otherwise an infinite number will be listed.

Similarly, after changing the function to generator, we basically never use next() to call it, but directly use the for loop to iterate:

>>> for n in fib(6):
...     print n
...

generator is a very powerful tool. In Python, you can Simply change the list generation into a generator, or you can also implement complex logic generators through functions.

To understand the working principle of the generator, it continuously calculates the next element during the for loop and ends the for loop under appropriate conditions. For a generator changed from a function, when the return statement is encountered or the last line of the function body is executed, it is the instruction to end the generator, and the for loop ends accordingly.