Python, Theory

Python. Generator functions. Yield statement. Next(), iter(), send() methods

Generator functions. Statement yield. Methods next(), iter(), send()

Contents

Search other resources:

1. Constructions that reproduce results on demand. Peculiarities. The concept of a deferred operation

The Python programming language has tools to provide results on demand. These funds include:

  • functions-generators (generator functions);
  • expressions-generators.

Generator functions allow you to generate a sequence of values on demand. These functions differ from ordinary functions in that, having returned a value, they can continue their work from the place where they were stopped. As you know, ordinary functions, returning a value, stop their work.

  ⇑

2. The concept of the iteration protocol. The __next __() and iter() methods. Iteration protocol support by generator functions

You can use generator functions or other mechanisms (such as for loops) to traverse sequences or generator values. For this, the so-called iteration protocol is used. If the protocol is not supported, then the execution of the for loop is considered unsuccessful and this instruction performs the sequence indexing operation.

According to this protocol, all iterated objects define a __next __() method. This method is for returning the next item in the iteration. If the iterations complete, the method throws a StopIteration exception. For each iterated object (string, list, etc.), you can access the iterator of that object by calling the iter() built-in function.

For example, for a given string s, you can call the iter() function to get an iterator like this

# Functions-generators
# Function iter()

# 1. A string is given - this is an iterable object
s = "abcde axsadlk sdw"

# 2. Get an iterator of an object s
iterObj = iter(s)

# 3. Use the iterator to traverse the string s
count_a = 0
for c in iterObj:
    if c=='a':
        count_a = count_a+1

print("count of 'a' = ", count_a) # count of 'a' = 3

Just like the for loop, generator functions support the iteration protocol. If a generator function is called, it returns a generator object. This generator object contains an automatically generated __next__() method. The presence of the __next__() method allows you to continue iterating in the next steps.

  ⇑

3. Differences between regular functions and generator functions. Freezing the state in generator functions

For ordinary functions and generator functions, the following common and distinctive features can be distinguished:

  1. Both kinds of functions are created using the def statement.
  2. Ordinary functions return the value of the return statement. Generator functions return the value of the yield statement.
  3. Regular functions do not support the iteration protocol. Generator functions support this protocol.
  4. When a value is returned by the return statement, normal functions stop working. When a value is returned to yield statements, the generator functions automatically pause and resume their execution, retaining the information needed to generate the values.
  5. Generator functions provide value. Ordinary functions return a value.

The most important feature of generator functions is that they can pause and automatically save information about their state. This state determines the entire local scope of the function, with all local variables. When the generator function is resumed, the previously saved state becomes available to retrieve the next generated value.
Therefore, the process of pausing the execution of a function and storing information about the state of local variables or parameters is called freezing the state.

In terms of memory usage, generator functions avoid having to do all the work at once. This property is enhanced when the list of results is significant or when each new value takes a long time to compute. In this case, the use of generator functions gives a more efficient allocation of the time required to create the entire sequence of values.

  ⇑

4. Functions-generators. Features of the implementation. The yield statement General form

For a function to be able to generate a sequence of values and freeze its state, the function must support the iteration protocol. In this case, the function must return the result with a yield statement instead of a return statement.
The formation of values returned from a function can be performed by one of two well-known loop operators: for or while.

If the values in the body of a generator function are formed using a for loop, then the general form of such a function is as follows:

def FuncName(list_of_parameters):
    ...
    for value in iterObj:
        ...
        yield value

here

  • FuncName – the name of function;
  • list_of_parameters – a list of parameters that the function receives;
  • iterObj – an iterable object. This object can be created using standard tools such as the range() method;
  • value – an element from the set of values generated in the iterObj object.

If a generator function uses a while loop to generate values, then the general form of such a function is as follows

def FuncName(list_of_parameters):
    ...
    while condition:
        ...
        yield return_value

here

  • FuncName – the name of function;
  • list_of_parameters – a list of parameters that the function receives;
  • condition – condition for execution of the while loop;
  • return_value – return value from the function.

  ⇑

5. Examples of generator functions
5.1. A function that raises a number to the power of 3. Using a for loop

The example shows a generator function that returns a number to the power of 3. The result is returned using the yield construct. Such a function supports the iteration protocol.

# Functions-generators

# A function that raises a number to the power of 3.
# The result is returned by the yield statement
def Power3(value):
    for i in range(value):
        yield i*i*i

for i in Power3(10):
    print(i)

In the for loop, the Power3() function is called. On each iteration of the for loop, the function returns the next value. This means that the function automatically pauses and resumes its execution at the next iteration of the loop.

The result of the program

0
1
8
27
64
125
216
343
512
729

If instead of a yield statement in a function you use a return statement, this means that the function will be ordinary and will not support the iteration protocol. In this case, when starting the application, an error of type TypeError will be raised.

TypeError: 'int' object is not iterable

  ⇑

5.2. A function that generates a sequence of values using a while loop

Below is a simple function that generates a sequence of values

0, 0.2, 0.4, ..., value

The values change in steps of 0.2.

# A generator function that generates numbers 0, 0.2, 0.4, ..., value.
# The function uses a while loop to form values.
def GetFloatValues(value):
    t = 0
    while t<value:
        t = t + 0.2
        yield t

# Using a generator function in external code
for i in GetFloatValues(2):
    print(i)

Program result

0.2
0.4
0.6000000000000001
0.8
1.0
1.2
1.4
1.5999999999999999
1.7999999999999998
1.9999999999999998
2.1999999999999997

  ⇑

5.3. A function that generates random numbers in a specified range

The example demonstrates the use of the GetRandomValues() generator function, which generates random numbers using the random.randint() method from the random module. The function receives parameters:

  • n – the number of random numbers to be generated;
  • a, b – respectively the upper and lower boundaries of the range of generated numbers.
# Generator functions

# Include the random module
import random

# 1. Declare a Generator Function that generates n random numbers
# using a while loop in the specified range [a; b]
def GetRandomValues(a, b, n):
    for t in range(n):
        yield random.randint(a, b)

# 2. Using a generator function in external code
# 2.1. Input data
n = int(input("n = "))
a = int(input("a = "))
b = int(input("b = "))

# 2.2. Form a list of n random numbers.
L = []
for i in GetRandomValues(a, b, n):
    L = L + [i]

# 2.3. Display the list
print("L = ", L)

Test example

n = 8
a = 20
b = 40
L = [34, 22, 25, 33, 22, 27, 29, 34]

  ⇑

6. Extended protocol of generator functions. Passing a value to a generator function. The send() method. Example

In addition to the next() method, the send() method has been added to the generator function protocol. This method provides a so-called extended protocol and is used for communication between the generator function and the calling program. Extended protocol support has been introduced since version 2.5.

The send() method can be used to pass the value of the yield expression. In this case, the yield expression is placed on the right side of the assignment statement. The general view of the yield expression can be as follows

value = yield return_value

here

  • return_value – the value returned by the yield statement;
  • value is the new value that is passed to the yield statement. In fact, the yield statement is an expression that returns an item passed to the send() method.

In the calling code, before passing the value of the function to the generator, you need to perform the following three steps:

1. Set a name that will be associated with the generator function, for example

FN = FuncName(list_of_parameters)

here

  • FuncName – the name of the generator function;
  • list_of_parameters – a list of parameters that are passed to the generator function;
  • FN – the name that is associated with the generator function.

2. Start the generator by calling the next() method

next(FN)

3. Pass a value to the generator by calling the send() method

FN.send(value)

here

  • value – a value to be passed to the generator function. This value in the generator function can be processed in some way.

  ⇑

7. An example demonstrating the use of the send() method to control the sequence of values received from a generator function

Task. Implement a generator function that supplies n random numbers. The value of each number ranges from 1 to 10, inclusive. If the number is located at a position multiple of 3, then the generator function must return 0. Consider that the positions of numbers are numbered starting from 1.

Solution.

# Functions-generators. The send() method

import random

# A generator function that uses the extended protocol.
# This function supplies random numbers in the range [1, 10].
# The number of numbers is n.
def FnGen(n):
    for t in range(n):
        # Get a random value
        value = yield random.randint(1, 10)

        # If the value -1 is passed to this function, then return 0
        if value==-1:
            yield 20

# 1. Specify the number of random numbers
n = int(input("n = "))

# 2. Start generator
V = FnGen(n+1)
next(V) # Get some number - this is the start of the generator

# 3. Get n random numbers excluding numbers,
# which are in multiples of 3: (3, 6, 9, ...).
# In these positions, enter the number 0.

# 4. Initialize the resulting list of numbers
L = []

# 5. The loop of forming the list
i = 1
while i<=n:
    if i%3==0:
        item = V.send(-1) # if the position is a multiple of 3, then pass -1
    else:
        item = next(V)
    L = L + [item]
    i=i+1

# 6. Display the result
print(L)

Test example

n = 20
[10, 2, 20, 8, 6, 20, 4, 9, 20, 5, 6, 20, 4, 9, 20, 4, 1, 20, 8, 7]

  ⇑

Related topics

  ⇑