Lecture 4 - Pythonics III
ADVIST
by
Dmitrii Nechaev & Dr. Lothar Richter
18.11.22
Recap
2/68
Recap: Iterables and Iterators
3/68
Recap: Iterables and Iterators
an iterable is an object that is capable of returning its elements one by one
and implements an __iter__ method;
an iterator is an object that represents a stream of data and implements an
__iter__ method and a __next__ method;
when we use an iterable in a for...in loop, an iterator is automatically
created and is used to produce values until exhaustion;
an iterator SHOULD NOT change the corresponding iterable.
3/68
Recap: Magic Methods
4/68
Recap: Magic Methods
methods that start and end with a double underscore are called magic or
dunder methods
implementing such methods allows us to use standard Python functions
and operators with instances of our custom classes
__str__ (__int__, …) method allows us to tune casting an object to a
string (integer, …)
__repr__ method allows us to change the way an object is represented
(think REPL and Jupyter Notebook)
many more, such as __len__, __eq__, __add__, etc.
4/68
Recap: Callables
5/68
Recap: Callables
we can make instances of our classes callable like functions by
implementing the __call__ method
5/68
Recap: Decorators (Class-Based)
6/68
Recap: Decorators (Class-Based)
we can decorate functions, that is, modify their behavior;
@‑notation gives us a concise way to decorate functions;
we can apply several decorators to one function.
6/68
Recap: More on Classes
7/68
Recap: More on Classes
implement class methods with the classmethod decorator
implement static methods with the staticmethod decorator
implement getters and setters with the property and
{property_name}.setter decorators
7/68
Recap: Context Managers (Class-Based)
8/68
Recap: Context Managers (Class-Based)
a class with __enter__ and__exit__ methods can be used as a context
manager
8/68
Going Deeper
9/68
Today
generators
context managers (function‑based)
decorators (function‑based)
type hints
data classes
10/68
Generators
11/68
Generators
A generator is a function that returns a generator iterator:
1 def my_generator():
2 i = 0
3 while i <= 2:
4 yield i
5 i += 1
6
7 gen = my_generator()
8 for i in gen:
9 print(i)
0
1
2
12/68
Generator Expressions
A generator expression combines lazy evaluation of generators with the beauty and
simplicity of list comprehensions:
1 numbers = range(10000) 1 for i in range(3):
2 squares_gen = ( 2 print(next(squares_gen))
3 n ** 2 for n in numbers
0
4 )
1
5 type(squares_gen)
4
generator
13/68
enumerate
Python has many built‑in classes that are iterators. We will look at the enumerate
iterator in the following example and then attempt to re‑implement it ourselves:
1 for index, value in enumerate(['Alice', 'Bob', 'Charles']):
2 print(str(index) + ' ' + value)
0 Alice
1 Bob
2 Charles
14/68
enumerate
We don’t have to start counting from zero:
1 for index, value in enumerate(['Alice', 'Bob', 'Charles'], 5):
2 print(str(index) + ' ' + value)
5 Alice
6 Bob
7 Charles
15/68
enumerate
1 def my_enumerate(iterable, index=0):
2 for value in iterable:
3 yield index, value
4 index += 1
5
6 for index, value in my_enumerate(
7 ['Alice', 'Bob', 'Charles'], 5
8 ):
9 print(str(index) + ' ' + value)
5 Alice
6 Bob
7 Charles
16/68
zip
Sometimes we want to process two iterables of the same length pairwise (elements
with the same indices are related). We might want do it with range:
1 names = ['Alice', 'Bob', 'Charles']
2 ages = [23, 21, 71]
3 for i in range(len(names)):
4 print(names[i], 'is', ages[i])
Alice is 23
Bob is 21
Charles is 71
DO NOT DO THIS!
17/68
zip
Alternatively, we might want to do it with enumerate:
1 names = ['Alice', 'Bob', 'Charles']
2 ages = [23, 21, 71]
3 for i, name in enumerate(names):
4 print(name, 'is', ages[i])
Alice is 23
Bob is 21
Charles is 71
DO NOT DO THIS!
18/68
zip
The Pythonic way to solve this problem is via the zip function:
1 names = ['Alice', 'Bob', 'Charles']
2 ages = [23, 21, 71]
3 for name, age in zip(names, ages):
4 print(name, 'is', age)
Alice is 23
Bob is 21
Charles is 71
zip wraps two (or more) iterables with a lazy generator that yields tuples containing
pairs of next values from each iterable.
19/68
zip
What happens, if our iterables contain different numbers of items?
1 names = ['Alice', 'Bob', 'Charles']
2 ages = [23, 21, 71, 46]
3 for name, age in zip(names, ages):
4 print(name, 'is', age)
Alice is 23
Bob is 21
Charles is 71
20/68
zip
1 def my_zip(iterable_1, iterable_2): 1 names = ['Alice', 'Bob', 'Charles']
2 iter_1 = iter(iterable_1) 2 ages = [23, 21, 71, 46]
3 iter_2 = iter(iterable_2) 3 for name, age in my_zip(names, ages):
4 while True: 4 print(name, 'is', age)
5 try:
6 item_1 = next(iter_1) Alice is 23
7 item_2 = next(iter_2) Bob is 21
8 except StopIteration: Charles is 71
9 break
10 yield item_1, item_2
21/68
map&filter
We have already used lambdas for sorting:
1 sorted(['Bob', 'Charles', 'Alice'], key=lambda x: x[1])
['Charles', 'Alice', 'Bob']
We can similary use lambdas with the map & filter functions to respectively map or
filter iterables (these functions return an iterator):
1 list(map(lambda x: x ** 2, [1, 2, 3]))
[1, 4, 9]
1 list(filter(lambda x: x % 2 != 0, [1, 2, 3]))
[1, 3]
22/68
map&filter
I would advise to use comprehensions and generator expressions instead of map and
filter whenever possible, compare the following snippets:
1 list(map(lambda x: x ** 2, [1, 2, 3])) 1 [x ** 2 for x in [1, 2, 3]]
[1, 4, 9] [1, 4, 9]
1 list(filter(lambda x: x % 2 != 0, [1, 2, 3])) 1 [x for x in [1, 2, 3] if x % 2]
[1, 3] [1, 3]
23/68
Generators: Overview
a generator is a function that returns a generator iterator (use the yield
keyword)
a generator expression combines lazy evaluation of generators with the
beauty and simplicity of list comprehensions
24/68
Context Managers (Function-Based)
25/68
Class-Based Context Managers
We have already implemented a context manager to work with files:
1 class SafeFile:
2 def __init__(self, file_path):
3 self.file_path = file_path
4
5 def __enter__(self):
6 self.file_obj = open(self.file_path)
7 return self.file_obj
8
9 def __exit__(self, exc_type, exc_val, exc_tb):
10 if self.file_obj:
11 self.file_obj.close()
1 with SafeFile('gremlins.txt') as input_file:
2 for line in input_file:
3 print(line)
Hello, my little gremlins!
26/68
Function-Based Context Managers
1 from contextlib import contextmanager
2
3 @contextmanager
4 def safe_file(file_path):
5 file = open(file_path) # equivalent to __enter__
6 try: # equivalent to __enter__
7 yield file # equivalent to __enter__ return value
8 finally:
9 file.close() # equivalent to __exit__
1 with safe_file('gremlins.txt') as input_file:
2 for line in input_file:
3 print(line)
Hello, my little gremlins!
27/68
Context Managers (Function-Based): Overview
we can create a context manager using a function that yields and a
contextmanager decorator
28/68
Decorators (Function-Based)
29/68
Decorators (Function-Based)
We have seen that to implement a decorator we need a callable that accepts a callable
and returns another callable. We have implemented decorators with classes that define
the __call__ method. Can we create function‑based decorators? Can we create
functions that accept functions and return functions?
30/68
Nested Functions
We can define functions within functions:
1 def number_cruncher(a, b):
2 def secret_operation(c):
3 return c ** 2
4 return secret_operation(a + b)
1 number_cruncher(3, 4)
49
1 secret_operation(10)
NameError: name 'secret_operation' is not defined
31/68
Nested Functions
The function secret_operation (let’s call it an inner function) is only accessible
within the function number_cruncher (let’s call it an outer function).
32/68
Returning Functions from Functions
We can return functions from functions. In the following example we return the inner
function, assign the function object to a variable, and invoke it:
1 def number_cruncher():
2 def not_so_secret_operation(a, b):
3 return (a + b) ** 2
4 return not_so_secret_operation
1 number_operation = number_cruncher()
2 number_operation(3, 4)
49
33/68
Passing Functions to Functions
We can also pass function objects to functions:
1 def number_cruncher(a, b, passed_function):
2 def secret_operation(a, b):
3 return passed_function(a + b)
4 return secret_operation(a, b)
1 number_cruncher(3, 4, lambda x: x ** 2)
49
34/68
Accessing Variables of the Outer Function
As you can see in this example, the inner function “remembers” the value of the power
argument even after the outer function has completed its execution:
1 def power_factory(power): 1 square = power_factory(2)
2 def not_so_secred_operation(x): 2 square(5)
3 return x ** power
4 return not_so_secred_operation 25
1 cube = power_factory(3)
2 cube(2)
8
35/68
Closures
That, by the way, is a closure. A closure isn’t just a nested (inner) function. The closure
mechanism allows the inner function to keep access to its environment as it was when
the closure was created. In other words,
1. The outer function defines the inner function and returns it.
2. The nested (inner) function can access all the variables of the outer function
(that means the outer_param and the outer_var).
3. The inner function keeps access to the variable of the outer function even
after the outer function has finished its execution.
36/68
Closures: Example
1 def outer_function():
2 print('outer function started')
3
4 outer_var = 'Rostlab'
5 def inner_function():
6 print(outer_var)
7
8 print('outer function finished')
9 return inner_function
10
11 inner = outer_function()
12 inner()
outer function started
outer function finished
Rostlab
37/68
Closures: Use Cases
data hiding/protection (not bulletproof)
implementing function factories (creating functions at runtime)
decorators
38/68
Closures: Data Hiding/Protection
1 def counter_factory():
2 value = 0
3
4 def inner():
5 nonlocal value
6 value += 1
7 return value
8
9 return inner
10
11 counter_1 = counter_factory()
12 counter_2 = counter_factory()
13 for i in range(5):
14 counter_1()
15 for i in range(3):
16 counter_2()
17 print(counter_1())
18 print(counter_2())
6
4
39/68
Closures
We can use closures to implement function factories that create functions at runtime:
1 def power_factory(power):
2 def not_so_secred_operation(x):
3 return x ** power
4 return not_so_secred_operation
1 square = power_factory(2)
2 square(5)
25
1 cube = power_factory(3)
2 cube(2)
40/68
Closures
We can use closures to modify behavior of other functions (to decorate them), since the
value that is “remembered” can be a function object, too:
1 def my_function_that_i_pass(x):
2 return x ** 2
3
4 my_function_that_i_pass(9)
81
1 def decorating_function(func):
2 def wrapper(number):
3 print('Execution of the decorated function started')
4 result = func(number)
5 print('Execution of the decorated function completed')
6 return result
7 return wrapper
1 my_decorated_function = decorating_function(my_function_that_i_pass)
2 my_decorated_function(9)
Execution of the decorated function started
Execution of the decorated function completed
81
41/68
Anatomy of a Decorating Function
1 def decorating_function(function_that_i_want_to_decorate):
2 def wrapper(something something here):
3 # HERE WE DEFINE THE WRAPPER FUNCTION
4 return wrapper
5
6 my_decorated_function = decorating_function(my_function)
42/68
*args and **kwargs
We can use args and kwargs just like we did with a class‑based decorator:
1 def decorator(func):
2 def wrapper(*args, **kwargs):
3 print('Execution started')
4 result = func(*args, **kwargs)
5 print('Execution completed')
6 return result
7 return wrapper
8
9 @decorator
10 def my_power_of_a_sum(*args, power=2):
11 return sum(args) ** power
1 my_power_of_a_sum(2, 3, 5, power=3)
Execution started
Execution completed
1000
43/68
Decorators with Parameters
Things get even denser now, so please, pay attention!
1 def decorator_factory(argument):
2 def decorator(function):
3 def wrapper(*args, **kwargs):
4 action_1()
5 some_action_2_with_argument(argument)
6 result = function(*args, **kwargs)
7 action_3()
8 return result
9 return wrapper
10 return decorator
Applying the @decorator_factory decorator is equivalent to the following:
1 function = (decorator_factory(argument))(
2 function
3 )
44/68
Decorators with Parameters
This should look familiar:
1 def is_subset_of(decorator_arg):
2 def is_subset(func):
3 def wrapped_func(func_arg):
4 if not set(func_arg).issubset(set(decorator_arg)):
5 raise ValueError('Invalid elements')
6 return func(func_arg)
7 return wrapped_func
8 return is_subset
45/68
Decorators with Parameters
1 @is_subset_of({2, 4, 6})
2 def method_1(arg):
3 pass
4
5 @is_subset_of({1, 3, 5})
6 def method_2(arg):
7 pass
46/68
Decorators with Parameters
1 method_1([2, 4, 6])
1 method_1(['Andy', 'Bob', 'Charles'])
ValueError: Invalid elements
47/68
Decorators with Parameters
1 method_2([1, 3, 5])
1 method_2([3.50])
ValueError: Invalid elements
48/68
functools.wraps
Decorators are great, however, there is one problem:
1 def my_decorator(func):
2 def wrapped_func(*args, **kwargs):
3 return func(*args, **kwargs)
4 return wrapped_func
5
6 @my_decorator
7 def function_with_a_good_name(arg_1, arg_2):
8 ”””I am a good docstring”””
9 return arg_1 + arg_2
10
11 help(function_with_a_good_name)
Help on function wrapped_func in module __main__:
wrapped_func(*args, **kwargs)
49/68
functools.wraps
functools.wraps helps to solve the problem (otherwise we’d have to do it
manually):
1 from functools import wraps
2
3 def my_decorator(func):
4 @wraps(func)
5 def wrapped_func(*args, **kwargs):
6 return func(*args, **kwargs)
7 return wrapped_func
8
9 @my_decorator
10 def function_with_a_good_name(arg_1, arg_2):
11 ”””I am a good docstring”””
12 return arg_1 + arg_2
13
14 help(function_with_a_good_name)
Help on function function_with_a_good_name in module __main__:
function_with_a_good_name(arg_1, arg_2)
I am a good docstring
50/68
Function-Based Decorators: Overview
we can nest functions, i.o.w. we can have an “inner” function defined inside
an “outer” function
we can pass functions as arguments to other functions
we can return functions from other functions
a closure is a function that keeps access to its environment as it was when
the function was defined
closures help us to hide/protect the data
closures help us to generate functions at runtime
closures help us to create decorators
functools.wraps helps us to keep original names and docstrings
51/68
Type Hints
52/68
Type Hints
Python is a dynamic language. On one hand, that saves us from writing a lot of
boiler‑plate code like
1 Soup soup = new Soup()
53/68
Type Hints
On the other hand, information about types is useful:
1 def losses(self, X, Y):
2 ”””Computes the loss for each point.
3
4 The loss is computed as a difference between the actual y-coordinate
5 and the predicted y-coordinate (y_hat).
6
7 Parameters
8 ----------
9 X : list[float]
10 x-coordinates of the points.
11 Y : list[float]
12 y-coordinates of the points.
13
14 Returns
15 -------
16 list[float]
17 Losses computed for each point.
18 ”””
19 pass
We can specify type information via the type hinting mechanism that became available
starting with Python 3.5.
54/68
Type Hints - Variables
We can specify variable types using the following notation:
1 age: int
2 name: str
3 age = 'Bob'
Do not forget that hints are just hints! They do not stop us if we assign a string to a
variable that is supposed to refer to integer values.
55/68
Type Hints - Variables
We can specify variable types and assign values at the same time:
1 ages: list[int] = [23, 21, 71]
2 names: list[str] = ['Alice', 'Bob', 'Charles']
56/68
Type Hints - Functions
We can specify parameter types and return value types for the functions:
1 def sum_of_chars(string: str) -> int:
2 return sum(ord(x) for x in string)
1 sum_of_chars('Rostlab')
727
57/68
Type Hints - 3rd Party Tools
IDEs/editors like PyCharm and VS Code are capable of parsing type hints and
notifying a user when type hints have been violated. Tools like mypy and pyre allow us
to run static type checking on our code base from a terminal. Libraries like Pydantic
allow us to perform validation based on hints.
58/68
Type Hints - Validation with Pydantic
1 from pydantic import validate_arguments
2
3 @validate_arguments
4 def sum_of_chars(my_string: str) -> int:
5 return sum(ord(x) for x in my_string)
6
7 sum_of_chars(('a', 'b', 'c', 'd'))
ValidationError: 1 validation error for SumOfChars
my_string
str type expected (type=type_error.str)
59/68
Type Hints: Overview
type hints allow us to specify types for variables, function parameters, and
function return values
IDEs/editors and 3rd party tools can perform static type checking
libraries like pydantic allow us to perform validation based on type hints
60/68
Data Classes
61/68
Data Classes
Now, let’s take a look at the @dataclass decorator:
1 from dataclasses import dataclass
2
3 @dataclass
4 class DataChocoCow:
5 name: str
6 color: str
7 cocoa_content: int = 65
8
9 goldy = DataChocoCow('Goldy', 'golden')
10 andy = DataChocoCow(name='Andy', color='pink gold', cocoa_content=70)
11 print(andy)
DataChocoCow(name='Andy', color='pink gold', cocoa_content=70)
62/68
Data Classes
The @dataclass decorator can add the following methods:
__init__
__repr__
__eq__
__lt__
__le__
__gt__
__ge__
__init__, __repr__, and __eq__ are added by default, while the rest are not. We
can change that by passing parameters to the decorator.
63/68
Data Classes
1 @dataclass(init=True, repr=False, eq=True, order=True)
2 class DataChocoCow:
3 name: str
4 color: str
5 cocoa_content: int = 65
6
7 goldy = DataChocoCow('Goldy', 'golden')
8 andy = DataChocoCow(name='Andy', color='pink gold', cocoa_content=70)
9 print(andy)
10 andy < goldy
<__main__.DataChocoCow object at 0x7f2483666b90>
True
64/68
Data Classes
We can make instances of our class read‑only:
1 @dataclass(frozen=True)
2 class DataChocoCow:
3 name: str
4 color: str
5 cocoa_content: int = 65
6
7 goldy = DataChocoCow('Goldy', 'golden')
8 goldy
DataChocoCow(name='Goldy', color='golden', cocoa_content=65)
1 goldy.cocoa_content = 99
FrozenInstanceError: cannot assign to field 'cocoa_content'
65/68
Data Classes
We can define methods the usual way:
1 @dataclass
2 class DataChocoCow:
3 name: str
4 color: str
5 cocoa_content: int = 65
6
7 def talk(self):
8 return f'Moo! I\'m {self.name}.'
9
10 goldy = DataChocoCow('Goldy', 'golden')
11 goldy.talk()
”Moo! I'm Goldy.”
66/68
Data Classes: Overview
the @dataclass decorator simplifies class creation
it adds __init__, __repr__, and __eq__ methods by default
we can control method creation by invoking the @dataclass decorator
with parameters
67/68
Thank you!
QUESTIONS?
68/68
