Abstract image of a woman sitting on books and studying on a laptop

Linear algebra havard university

V
Valentine Orovwegodo

This document provides examples of how linear algebra is useful across many domains: 1) Linear algebra can be used to represent and analyze networks and graphs through adjacency matrices. 2) Differential equations describing complex systems like bridges and molecules can be understood through matrix representations and eigenvalues. 3) Quantum computing uses linear algebra operations like matrix multiplication to represent computations on quantum bits. 4) Many other areas like coding/encryption, data compression, solving systems of equations, computer graphics, statistics, games, and neural networks rely on concepts from linear algebra.

1 of 70
Downloaded 16 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
USE OF LINEAR ALGEBRA I Math 21b, O. Knill This is not a list of topics covered in the course. It is rather a lose selection of subjects, in which linear algebra is useful or relevant. The aim is to convince you that it is worth learning this subject. Most likely, some of this handout does not make much sense yet to you. Look at this page at the end of the course again, some of the content will become more interesting then. 1 2 3 4 GRAPHS, NETWORKS. Linear al- gebra can be used to understand networks which is a collection of nodes connected by edges. Networks are also called graphs. The adja- cency matrix of a graph is defined by Aij = 1 if there is an edge from node i to node j in the graph. Oth- erwise the entry is zero. A prob- lem using such matrices appeared on a blackboard at MIT in the movie ”Good will hunting”. Application: An ij is the number of n- step walks in the graph which start at the vertex i and end at the vertex j. CHEMISTRY, MECHANICS Complicated objects like a bridge (the picture shows Storrow Drive connection bridge which is part of the ”big dig”), or a molecule (i.e. a protein) can be modeled by finitely many parts (bridge elements or atoms) coupled with attractive and repelling forces. The vibrations of the system are described by a differential equation ˙x = Ax, where x(t) is a vector which depends on time. Differential equations are an important part of this course. The solution x(t) = exp(At) of the differential equation ˙x = Ax can be understood and computed by find- ing the eigenvalues of the matrix A. Knowing these frequencies is impor- tant for the design of a mechani- cal object because the engineer can damp dangerous frequencies. In chemistry or medicine, the knowl- edge of the vibration resonances al- lows to determine the shape of a molecule. QUANTUM COMPUTING A quantum computer is a quantum mechanical system which is used to perform computations. The state x of a machine is no more a sequence of bits like in a classical computer but a sequence of qubits, where each qubit is a vector. The memory of the computer can be represented as a vector. Each computation step is a multiplication x → Ax with a suitable matrix A. Theoretically, quantum computa- tions could speed up conventional computations significantly. They could be used for example for cryp- tological purposes. Freely available quantum computer language (QCL) interpreters can simulate quantum computers with an arbitrary num- ber of qubits. CHAOS THEORY. Dynamical systems theory deals with the iteration of maps or the analysis of solutions of differential equations. At each time t, one has a map T(t) on the vector space. The linear approximation DT(t) is called Jacobean is a matrix. If the largest eigenvalue of DT(t) grows exponentially in t, then the system shows ”sensitive dependence on initial conditions” which is also called ”chaos”. Examples of dynamical systems are our solar system or the stars in a galaxy, electrons in a plasma or par- ticles in a fluid. The theoretical study is intrinsically linked to linear algebra because stability properties often depends on linear approxima- tions.
USE OF LINEAR ALGEBRA II Math 21b, O. Knill CODING, ERROR CORRECTION Coding theory is used for encryp- tion or error correction. In the first case, the data x are maped by a map T into code y=Tx. For a good code, T is a ”trapdoor function” in the sense that it is hard to get x back when y is known. In the sec- ond case, a code is a linear subspace X of a vector space and T is a map describing the transmission with er- rors. The projection of Tx onto the subspace X corrects the error. Linear algebra enters in different ways, often directly because the ob- jects are vectors but also indirectly like for example in algorithms which aim at cracking encryption schemes. DATA COMPRESSION Image- (i.e. JPG), video- (MPG4) and sound compression algorithms (i.e. MP3) make use of linear transformations like the Fourier transform. In all cases, the com- pression makes use of the fact that in the Fourier space, information can be cut away without disturbing the main information. Typically, a picture, a sound or a movie is cut into smaller junks. These parts are represented by vec- tors. If U denotes the Fourier trans- form and P is a cutoff function, then y = PUx is transferred or stored on a CD or DVD. The receiver obtains back UT y which is close to x in the sense that the human eye or ear does not notice a big difference. SOLVING SYSTEMS OR EQUA- TIONS When extremizing a func- tion f on data which satisfy a con- straint g(x) = 0, the method of Lagrange multipliers asks to solve a nonlinear system of equations f(x) = λ g(x), g(x) = 0 for the (n + 1) unknowns (x, l), where f is the gradient of f. Solving systems of nonlinear equa- tions can be tricky. Already for sys- tems of polynomial equations, one has to work with linear spaces of polynomials. Even if the Lagrange system is a linear system, the task of solving it can be done more ef- ficiently using a solid foundation of linear algebra. GAMES Moving around in a world described in a computer game re- quires rotations and translations to be implemented efficiently. Hard- ware acceleration can help to handle this. Rotations are represented by or- thogonal matrices. For example, if an object located at (0, 0, 0), turning around the y-axes by an angle φ, every point in the ob- ject gets transformed by the matrix  cos(φ) 0 sin(φ) 0 1 0 − sin(φ) 0 cos(φ)  
USE OF LINEAR ALGEBRA (III) Math 21b, Oliver Knill STATISTICS When analyzing data statistically, one often is interested in the correlation matrix Aij = E[YiYj] of a random vector X = (X1, . . . , Xn) with Yi = Xi − E[Xi]. This matrix is derived from the data and determines often the random variables when the type of the dis- tribution is fixed. For example, if the random variables have a Gaussian (=Bell shaped) dis- tribution, the correlation matrix to- gether with the expectation E[Xi] determines the random variables. GAME THEORY Abstract Games are often represented by pay-off ma- trices. These matrices tell the out- come when the decisions of each player are known. A famous example is the prisoner dilemma. Each player has the choice to corporate or to cheat.. The game is described by a 2x2 matrix like for example 3 0 5 1 . If a player cooperates and his partner also, both get 3 points. If his partner cheats and he cooperates, he gets 5 points. If both cheat, both get 1 point. More generally, in a game with two players where each player can chose from n strategies, the pay- off matrix is a n times n matrix A. A Nash equilibrium is a vector p ∈ S = { i pi = 1, pi ≥ 0 } for which qAp ≤ pAp for all q ∈ S. NEURAL NETWORK In part of neural network theory, for exam- ple Hopfield networks, the state space is a 2n-dimensional vector space. Every state of the network is given by a vector x, where each component takes the values −1 or 1. If W is a symmetric nxn matrix, one can define a ”learning map” T : x → signWx, where the sign is taken component wise. The en- ergy of the state is the dot prod- uct −(x, Wx)/2. One is interested in fixed points of the map. For example, if Wij = xiyj, then x is a fixed point of the learning map.
USE OF LINEAR ALGEBRA (IV) Math 21b, Oliver Knill MARKOV. Suppose we have three bags with 10 balls each. Every time we throw a dice and a 5 shows up, we move a ball from bag 1 to bag 2, if the dice shows 1 or 2, we move a ball from bag 2 to bag 3, if 3 or 4 turns up, we move a ball from bag 3 to bag 1 and a ball from bag 3 to bag 2. What distribution of balls will we see in average? The problem defines a Markov chain described by a matrix  5/6 1/6 0 0 2/3 1/3 1/6 1/6 2/3  . From this matrix, the equilibrium distribution can be read off as an eigenvector of a matrix. SPLINES In computer aided de- sign (CAD) used for example to construct cars, one wants to inter- polate points with smooth curves. One example: assume you want to find a curve connecting two points P and Q and the direction is given at each point. Find a cubic function f(x, y) = ax3 + bx2 y + cxy2 + dy3 which interpolates. If we write down the conditions, we will have to solve a system of 4 equa- tions for four unknowns. Graphic artists (i.e. at the company ”Pixar”) need to have linear algebra skills also at many other topics in computer graphics. a b c SYMBOLIC DYNAMICS Assume that a system can be in three dif- ferent states a, b, c and that tran- sitions a → b, b → a, b → c, c → c, c → a are allowed. A possible evolution of the system is then a, b, a, b, a, c, c, c, a, b, c, a... One calls this a description of the system with symbolic dynamics. This language is used in information the- ory or in dynamical systems theory. The dynamics of the system is coded with a symbolic dynamical system. The transition matrix is  0 1 0 1 0 1 1 0 1  . Information theoretical quantities like the ”entropy” can be read off from this matrix. o p q r a b c d INVERSE PROBLEMS The recon- struction of a density function from projections along lines reduces to the solution of the Radon trans- form. Studied first in 1917, it is to- day a basic tool in applications like medical diagnosis, tokamak moni- toring, in plasma physics or for as- trophysical applications. The re- construction is also called tomogra- phy. Mathematical tools developed for the solution of this problem lead to the construction of sophisticated scanners. It is important that the inversion h = R(f) → f is fast, ac- curate, robust and requires as few datas as possible. Toy problem: We have 4 containers with density a, b, c, d arranged in a square. We are able and measure the light absorption by by sending light through it. Like this, we get o = a + b,p = c + d,q = a + c and r = b + d. The problem is to re- cover a, b, c, d. The system of equa- tions is equivalent to Ax = b, with x = (a, b, c, d) and b = (o, p, q, r) and A =     1 1 0 0 0 0 1 1 1 0 1 0 0 1 0 1    .
ICE: A BLACKBOARD PROBLEM Math 21b, O. Knill In the movie ”Good Will Hunting”, the main character Will Hunting (Matt Damon) solves a black- board challenge problem, which is given as a challenge to a linear algebra class. THE ”WILL HUNTING” PROBLEM. G is the graph 1 2 3 4 Find. 1) the adjacency matrix A. 2) the matrix giving the number of 3 step walks. 3) the generating function for walks from point i → j. 4) the generating function for walks from points 1 → 3. This problem belongs to linear algebra and calculus evenso the problem origins from graph the- ory or combinatorics. For a calculus student who has never seen the connection between graph theory, calculus and linear algebra, the assignment is actually hard – probably too hard - as the movie correctly indicates. The problem was posed in the last part of a linear algebra course. An explanation of some terms: THE ADJACENCY MATRIX. The structure of the graph can be encoded with a 4 × 4 array which encodes how many paths of length 1, one can take in the graph from one node to an other: no one no one one none two one no two no no one one no no which can more conve- niently be written as an array of numbers called a matrix: L =      0 1 0 1 1 0 2 1 0 2 0 0 1 1 0 0      Problem 2 asks to find the matrix which encodes all possible paths of length 3. GENERATING FUNCTION. To any graph one can assign for every pair of nodes i, j a series f(z) = ∞ n=0 a(ij) n zn , where a(ij) n is the number of possible walks from node i to node j with n steps. Problem 3) asks for an explicit expression of f(z) and problem 4) asks for an explicit expression in the special case i = 1, j = 3. Linear algebra has many relations to other fields in mathematics. It is not true that linear algebra is just about solving systems of linear equations.
SOLUTION TO 2). [L2 ]ij is by definition of the matrix product the sum Li1L1j + Li2L2j + ... + Li4Lhj. Each term Li1L1j is 1 if and only if there is a path of length 2 going from i to j passing through k. Therefore [L2 ]ij is the number of paths of length 2 going from node i to j. Similarly, [Ln ]ij is the number of paths of length n going from i to j. The answer is L3 =      2 7 2 3 7 2 12 7 2 12 0 2 3 7 2 2      SOLUTION TO 3). The geometric series formula n xn = (1 − x)−1 holds also for matrices: f(z) = ∞ n=0 [Ln ]ijzn = [ ∞ n=0 Ln zn )]ij = (1 − Lz)−1 Cramers formula for the inverse of a matrix which involves the determinant A−1 = det(Aij)/det(A) This leads to an explicit formula det(1 − zLij)/det(1 − zL) which can also be written as det(Lij − z)/det(L − z) . SOLUTION TO 4). This is the special case, when i = 1 and j = 3. det(L − z) = det(    1 −z 1 0 2 0 1 1 −z   ) = −2 − z + 3z2 − z3 . det(L − z) =      −z 1 0 1 1 −z 2 1 0 2 −z 0 1 1 0 −z      = 4 − 2z − 7z2 + z4 The final result is (z3 + 3z2 − z − 2)/(z4 − 7z2 − 2z + 4) .
LINEAR EQUATIONS Math 21b, O. Knill SYSTEM OF LINEAR EQUATIONS. A collection of linear equations is called a system of linear equations. An example is 3x − y − z = 0 −x + 2y − z = 0 −x − y + 3z = 9 . It consists of three equations for three unknowns x, y, z. Linear means that no nonlinear terms like x2 , x3 , xy, yz3 , sin(x) etc. appear. A formal definition of linearity will be given later. LINEAR EQUATION. More precicely, ax + by = c is the general linear equation in two variables. ax + by + cz = d is the general linear equation in three variables. The general linear equation in n variables is a1x1 + a2x2 + ... + anxn = 0 Finitely many such equations form a system of linear equations. SOLVING BY ELIMINATION. Eliminate variables. In the example the first equation gives z = 3x − y. Putting this into the second and third equation gives −x + 2y − (3x − y) = 0 −x − y + 3(3x − y) = 9 or −4x + 3y = 0 8x − 4y = 9 . The first equation gives y = 4/3x and plugging this into the other equation gives 8x − 16/3x = 9 or 8x = 27 which means x = 27/8. The other values y = 9/2, z = 45/8 can now be obtained. SOLVE BY SUITABLE SUBTRACTION. Addition of equations. If we subtract the third equation from the second, we get 3y − 4z = −9 and add three times the second equation to the first, we get 5y − 4z = 0. Subtracting this equation to the previous one gives −2y = −9 or y = 2/9. SOLVE BY COMPUTER. Use the computer. In Mathematica: Solve[{3x − y − z == 0, −x + 2y − z == 0, −x − y + 3z == 9}, {x, y, z}] . But what did Mathematica do to solve this equation? We will look in this course at some efficient algorithms. GEOMETRIC SOLUTION. Each of the three equations represents a plane in three-dimensional space. Points on the first plane satisfy the first equation. The second plane is the solution set to the second equation. To satisfy the first two equations means to be on the intersection of these two planes which is here a line. To satisfy all three equations, we have to intersect the line with the plane representing the third equation which is a point. LINES, PLANES, HYPERPLANES. The set of points in the plane satisfying ax + by = c form a line. The set of points in space satisfying ax + by + cd = d form a plane. The set of points in n-dimensional space satisfying a1x1 + ... + anxn = a0 define a set called a hyperplane.
RIDDLES: ”15 kids have bicycles or tricycles. Together they count 37 wheels. How many have bicycles?” Solution. With x bicycles and y tricycles, then x + y = 15, 2x + 3y = 37. The solution is x = 8, y = 7. ”Tom, the brother of Carry has twice as many sisters as brothers while Carry has equal number of systers and brothers. How many kids is there in total in this family?” Solution If there are x brothers and y systers, then Tom has y sisters and x−1 brothers while Carry has x brothers and y − 1 sisters. We know y = 2(x − 1), x = y − 1 so that x + 1 = 2(x − 1) and so x = 3, y = 4. INTERPOLATION. Find the equation of the parabola which passes through the points P = (0, −1), Q = (1, 4) and R = (2, 13). Solution. Assume the parabola is ax2 +bx+c = 0. So, c = −1, a+b+c = 4, 4a+2b+c = 13. Elimination of c gives a + b = 5, 4a + 2b = 14 so that 2b = 6 and b = 3, a = 2. The parabola has the equation 2x2 + 3x − 1 = 0 TOMOGRAPHY Here is a toy example of a problem one has to solve for magnetic resonance imaging (MRI), which makes use of the absorption and emission of energy in the radio frequency range of the electromag- netic spectrum. Assume we have 4 hydrogen atoms whose nuclei are excited with intensity a, b, c, d. We measure the spin echo intensity in 4 different directions. 3 = a + b,7 = c + d,5 = a + c and 5 = b + d. What is a, b, c, d? Solution: a = 2, b = 1, c = 3, d = 4. However, also a = 0, b = 3, c = 5, d = 2 solves the problem. This system has not a unique solution even so there are 4 equations and 4 unknowns. A good introduction into MRI can be found online at (http://www.cis.rit.edu/htbooks/mri/inside.htm). o p q r a b c d INCONSISTENT. x − y = 4, y + z = 5, x + z = 6 is a system with no solutions. It is called inconsistent. x11 x12 x21 x22 a11 a12 a13 a14 a21 a24 a31 a34 a41 a42 a43 a44 EQUILIBRIUM. As an example of a system with many variables, consider a drum modeled by a fine net. The heights at each interior node needs the average the heights of the 4 neighboring nodes. The height at the boundary is fixed. With n2 nodes in the interior, we have to solve a system of n2 equations. For exam- ple, for n = 2 (see left), the n2 = 4 equations are x11 = a21 + a12 + x21 + x12, x12 = x11 + x13 + x22 + x22, x21 = x31+x11+x22+a43, x22 = x12+x21+a43+a34. To the right, we see the solution to a problem with n = 300, where the computer had to solve a system with 90 000 variables. LINEAR OR NONLINEAR? a) The ideal gas law PV = nKT for the P, V, T, the pressure, volume and temperature of the gas. b) The Hook law F = k(x − x0) relates the force F pulling a string at position x which is relaxed at x0.
MATRICES AND GAUSS-JORDAN Math 21b, O. Knill MATRIX FORMULATION. Consider the sys- tem of linear equations 3x − y − z = 0 −x + 2y − z = 0 −x − y + 3z = 9 The system can be written as Ax = b, where A is a matrix (called coefficient matrix) and and x and b are vectors. A =   3 −1 −1 −1 2 −1 −1 −1 3   , x =   x y z   , b =   0 0 9   . ((Ax)i is the dot product of the i’th row with x). We also look at the augmented matrix where one puts separators for clarity reasons. B =   3 −1 −1 | 0 −1 2 −1 | 0 −1 −1 3 | 9   . MATRIX ”JARGON”. A rectangular array of numbers is called a matrix. If the matrix has m rows and n columns, it is called a m × n matrix. A matrix with one column only is called a column vector, a matrix with one row a row vector. The entries of a matrix are denoted by aij, where i is the row and j is the column. In the case of the linear equation above, the matrix A is a square matrix and the augmented matrix B above is a 3 × 4 matrix. m n GAUSS-JORDAN ELIMINATION. Gauss-Jordan Elimination is a process, where successive subtraction of multiples of other rows or scaling brings the matrix into reduced row echelon form. The elimination process consists of three possible steps which are called elementary row operations: • Swap two rows. • Divide a row by a scalar • Subtract a multiple of a row from an other row. The process transfers a given matrix A into a new matrix rref(A) REDUCED ECHELON FORM. A matrix is called in reduced row echelon form 1) if a row has nonzero entries, then the first nonzero entry is 1. (leading one) 2) if a column contains a leading 1, then the other column entries are 0. 3) if a row has a leading 1, then every row above has leading 1’s to the left. Pro memoriam: Leaders like to be number one, are lonely and want other leaders above to their left. RANK. The number of leading 1 in rref(A) is called the rank of A. SOLUTIONS OF LINEAR EQUATIONS. If Ax = b is a linear system of equations with m equations and n unknowns, then A is a m × n matrix. We have the following three possibilities: • Exactly one solution. There is a leading 1 in each row but not in the last row. • Zero solutions. There is a leading 1 in the last row. • Infinitely many solutions. There are rows without leading 1 and no leading 1 is in the last row. JIUZHANG SUANSHU. The technique of successively eliminating variables from systems of linear equations is called Gauss elimination or Gauss Jordan elimination and appeared already in the Chinese manuscript ”Jiuzhang Suanshu” (’Nine Chapters on the Mathematical art’). The manuscript appeared around 200 BC in the Han dynasty and was probably used as a textbook. For more history of Chinese Mathematics, see http://aleph0.clarku.edu/ ˜ djoyce/mathhist/china.html.
EXAMPLES. The reduced echelon form of the augmented matrix B determines on how many solutions the linear system Ax = b has. THE GOOD (1 solution) THE BAD (0 solution) THE UGLY (∞ solutions) 0 1 2 | 2 1 −1 1 | 5 2 1 −1 | −2 1 −1 1 | 5 0 1 2 | 2 2 1 −1 | −2 1 −1 1 | 5 0 1 2 | 2 0 3 −3 | −12 1 −1 1 | 5 0 1 2 | 2 0 1 −1 | −4 1 0 3 | 7 0 1 2 | 2 0 0 −3 | −6 1 0 3 | 7 0 1 2 | 2 0 0 1 | 2   1 0 0 | 1 0 1 0 | −2 0 0 1 | 2   Rank(A) = 3, Rank(B) = 3.   0 1 2 | 2 1 −1 1 | 5 1 0 3 | −2     1 −1 1 | 5 0 1 2 | 2 1 0 3 | −2     1 −1 1 | 5 0 1 2 | 2 0 1 2 | −7     1 −1 1 | 5 0 1 2 | 2 0 0 0 | −9     1 0 3 | 7 0 1 2 | 2 0 0 0 | −9     1 0 3 | 7 0 1 2 | 2 0 0 0 | 1   Rank(A) = 2, Rank(B) = 3.   0 1 2 | 2 1 −1 1 | 5 1 0 3 | 7     1 −1 1 | 5 0 1 2 | 2 1 0 3 | 7     1 −1 1 | 5 0 1 2 | 2 0 1 2 | 2     1 −1 1 | 5 0 1 2 | 2 0 0 0 | 0     1 0 3 | 7 0 1 2 | 2 0 0 0 | 0   Rank(A) = 2, Rank(B) = 2. JORDAN. The German geodesist Wilhelm Jordan (1842-1899) applied the Gauss-Jordan method to finding squared errors to work on surveying. (An other ”Jordan”, the French Mathematician Camille Jordan (1838- 1922) worked on linear algebra topics also (Jordan form) and is often mistakenly credited with the Gauss-Jordan process.) GAUSS. Gauss developed Gaussian elimination around 1800 and used it to solve least squares problems in celestial mechanics and later in geodesic compu- tations. In 1809, Gauss published the book ”Theory of Motion of the Heavenly Bodies” in which he used the method for solving astronomical problems. One of Gauss successes was the prediction of an asteroid orbit using linear algebra. CERES. On 1. January of 1801, the Italian astronomer Giuseppe Piazzi (1746- 1826) discovered Ceres, the first and until 2001 the largest known asteroid in the solar system. (A new found object called 2001 KX76 is estimated to have a 1200 km diameter, half the size of Pluto) Ceres is a rock of 914 km diameter. (The pictures Ceres in infrared light). Gauss was able to predict the orbit of Ceres from a few observations. By parameterizing the orbit with parameters and solving a linear system of equations (similar to one of the homework problems, where you will fit a cubic curve from 4 observations), he was able to derive the orbit parameters.
ON SOLUTIONS OF LINEAR EQUATIONS Math 21b, O. Knill MATRIX. A rectangular array of numbers is called a matrix. A =     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn     = w w w w v v v 1 2 3 4 1 2 3 A matrix with m rows and n columns is called a m × n matrix. A matrix with one column is a column vector. The entries of a matrix are denoted aij, where i is the row number and j is the column number. ROW AND COLUMN PICTURE. Two interpretations Ax =     −w1− −w2− . . . −wm−       | x |   =     w1 · x w2 · x . . . wm · x     Ax =   | | · · · | v1 v2 · · · vn | | · · · |       x1 x2 · · · xm     = x1v1 +x2v2 +· · ·+xmvm = b . Row picture: each bi is the dot product of a row vector wi with x. Column picture: b is a sum of scaled column vectors vj. ”Row and Column at Harvard” EXAMPLE. The system of linear equations 3x − 4y − 5z = 0 −x + 2y − z = 0 −x − y + 3z = 9 is equivalent to Ax = b, where A is a coefficient matrix and x and b are vectors. A =   3 −4 −5 −1 2 −1 −1 −1 3   , x =   x y z   , b =   0 0 9   . The augmented matrix (separators for clarity) B =   3 −4 −5 | 0 −1 2 −1 | 0 −1 −1 3 | 9   . In this case, the row vectors of A are w1 = 3 −4 −5 w2 = −1 2 −1 w3 = −1 −1 3 The column vectors are v1 =   3 −1 −1   , v2 =   −4 −2 −1   , v3 =   −5 −1 3   Row picture: 0 = b1 = 3 −4 −5 ·   x y z   Column picture:   0 0 9   = x1   3 −1 −1   + x2   3 −1 −1   + x3   3 −1 −1   SOLUTIONS OF LINEAR EQUATIONS. A system Ax = b with m equations and n unknowns is defined by the m × n matrix A and the vector b. The row reduced matrix rref(B) of B determines the number of solutions of the system Ax = b. There are three possibilities: • Consistent: Exactly one solution. There is a leading 1 in each row but none in the last row of B. • Inconsistent: No solutions. There is a leading 1 in the last row of B. • Infinitely many solutions. There are rows of B without leading 1. If m < n (less equations then unknowns), then there are either zero or infinitely many solutions. The rank rank(A) of a matrix A is the number of leading ones in rref(A). 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
(exactly one solution) (no solution) (infinitely many solutions) MURPHYS LAW. ”If anything can go wrong, it will go wrong”. ”If you are feeling good, don’t worry, you will get over it!” ”Whenever you do Gauss-Jordan elimination, you screw up during the first couple of steps.” MURPHYS LAW IS TRUE. Two equations could contradict each other. Geometrically this means that the two planes do not intersect. This is possible if they are parallel. Even without two planes be- ing parallel, it is possible that there is no intersection between all three of them. Also possible is that we don’t have enough equations (for example because two equations are the same) and that there are many solutions. Furthermore, we can have too many equations and the four planes would not intersect. RELEVANCE OF EXCEPTIONAL CASES. There are important applications, where ”unusual” situations happen: For example in medical tomography, systems of equations appear which are ”ill posed”. In this case one has to be careful with the method. The linear equations are then obtained from a method called the Radon transform. The task for finding a good method had led to a Nobel prize in Medicis 1979 for Allan Cormack. Cormack had sabbaticals at Harvard and probably has done part of his work on tomography here. Tomography helps today for example for cancer treatment. MATRIX ALGEBRA. Matrices can be added, subtracted if they have the same size: A+B =     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn    +     b11 b12 · · · b1n b21 b22 · · · b2n · · · · · · · · · · · · bm1 bm2 · · · bmn     =     a11 + b11 a12 + b12 · · · a1n + b1n a21 + b21 a22 + b22 · · · a2n + b2n · · · · · · · · · · · · am1 + bm2 am2 + bm2 · · · amn + bmn     They can also be scaled by a scalar λ: λA = λ     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn     =     λa11 λa12 · · · λa1n λa21 λa22 · · · λa2n · · · · · · · · · · · · λam1 λam2 · · · λamn    
LINEAR TRANSFORMATIONS Math 21b, O. Knill TRANSFORMATIONS. A transformation T from a set X to a set Y is a rule, which assigns to every element in X an element y = T(x) in Y . One calls X the domain and Y the codomain. A transformation is also called a map. LINEAR TRANSFORMATION. A map T from Rn to Rm is called a linear transformation if there is a m × n matrix A such that T(x) = Ax . EXAMPLES. • To the linear transformation T(x, y) = (3x+4y, x+5y) belongs the matrix 3 4 1 5 . This transformation maps the plane onto itself. • T(x) = −3x is a linear transformation from the real line onto itself. The matrix is A = [−3]. • To T(x) = y ·x from R3 to R belongs the matrix A = y = y1 y2 y3 . This 1×3 matrix is also called a row vector. If the codomain is the real axes, one calls the map also a function. function defined on space. • T(x) = xy from R to R3 . A = y =   y1 y2 y3   is a 3 × 1 matrix which is also called a column vector. The map defines a line in space. • T(x, y, z) = (x, y) from R3 to R2 , A is the 2 × 3 matrix A =   1 0 0 1 0 0  . The map projects space onto a plane. • To the map T(x, y) = (x + y, x − y, 2x − 3y) belongs the 3 × 2 matrix A = 1 1 2 1 −1 −3 . The image of the map is a plane in three dimensional space. • If T(x) = x, then T is called the identity transformation. PROPERTIES OF LINEAR TRANSFORMATIONS. T(0) = 0 T(x + y) = T(x) + T(y) T(λx) = λT(x) In words: Linear transformations are compatible with addition and scalar multiplication. It does not matter, whether we add two vectors before the transformation or add the transformed vectors. ON LINEAR TRANSFORMATIONS. Linear transformations generalize the scaling transformation x → ax in one dimensions. They are important in • geometry (i.e. rotations, dilations, projections or reflections) • art (i.e. perspective, coordinate transformations), • CAD applications (i.e. projections), • physics (i.e. Lorentz transformations), • dynamics (linearizations of general maps are linear maps), • compression (i.e. using Fourier transform or Wavelet trans- form), • coding (many codes are linear codes), • probability (i.e. Markov processes). • linear equations (inversion is solving the equation)
LINEAR TRANSFORMATION OR NOT? (The square to the right is the image of the square to the left): COLUMN VECTORS. A linear transformation T(x) = Ax with A =   | | · · · | v1 v2 · · · vn | | · · · |   has the property that the column vector v1, vi, vn are the images of the standard vectors e1 =       1 · · · 0       . ei =       0 · 1 · 0       . en =       0 · · · 1       . In order to find the matrix of a linear transformation, look at the image of the standard vectors and use those to build the columns of the matrix. QUIZ. a) Find the matrix belonging to the linear transformation, which rotates a cube around the diagonal (1, 1, 1) by 120 degrees (2π/3). b) Find the linear transformation, which reflects a vector at the line containing the vector (1, 1, 1). INVERSE OF A TRANSFORMATION. If S is a second transformation such that S(Tx) = x, for every x, then S is called the inverse of T. We will discuss this more later. SOLVING A LINEAR SYSTEM OF EQUATIONS. Ax = b means to invert the linear transformation x → Ax. If the linear system has exactly one solution, then an inverse exists. We will write x = A−1 b and see that the inverse of a linear transformation is again a linear transformation. THE BRETSCHER CODE. Otto Bretschers book contains as a motivation a ”code”, where the encryption happens with the linear map T(x, y) = (x+3y, 2x+ 5y). The map has the inverse T −1 (x, y) = (−5x + 3y, 2x − y). Cryptologists use often the following approach to crack a encryption. If one knows the input and output of some data, one often can decode the key. Assume we know, the enemy uses a Bretscher code and we know that T(1, 1) = (3, 5) and T(2, 1) = (7, 5). How do we get the code? The problem is to find the matrix A = a b c d . 2x2 MATRIX. It is useful to decode the Bretscher code in general If ax + by = X and cx + dy = Y , then x = (dX − bY )/(ad − bc), y = (cX − aY )/(ad − bc). This is a linear transformation with matrix A = a b c d and the corresponding matrix is A−1 = d −b −c a /(ad − bc). ”Switch diagonally, negate the wings and scale with a cross”.
LINEAR TRAFOS IN GEOMETRY Math 21b, O. Knill LINEAR TRANSFORMATIONS DE- FORMING A BODY A CHARACTERIZATION OF LINEAR TRANSFORMATIONS: a transformation T from Rn to Rm which satisfies T(0) = 0, T(x + y) = T(x) + T(y) and T(λx) = λT(x) is a linear transformation. Proof. Call vi = T(ei) and define S(x) = Ax. Then S(ei) = T(ei). With x = x1e1 + ... + xnen, we have T(x) = T(x1e1 + ... + xnen) = x1v1 + ... + xnvn as well as S(x) = A(x1e1 + ... + xnen) = x1v1 + ... + xnvn proving T(x) = S(x) = Ax. SHEAR: A = 1 0 1 1 A = 1 1 0 1 In general, shears are transformation in the plane with the property that there is a vector w such that T(w) = w and T(x) − x is a multiple of w for all x. If u is orthogonal to w, then T(x) = x + (u · x)w. SCALING: A = 2 0 0 2 A = 1/2 0 0 1/2 One can also look at transformations which scale x differently then y and where A is a diagonal matrix. REFLECTION: A = cos(2α) sin(2α) sin(2α) − cos(2α) A = 1 0 0 −1 Any reflection at a line has the form of the matrix to the left. A reflection at a line containing a unit vector u is T(x) = 2(x · u)u − x with matrix A = 2u2 1 − 1 2u1u2 2u1u2 2u2 2 − 1 PROJECTION: A = 1 0 0 0 A = 0 0 0 1 A projection onto a line containing unit vector u is T(x) = (x · u)u with matrix A = u1u1 u2u1 u1u2 u2u2
ROTATION: A = −1 0 0 −1 A = cos(α) − sin(α) sin(α) cos(α) Any rotation has the form of the matrix to the right. ROTATION-DILATION: A = 2 −3 3 2 A = a −b b a A rotation dilation is a composition of a rotation by angle arctan(y/x) and a dilation by a factor x2 + y2. If z = x + iy and w = a + ib and T(x, y) = (X, Y ), then X + iY = zw. So a rotation dilation is tied to the process of the multiplication with a complex number. BOOST: A = cosh(α) sinh(α) sinh(α) cosh(α) The boost is a basic Lorentz transformation in special relativity. It acts on vectors (x, ct), where t is time, c is the speed of light and x is space. Unlike in Galileo transformation (x, t) → (x + vt, t) (which is a shear), time t also changes during the transformation. The transformation has the effect that it changes length (Lorentz contraction). The angle α is related to v by tanh(α) = v/c. One can write also A(x, ct) = ((x + vt)/γ, t + (v/c2 )/γx), with γ = 1 − v2/c2. ROTATION IN SPACE. Rotations in space are defined by an axes of rotation and an angle. A rotation by 120◦ around a line containing (0, 0, 0) and (1, 1, 1) blongs to A =   0 0 1 1 0 0 0 1 0   which permutes e1 → e2 → e3. REFLECTION AT PLANE. To a reflection at the xy-plane belongs the matrix A =   1 0 0 0 1 0 0 0 −1   as can be seen by looking at the images of ei. The picture to the right shows the textbook and reflections of it at two different mirrors. PROJECTION ONTO SPACE. To project a 4d-object into xyz-space, use for example the matrix A =     1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0    . The picture shows the pro- jection of the four dimensional cube (tesseract, hypercube) with 16 edges (±1, ±1, ±1, ±1). The tesseract is the theme of the horror movie ”hypercube”.
THE INVERSE Math 21b, O. Knill INVERTIBLE TRANSFORMATIONS. A map T from X to Y is invertible if there is for every y ∈ Y a unique point x ∈ X such that T(x) = y. ⇒ EXAMPLES. 1) T(x) = x3 is invertible from X = R to X = Y . 2) T(x) = x2 is not invertible from X = R to X = Y . 3) T(x, y) = (x2 + 3x − y, x) is invertible from X = R2 to Y = R2 . 4) T(x) = Ax linear and rref(A) has an empty row, then T is not invertible. 5) If T(x) = Ax is linear and rref(A) = 1n, then T is invertible. INVERSE OF LINEAR TRANSFORMATION. If A is a n × n matrix and T : x → Ax has an inverse S, then S is linear. The matrix A−1 belonging to S = T−1 is called the inverse matrix of A. First proof: check that S is linear using the characterization S(a + b) = S(a) + S(b), S(λa) = λS(a) of linearity. Second proof: construct the inverse using Gauss-Jordan elimination. FINDING THE INVERSE. Let 1n be the n × n identity matrix. Start with [A|1n] and perform Gauss-Jordan elimination. Then rref([A|1n]) = 1n|A−1 Proof. The elimination process actually solves Ax = ei simultaneously. This leads to solutions vi which are the columns of the inverse matrix A−1 because A−1 ei = vi. EXAMPLE. Find the inverse of A = 2 6 1 4 . 2 6 | 1 0 1 4 | 0 1 A | 12 1 3 | 1/2 0 1 4 | 0 1 .... | ... 1 3 | 1/2 0 0 1 | −1/2 1 .... | ... 1 0 | 2 −3 0 1 | −1/2 1 12 | A−1 The inverse is A−1 = 2 −3 −1/2 1 . THE INVERSE OF LINEAR MAPS R2 → R2 : If ad−bc = 0, the inverse of a linear transformation x → Ax with A = a b c d is given by the matrix A−1 = d −b −d a /(ad − bc). SHEAR: A = 1 0 −1 1 A−1 = 1 0 1 1
DIAGONAL: A = 2 0 0 3 A−1 = 1/2 0 0 1/3 REFLECTION: A = cos(2α) sin(2α) sin(2α) − cos(2α) A−1 = A = cos(2α) sin(2α) sin(2α) − cos(2α) ROTATION: A = cos(α) sin(α) − sin(α) cos(−α) A−1 = cos(α) − sin(α) sin(α) cos(α) ROTATION-DILATION: A = a −b b a A−1 = a/r2 b/r2 −b/r2 a/r2 , r2 = a2 + b2 BOOST: A = cosh(α) sinh(α) sinh(α) cosh(α) A−1 = cosh(α) − sinh(α) − sinh(α) cosh(α) NONINVERTIBLE EXAMPLE. The projection x → Ax = 1 0 0 0 is a non-invertible transformation. MORE ON SHEARS. The shears T(x, y) = (x + ay, y) or T(x, y) = (x, y + ax) in R2 can be generalized. A shear is a linear transformation which fixes some line L through the origin and which has the property that T(x) − x is parallel to L for all x. PROBLEM. T(x, y) = (3x/2 + y/2, y/2 − x/2) is a shear along a line L. Find L. SOLUTION. Solve the system T(x, y) = (x, y). You find that the vector (1, −1) is preserved. MORE ON PROJECTIONS. A linear map T with the property that T(T(x)) = T(x) is a projection. Examples: T(x) = (y · x)y is a projection onto a line spanned by a unit vector y. WHERE DO PROJECTIONS APPEAR? CAD: describe 3D objects using projections. A photo of an image is a projection. Compression algorithms like JPG or MPG or MP3 use projections where the high frequencies are cut away. MORE ON ROTATIONS. A linear map T which preserves the angle between two vectors and the length of each vector is called a rotation. Rotations form an important class of transformations and will be treated later in more detail. In two dimensions, every rotation is of the form x → A(x) with A = cos(φ) − sin(φ) sin(φ) cos(φ) . An example of a rotations in three dimensions are x → Ax, with A =   cos(φ) − sin(φ) 0 sin(φ) cos(φ) 0 0 0 1  . it is a rotation around the z axis. MORE ON REFLECTIONS. Reflections are linear transformations different from the identity which are equal to their own inverse. Examples: 2D reflections at the origin: A = −1 0 0 1 , 2D reflections at a line A = cos(2φ) sin(2φ) sin(2φ) − cos(2φ) . 3D reflections at origin: A =   −1 0 0 0 −1 0 0 0 −1  . 3D reflections at a line A =   −1 0 0 0 −1 0 0 0 1  . By the way: in any dimensions, to a reflection at the line containing the unit vector u belongs the matrix [A]ij = 2(uiuj) − [1n]ij, because [B]ij = uiuj is the matrix belonging to the projection onto the line. The reflection at a line containng the unit vector u = [u1, u2, u3] is A =   u2 1 − 1 u1u2 u1u3 u2u1 u2 2 − 1 u2u3 u3u1 u3u2 u2 3 − 1  . 3D reflection at a plane A =   1 0 0 0 1 0 0 0 −1  . Reflections are important symmetries in physics: T (time reflection), P (reflection at a mirror), C (change of charge) are reflections. It seems today that the composition of TCP is a fundamental symmetry in nature.
MATRIX PRODUCT Math 21b, O. Knill MATRIX PRODUCT. If B is a m × n matrix and A is a n × p matrix, then BA is defined as the m × p matrix with entries (BA)ij = n k=1 BikAkj. EXAMPLE. If B is a 3 × 4 matrix, and A is a 4 × 2 matrix then BA is a 3 × 2 matrix. B =   1 3 5 7 3 1 8 1 1 0 9 2  , A =     1 3 3 1 1 0 0 1    , BA =   1 3 5 7 3 1 8 1 1 0 9 2       1 3 3 1 1 0 0 1     =   15 13 14 11 10 5  . COMPOSING LINEAR TRANSFORMATIONS. If S : Rm → Rn , x → Ax and T : Rn → Rp , y → By are linear transformations, then their composition T ◦ S : x → B(A(x)) is a linear transformation from Rm to Rp . The corresponding matrix is the matrix product BA. EXAMPLE. Find the matrix which is a composition of a rotation around the x-axes by an agle π/2 followed by a rotation around the z-axes by an angle π/2. SOLUTION. The first transformation has the property that e1 → e1, e2 → e3, e3 → −e2, the second e1 → e2, e2 → −e1, e3 → e3. If A is the matrix belonging to the first transformation and B the second, then BA is the matrix to the composition. The composition maps e1 → −e2 → e3 → e1 is a rotation around a long diagonal. B =   0 −1 0 1 0 0 0 0 1   A =   1 0 0 0 0 −1 0 1 0  , BA =   0 0 1 1 0 0 0 1 0  . EXAMPLE. A rotation dilation is the composition of a rotation by α = arctan(b/a) and a dilation (=scale) by r = √ a2 + b2. REMARK. Matrix multiplication is a generalization of usual multiplication of numbers or the dot product. MATRIX ALGEBRA. Note that AB = BA in general! Otherwise, the same rules apply as for numbers: A(BC) = (AB)C, AA−1 = A−1 A = 1n, (AB)−1 = B−1 A−1 , A(B + C) = AB + AC, (B + C)A = BA + CA etc. PARTITIONED MATRICES. The entries of matrices can themselves be matrices. If B is a m × n matrix and A is a n × p matrix, and assume the entries are k × k matrices, then BA is a m × p matrix where each entry (BA)ij = n k=1 BikAkj is a k × k matrix. Partitioning matrices can be useful to improve the speed of matrix multiplication (i.e. Strassen algorithm). EXAMPLE. If A = A11 A12 0 A22 , where Aij are k × k matrices with the property that A11 and A22 are invertible, then B = A−1 11 −A−1 11 A12A−1 22 0 A−1 22 is the inverse of A. APPLICATIONS. (The material which follows is for motivation puposes only, more applications appear in the homework). 1 4 2 3 NETWORKS. Let us associate to the computer network (shown at the left) a matrix     0 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0     To a worm in the first computer we associate a vector     1 0 0 0    . The vector Ax has a 1 at the places, where the worm could be in the next step. The vector (AA)(x) tells, in how many ways the worm can go from the first computer to other hosts in 2 steps. In our case, it can go in three different ways back to the computer itself. Matrices help to solve combinatorial problems (see movie ”Good will hunting”). For example, what does [A1000 ]22 tell about the worm infection of the network? What does it mean if A100 has no zero entries?
FRACTALS. Closely related to linear maps are affine maps x → Ax + b. They are compositions of a linear map with a translation. It is not a linear map if B(0) = 0. Affine maps can be disguised as linear maps in the following way: let y = x 1 and defne the (n+1)∗(n+1) matrix B = A b 0 1 . Then By = Ax + b 1 . Fractals can be constructed by taking for example 3 affine maps R, S, T which contract area. For a given object Y0 define Y1 = R(Y0) ∪ S(Y0) ∪ T(Y0) and recursively Yk = R(Yk−1) ∪ S(Yk−1) ∪ T(Yk−1). The above picture shows Yk after some iterations. In the limit, for example if R(Y0), S(Y0) and T(Y0) are disjoint, the sets Yk converge to a fractal, an object with dimension strictly between 1 and 2. CHAOS. Consider a map in the plane like T : x y → 2x + 2 sin(x) − y x We apply this map again and again and follow the points (x1, y1) = T(x, y), (x2, y2) = T(T(x, y)), etc. One writes T n for the n-th iteration of the map and (xn, yn) for the image of (x, y) under the map T n . The linear approximation of the map at a point (x, y) is the matrix DT(x, y) = 2 + 2 cos(x) − 1 1 . (If T x y = f(x, y) g(x, y) , then the row vectors of DT(x, y) are just the gradients of f and g). T is called chaotic at (x, y), if the entries of D(T n )(x, y) grow exponentially fast with n. By the chain rule, D(T n ) is the product of matrices DT(xi, yi). For example, T is chaotic at (0, 0). If there is a positive probability to hit a chaotic point, then T is called chaotic. FALSE COLORS. Any color can be represented as a vector (r, g, b), where r ∈ [0, 1] is the red g ∈ [0, 1] is the green and b ∈ [0, 1] is the blue component. Changing colors in a picture means applying a transformation on the cube. Let T : (r, g, b) → (g, b, r) and S : (r, g, b) → (r, g, 0). What is the composition of these two linear maps? OPTICS. Matrices help to calculate the motion of light rays through lenses. A light ray y(s) = x + ms in the plane is described by a vector (x, m). Following the light ray over a distance of length L corresponds to the map (x, m) → (x + mL, m). In the lens, the ray is bent depending on the height x. The transformation in the lens is (x, m) → (x, m − kx), where k is the strength of the lense. x m → AL x m = 1 L 0 1 x m , x m → Bk x m = 1 0 −k 1 x m . Examples: 1) Eye looking far: ARBk. 2) Eye looking at distance L: ARBkAL. 3) Telescope: Bk2 ALBk1 . (More about it in problem 80 in section 2.4).
IMAGE AND KERNEL Math 21b, O. Knill IMAGE. If T : Rn → Rm is a linear transformation, then {T(x) | x ∈ Rn } is called the image of T. If T(x) = Ax, then the image of T is also called the image of A. We write im(A) or im(T). EXAMPLES. 1) If T(x, y, z) = (x, y, 0), then T(x) = A   x y z   =   1 0 0 0 1 0 0 0 0     x y z  . The image of T is the x − y plane. 2) If T(x, y)(cos(φ)x − sin(φ)y, sin(φ)x + cos(φ)y) is a rotation in the plane, then the image of T is the whole plane. 3) If T(x, y, z) = x + y + z, then the image of T is R. SPAN. The span of vectors v1, . . . , vk in Rn is the set of all combinations c1v1 + . . . ckvk, where ci are real numbers. PROPERTIES. The image of a linear transformation x → Ax is the span of the column vectors of A. The image of a linear transformation contains 0 and is closed under addition and scalar multiplication. KERNEL. If T : Rn → Rm is a linear transformation, then the set {x | T(x) = 0 } is called the kernel of T. If T(x) = Ax, then the kernel of T is also called the kernel of A. We write ker(A) or ker(T). EXAMPLES. (The same examples as above) 1) The kernel is the z-axes. Every vector (0, 0, z) is mapped to 0. 2) The kernel consists only of the point (0, 0, 0). 3) The kernel consists of all vector (x, y, z) for which x + y + z = 0. The kernel is a plane. PROPERTIES. The kernel of a linear transformation contains 0 and is closed under addition and scalar multiplication. IMAGE AND KERNEL OF INVERTIBLE MAPS. A linear map x → Ax, Rn → Rn is invertible if and only if ker(A) = {0} if and only if im(A) = Rn . HOW DO WE COMPUTE THE IMAGE? The rank of rref(A) is the dimension of the image. The column vectors of A span the image. (Dimension will be discussed later in detail). EXAMPLES. (The same examples as above) 1)   1 0 0   and   0 1 0   span the image. 2) cos(φ) − sin(φ) and sin(φ) cos(φ) span the image. 3) The 1D vector 1 spans the image. HOW DO WE COMPUTE THE KERNEL? Just solve Ax = 0. Form rref(A). For every column without leading 1 we can introduce a free variable si. If x is the solution to Axi = 0, where all sj are zero except si = 1, then x = j sjxj is a general vector in the kernel. EXAMPLE. Find the kernel of the linear map R3 → R4 , x → Ax with A =     1 3 0 2 6 5 3 9 1 −2 −6 0    . Gauss-Jordan elimination gives: B = rref(A) =     1 3 0 0 0 1 0 0 0 0 0 0    . We see one column without leading 1 (the second one). The equation Bx = 0 is equivalent to the system x + 3y = 0, z = 0. After fixing z = 0, can chose y = t freely and obtain from the first equation x = −3t. Therefore, the kernel consists of vectors t   −3 1 0  . In the book, you have a detailed calculation, in a case, where the kernel is 2 dimensional.
domain codomain kernel image WHY DO WE LOOK AT THE KERNEL? • It is useful to understand linear maps. To which degree are they non-invertible? • Helpful to understand quantitatively how many solutions a linear equation Ax = b has. If x is a solution and y is in the kernel of A, then also A(x + y) = b, so that x + y solves the system also. WHY DO WE LOOK AT THE IMAGE? • A solution Ax = b can be solved if and only if b is in the image of A. • Knowing about the kernel and the image is use- ful in the similar way that it is useful to know about the domain and range of a general map and to understand the graph of the map. In general, the abstraction helps to understand topics like error correcing codes (Problem 53/54 in Bretschers book), where two matrices H, M with the property that ker(H) = im(M) appear. The encoding x → Mx is robust in the sense that adding an error e to the result Mx → Mx + e can be corrected: H(Mx + e) = He allows to find e and so Mx. This allows to recover x = PMx with a projection P. PROBLEM. Find ker(A) and im(A) for the 1 × 3 matrix A = [5, 1, 4], a row vector. ANSWER. A · x = Ax = 5x + y + 4z = 0 shows that the kernel is a plane with normal vector [5, 1, 4] through the origin. The image is the codomain, which is R. PROBLEM. Find ker(A) and im(A) of the linear map x → v × x, (the cross product with v. ANSWER. The kernel consists of the line spanned by v, the image is the plane orthogonal to v. PROBLEM. Fix a vector w in space. Find ker(A) and image im(A) of the linear map from R6 to R3 given by x, y → [x, v, y] = (x × y) · w. ANSWER. The kernel consist of all (x, y) such that their cross product orthogonal to w. This means that the plane spanned by x, y contains w. PROBLEM Find ker(T) and im(T) if T is a composition of a rotation R by 90 degrees around the z-axes with with a projection onto the x-z plane. ANSWER. The kernel of the projection is the y axes. The x axes is rotated into the y axes and therefore the kernel of T. The image is the x-z plane. PROBLEM. Can the kernel of a square matrix A be trivial if A2 = 0, where 0 is the matrix containing only 0? ANSWER. No: if the kernel were trivial, then A were invertible and A2 were invertible and be different from 0. PROBLEM. Is it possible that a 3 × 3 matrix A satisfies ker(A) = R3 without A = 0? ANSWER. No, if A = 0, then A contains a nonzero entry and therefore, a column vector which is nonzero. PROBLEM. What is the kernel and image of a projection onto the plane Σ : x − y + 2z = 0? ANSWER. The kernel consists of all vectors orthogonal to Σ, the image is the plane Σ. PROBLEM. Given two square matrices A, B and assume AB = BA. You know ker(A) and ker(B). What can you say about ker(AB)? ANSWER. ker(A) is contained in ker(BA). Similar ker(B) is contained in ker(AB). Because AB = BA, the kernel of AB contains both ker(A) and ker(B). (It can be bigger: A = B = 0 1 0 0 .) PROBLEM. What is the kernel of the partitioned matrix A 0 0 B if ker(A) and ker(B) are known? ANSWER. The kernel consists of all vectors (x, y), where x in ker(A) and y ∈ ker(B).
BASIS Math 21b, O. Knill LINEAR SUBSPACE. A subset X of Rn which is closed under addition and scalar multiplication is called a linear subspace of Rn . WHICH OF THE FOLLOWING SETS ARE LINEAR SPACES? a) The kernel of a linear map. b) The image of a linear map. c) The upper half plane. d) the line x + y = 0. e) The plane x + y + z = 1. f) The unit circle. BASIS. A set of vectors v1, . . . , vm is a basis of a linear subspace X of Rn if they are linear independent and if they span the space X. Linear independent means that there are no nontrivial linear relations aiv1 + . . . + amvm = 0. Spanning the space means that very vector v can be written as a linear combination v = a1v1 +. . .+amvm of basis vectors. A linear subspace is a set containing {0} which is closed under addition and scaling. EXAMPLE 1) The vectors v1 =   1 1 0   , v2 =   0 1 1   , v3 =   1 0 1   form a basis in the three dimensional space. If v =   4 3 5  , then v = v1 + 2v2 + 3v3 and this representation is unique. We can find the coefficients by solving Ax = v, where A has the vi as column vectors. In our case, A =   1 0 1 1 1 0 0 1 1     x y z   =   4 3 5   had the unique solution x = 1, y = 2, z = 3 leading to v = v1 + 2v2 + 3v3. EXAMPLE 2) Two nonzero vectors in the plane which are not parallel form a basis. EXAMPLE 3) Three vectors in R3 which are in a plane form not a basis. EXAMPLE 4) Two vectors in R3 do not form a basis. EXAMPLE 5) Three nonzero vectors in R3 which are not contained in a single plane form a basis in R3 . EXAMPLE 6) The columns of an invertible n × n matrix form a basis in Rn as we will see. FACT. If v1, ..., vn is a basis, then every vector v can be represented uniquely as a linear combination of the vi. v = a1v1 + . . . + anvn. REASON. There is at least one representation because the vectors vi span the space. If there were two different representations v = a1v1 +. . .+anvn and v = b1v1 +. . .+bnvn, then subtraction would lead to 0 = (a1 − b1)v1 + ... + (an − bn)vn. This nontrivial linear relation of the vi is forbidden by assumption. FACT. If n vectors v1, ..., vn span a space and w1, ..., wm are linear independent, then m ≤ n. REASON. This is intuitively clear in dimensions up to 3. You can not have more then 4 vectors in space which are linearly independent. We will give a precise reason later. A BASIS DEFINES AN INVERTIBLE MATRIX. The n × n matrix A =   | | | v1 v2 . . . vn | | |   is invertible if and only if v1, . . . , vn define a basis in Rn . EXAMPLE. In the example 1), the 3 × 3 matrix A is invertible. FINDING A BASIS FOR THE KERNEL. To solve Ax = 0, we bring the matrix A into the reduced row echelon form rref(A). For every non-leading entry in rref(A), we will get a free variable ti. Writing the system Ax = 0 with these free variables gives us an equation x = i tivi. The vectors vi form a basis of the kernel of A.
REMARK. The problem to find a basis for all vectors wi which are orthogonal to a given set of vectors, is equivalent to the problem to find a basis for the kernel of the matrix which has the vectors wi in its rows. FINDING A BASIS FOR THE IMAGE. Bring the m × n matrix A into the form rref(A). Call a column a pivot column, if it contains a leading 1. The corresponding set of column vectors of the original matrix A form a basis for the image because they are linearly independent and are in the image. Assume there are k of them. They span the image because there are (k − n) non-leading entries in the matrix. REMARK. The problem to find a basis of the subspace generated by v1, . . . , vn, is the problem to find a basis for the image of the matrix A with column vectors v1, ..., vn. EXAMPLES. 1) Two vectors on a line are linear dependent. One is a multiple of the other. 2) Three vectors in the plane are linear dependent. One can find a relation av1 + bv2 = v3 by changing the size of the lengths of the vectors v1, v2 until v3 becomes the diagonal of the parallelogram spanned by v1, v2. 3) Four vectors in three dimensional space are linearly dependent. As in the plane one can change the length of the vectors to make v4 a diagonal of the parallelepiped spanned by v1, v2, v3. EXAMPLE. Let A be the matrix A = 0 0 1 1 1 0 1 1 1 . In reduced row echelon form is B = rref(A) = 1 1 0 0 0 1 0 0 0 . To determine a basis of the kernel we write Bx = 0 as a system of linear equations: x + y = 0, z = 0. The variable y is the free variable. With y = t, x = −t is fixed. The linear system rref(A)x = 0 is solved by x =   x y z   = t   −1 1 0  . So, v =   −1 1 0   is a basis of the kernel. EXAMPLE. Because the first and third vectors in rref(A) are columns with leading 1’s, the first and third columns v1 =   0 1 1   , v2 =   1 0 1   of A form a basis of the image of A. WHY DO WE INTRODUCE BASIS VECTORS? Wouldn’t it be just easier to look at the standard basis vectors e1, . . . , en only? The rea- son for more general basis vectors is that they allow a more flexible adaptation to the situation. A person in Paris prefers a different set of basis vectors than a person in Boston. We will also see that in many applications, problems can be solved easier with the right basis. For example, to describe the reflection of a ray at a plane or at a curve, it is preferable to use basis vec- tors which are tangent or orthogonal. When looking at a rotation, it is good to have one basis vector in the axis of rotation, the other two orthogonal to the axis. Choosing the right basis will be especially im- portant when studying differential equations. A PROBLEM. Let A =   1 2 3 1 1 1 0 1 2  . Find a basis for ker(A) and im(A). SOLUTION. From rref(A) = 1 0 −1 0 1 2 0 0 0 we see that = v =   1 −2 1   is in the kernel. The two column vectors   1 1 0   , 2, 1, 1 of A form a basis of the image because the first and third column are pivot columns.
DIMENSION Math 21b, O. Knill REVIEW LINEAR SPACE. X linear space: 0 ∈ X closed under addition and scalar multiplication. Examples: Rn , X = ker(A), X = im(A) are linear spaces. REVIEW BASIS. B = {v1, . . . , vn} ⊂ X B linear independent: c1v1 +...+cnvn = 0 implies c1 = ... = cn = 0. B span X: v ∈ X then v = a1v1 + ... + anvn. B basis: both linear independent and span. BASIS: ENOUGH BUT NOT TOO MUCH. The spanning condition for a basis assures that there are enough vectors to represent any other vector, the linear independence condi- tion assures that there are not too many vectors. A basis is, where J.Lo meets A.Hi: Left: J.Lopez in ”Enough”, right ”The man who new too much” by A.Hitchcock AN UNUSUAL EXAMPLE. Let X be the space of polynomials up to degree 4. For example p(x) = 3x4 + 2x3 + x + 5 is an element in this space. It is straightforward to check that X is a linear space. The ”zero vector” is the function f(x) = 0 which is zero everywhere. We claim that e1(x) = 1, e2(x) = x, e3(x) = x2 , e4(x) = x3 and e5(x) = x4 form a basis in X. PROOF. The vectors span the space: every polynomial f(x) = c0 + c1x + c2x2 + c3x3 + c4x4 is a sum f = c0e1 + c1e2 + c2e3 + c3e4 + c4e5 of basis elements. The vectors are linearly independent: a nontrivial relation 0 = c0e1 + c1e2 + c2e3 + c3e4 + c4e5 would mean that c0 + c1x + c2x2 + c3x3 + c4x4 = 0 for all x which is not possible unless all cj are zero. DIMENSION. The number of elements in a basis of X is independent of the choice of the basis. It is called the dimension of X. UNIQUE REPRESENTATION. v1, . . . , vn ∈ X ba- sis ⇒ every v ∈ X can be written uniquely as a sum v = a1v1 + . . . + anvn. EXAMPLES. The dimension of {0} is zero. The dimension of any line 1. The dimension of a plane is 2, the dimension of three dimensional space is 3. The dimension is independent on where the space is embedded in. For example: a line in the plane and a line in space have dimension 1. IN THE UNUSUAL EXAMPLE. The set of polynomials of degree ≤ 4 form a linear space of dimension 5. REVIEW: KERNEL AND IMAGE. We can construct a basis of the kernel and image of a linear transformation T(x) = Ax by forming B = rrefA. The set of Pivot columns in A form a basis of the image of T, a basis for the kernel is obtained by solving Bx = 0 and introducing free variables for each non-pivot column. EXAMPLE. Let X the linear space from above. Define the linear transformation T(f)(x) = f (x). For example: T(x3 + 2x4 ) = 3x2 + 8x3 . Find a basis for the kernel and image of this transformation. SOLUTION. Because T(e1) = 0, T(e2) = e1, T(e3) = 2e2, T(e4) = 3e3, T(e5) = 4e4, the matrix of T is A =       0 1 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0       which is almost in row reduced echelon form. You see that the last four columns are pivot columns. The kernel is spanned by e1 which corresponds to the con- stant function f(x) = 1. The image is the 4 dimensional space of polynomials of degree ≤ 3. Mathematicians call a fact a ”lemma” if it is used to prove a theorem and if does not deserve the be honored by the name ”theorem”:
LEMMA. If n vectors v1, ..., vn span a space and w1, ..., wm are linearly independent, then m ≤ n. REASON. Assume m > n. Because vi span, each vector wi can be written as j aijvj = wi. After doing Gauss-Jordan elimination of the augmented (m × (n + 1))-matrix a11 . . . a1n | w1 . . . . . . . . . | . . . am1 . . . amn | wm which belong to these relations, we end up with a matrix, which has in the last line 0 ... 0 | b1w1 + ... + bmwm . But this means that b1w1 + ... + bmwm = 0 and this is a nontrivial relation between the wi. Contradiction. The assumption m > n was absurd. THEOREM. Given a basis A = {v1, ..., vn } and a basis B = {w1, ..., .wm } of X, then m = n. PROOF. Because A spans X and B is linearly independent, we know that n ≤ m. Because B spans X and A is linearly independent also m ≤ n holds. Together, n ≤ m and m ≤ n implies n = m. DIMENSION OF THE KERNEL. The number of columns in rref(A) without leading 1’s is the dimension of the kernel dim(ker(A)): we can introduce a parameter for each such col- umn when solving Ax = 0 using Gauss-Jordan elimination. DIMENSION OF THE IMAGE. The number of leading 1 in rref(A), the rank of A is the dimension of the image dim(im(A)) because every such leading 1 produces a differ- ent column vector (called pivot column vectors) and these column vectors are linearly independent. 1 1 1 1 * * * * DIMENSION FORMULA: (A : Rn → Rm ) dim(ker(A)) + dim(im(A)) = n EXAMPLE: A invertible is equivalant that the dimension of the image is n and that the dim(ker)(A) = 0. PROOF. There are n columns. dim(ker(A)) is the number of columns without leading 1, dim(im(A)) is the number of columns with leading 1. EXAMPLE. In the space X of polynomials of degree 4 define T(f)(x) = f (x). The kernel consists of linear polynomials spanned by e1, e2, the image consists of all polynomials of degree ≤ 2. It is spanned by e3, e4, e5. Indeed dim(ker(T)) + dim(im(T)) = 2 + 3 = 5 = n. FRACTAL DIMENSION. Mathematicians study objects with non-integer dimension since the early 20’th cen- tury. The topic became fashion in the 80’ies, when people started to generate fractals on computers. To define fractals, the notion of dimension is extended: define a s-volume of accuracy r of a bounded set X in Rn as the infimum of all hs,r(X) = Uj |Uj|s , where Uj are cubes of length ≤ r covering X and |Uj| is the length of Uj. The s-volume is then defined as the limit hs(X) of hs(X) = hs,r(X) when r → 0. The dimension is the limiting value s, where hs(X) jumps from 0 to ∞. Examples: 1) A smooth curve X of length 1 in the plane can be covered with n squares Uj of length |Uj| = 1/n and hs,1/n(X) = n j=1(1/n)s = n(1/n)s . If s < 1, this converges, if s > 1 it diverges for n → ∞. So dim(X) = 1. 2) A square X in space of area 1 can be covered with n2 cubes Uj of length |Uj| = 1/n and hs,1/n(X) = n2 j=1(1/n)s = n2 (1/n)s which converges to 0 for s < 2 and diverges for s > 2 so that dim(X) = 2. 3) The Shirpinski carpet is constructed recur- sively by dividing a square in 9 equal squares and cutting away the middle one, repeating this proce- dure with each of the squares etc. At the k’th step, we need 8k squares of length 1/3k to cover the car- pet. The s−volume hs,1/3k (X) of accuracy 1/3k is 8k (1/3k )s = 8k /3ks , which goes to 0 for k → ∞ if 3ks < 8k or s < d = log(8)/ log(3) and diverges if s > d. The dimension is d = log(8)/ log(3) = 1.893.. INFINITE DIMENSIONS. Linear spaces also can have infinite dimensions. An example is the set X of all continuous maps from the real R to R. It contains all polynomials and because Xn the space of polynomials of degree n with dimension n + 1 is contained in X, the space X is infinite dimensional. By the way, there are functions like g(x) = ∞ n=0 sin(2n x)/2n in X which have graphs of fractal dimension > 1 and which are not differentiable at any point x.
COORDINATES Math 21b, O. Knill B-COORDINATES. Given a basis v1, . . . vn, define the matrix S =   | . . . | v1 . . . vn | . . . |  . It is invertible. If x = i civi, then ci are called the B-coordinates of v. We write [x]B =   c1 . . . cn  . If x =   x1 . . . xn  , we have x = S([x]B). B-coordinates of x are obtained by applying S−1 to the coordinates of the standard basis: [x]B = S−1 (x) . EXAMPLE. If v1 = 1 2 and v2 = 3 5 , then S = 1 3 2 5 . A vector v = 6 9 has the coordinates S−1 v = −5 3 2 −1 6 9 = −3 3 Indeed, as we can check, −3v1 + 3v2 = v. EXAMPLE. Let V be the plane x + y − z = 1. Find a basis, in which every vector in the plane has the form  a b 0  . SOLUTION. Find a basis, such that two vectors v1, v2 are in the plane and such that a third vector v3 is linearly independent to the first two. Since (1, 0, 1), (0, 1, 1) are points in the plane and (0, 0, 0) is in the plane, we can choose v1 =   1 0 1   v2 =   0 1 1   and v3 =   1 1 −1   which is perpendicular to the plane. EXAMPLE. Find the coordinates of v = 2 3 with respect to the basis B = {v1 = 1 0 , v2 = 1 1 }. We have S = 1 1 0 1 and S−1 = 1 −1 0 1 . Therefore [v]B = S−1 v = −1 3 . Indeed −1v1 + 3v2 = v. B-MATRIX. If B = {v1, . . . , vn} is a basis in Rn and T is a linear transformation on Rn , then the B-matrix of T is defined as B =   | . . . | [T(v1)]B . . . [T(vn)]B | . . . |   COORDINATES HISTORY. Cartesian geometry was introduced by Fermat and Descartes (1596-1650) around 1636. It had a large influence on mathematics. Algebraic methods were introduced into geometry. The beginning of the vector concept came only later at the beginning of the 19’th Century with the work of Bolzano (1781-1848). The full power of coordinates becomes possible if we allow to chose our coordinate system adapted to the situation. Descartes biography shows how far dedication to the teaching of mathematics can go ...: (...) In 1649 Queen Christina of Sweden persuaded Descartes to go to Stockholm. However the Queen wanted to draw tangents at 5 a.m. in the morning and Descartes broke the habit of his lifetime of getting up at 11 o’clock. After only a few months in the cold northern climate, walking to the palace at 5 o’clock every morning, he died of pneumonia. Fermat Descartes Christina Bolzano
CREATIVITY THROUGH LAZINESS? Legend tells that Descartes (1596-1650) introduced coordinates while lying on the bed, watching a fly (around 1630), that Archimedes (285-212 BC) discovered a method to find the volume of bodies while relaxing in the bath and that Newton (1643-1727) discovered New- ton’s law while lying under an apple tree. Other examples are August Kekul´e’s analysis of the Benzene molecular structure in a dream (a snake biting in its tail revieled the ring structure) or Steven Hawkings discovery that black holes can radiate (while shaving). While unclear which of this is actually true, there is a pattern: According David Perkins (at Harvard school of education): ”The Eureka effect”, many creative breakthroughs have in common: a long search without apparent progress, a prevailing moment and break through, and finally, a transformation and realization. A breakthrough in a lazy moment is typical - but only after long struggle and hard work. EXAMPLE. Let T be the reflection at the plane x + 2y + 3z = 0. Find the transformation matrix B in the basis v1 =   2 −1 0   v2 =   1 2 3   v3 =   0 3 −2  . Because T(v1) = v1 = [e1]B, T(v2) = v2 = [e2]B, T(v3) = −v3 = −[e3]B, the solution is B =   1 0 0 0 −1 0 0 0 1  . SIMILARITY. The B matrix of A is B = S−1 AS, where S =   | . . . | v1 . . . vn | . . . |  . One says B is similar to A. EXAMPLE. If A is similar to B, then A2 +A+1 is similar to B2 +B+1. B = S−1 AS, B2 = S−1 B2 S, S−1 S = 1, S−1 (A2 + A + 1)S = B2 + B + 1. PROPERTIES OF SIMILARITY. A, B similar and B, C similar, then A, C are similar. If A is similar to B, then B is similar to A. QUIZZ: If A is a 2 × 2 matrix and let S = 0 1 1 0 , What is S−1 AS? MAIN IDEA OF CONJUGATION S. The transformation S−1 maps the coordinates from the standard basis into the coordinates of the new basis. In order to see what a transformation A does in the new coordinates, we map it back to the old coordinates, apply A and then map it back again to the new coordinates: B = S−1 AS. The transformation in standard coordinates. v S ← w = [v]B A ↓ ↓ B Av S−1 → Bw The transformation in B-coordinates. QUESTION. Can the matrix A which belongs to a projection from R3 to a plane x + y + 6z = 0 be similar to a matrix which is a rotation by 20 degrees around the z axis? No: a non-invertible A can not be similar to an invertible B: if it were, the inverse A = SBS−1 would exist: A−1 = SB−1 S−1 . PROBLEM. Find a clever basis for the reflection of a light ray at the line x+2y = 0. v1 = 1 2 , v2 = −2 1 . SOLUTION. You can achieve B = 1 0 0 −1 with S = 1 −2 2 1 . PROBLEM. Are all shears A = 1 a 0 1 with a = 0 similar? Yes, use a basis v1 = ae1 and v2 = e2. PROBLEM. You know A = 3 0 −1 2 is similar to B = 1 0 0 −1 with S = 0 −1 1 1 . Find eA = 1+A+A2 +A3 /3!+.... SOLUTION. Because Bk = S−1 Ak S for every k we have eA = SeB S−1 = 1/e 0 e + 1/e e .
ORTHOGONALITY Math 21b, O. Knill ORTHOGONALITY. v and w are called orthogonal if v · w = 0. Examples. 1) 1 2 and 6 −3 are orthogonal in R2 . 2) v and w are both orthogonal to v × w in R3 . v is called a unit vector if ||v|| = √ v · v = 1. B = {v1, . . . , vn} are called orthogonal if they are pairwise orthogonal. They are called orthonormal if they are also unit vectors. A basis is called an orthonormal basis if it is orthonormal. For an orthonormal basis, the matrix Aij = vi · vj is the unit matrix. FACT. Orthogonal vectors are linearly independent and n orthogonal vectors in Rn form a basis. Proof. The dot product of a linear relation a1v1 + . . . + anvn = 0 with vk gives akvk · vk = ak||vk||2 = 0 so that ak = 0. If we have n linear independent vectors in Rn then they automatically span the space. ORTHOGONAL COMPLEMENT. A vector w ∈ Rn is called orthogonal to a linear space V if w is orthogonal to every vector in v ∈ V . The orthogonal complement of a linear space V is the set W of all vectors which are orthogonal to V . It forms a linear space because v · w1 = 0, v · w2 = 0 implies v · (w1 + w2) = 0. ORTHOGONAL PROJECTION. The orthogonal projection onto a linear space V with orthnormal basis v1, . . . , vn is the linear map T(x) = projV (x) = (v1 · x)v1 + . . . + (vn · x)vn The vector x − projV (x) is in the orthogonal complement of V . (Note that vi in the projection formula are unit vectors, they have also to be orthogonal.) SPECIAL CASE. For an orthonormal basis vi, one can write x = (v1 · x)v1 + ... + (vn · x)vn . PYTHAGORAS: If x and y are orthogonal, then ||x + y||2 = ||x||2 + ||y||2 . Proof. Expand (x + y) · (x + y). PROJECTIONS DO NOT INCREASE LENGTH: ||projV(x)|| ≤ ||x||. Proof. Use Pythagoras: on x = projV(x) + (x − projV(x))). If ||projV(x)|| = ||x||, then x is in V . CAUCHY-SCHWARTZ INEQUALITY: |x · y| ≤ ||x|| ||y|| . Proof: x · y = ||x||||y|| cos(α). If |x · y| = ||x||||y||, then x and y are parallel. TRIANGLE INEQUALITY: ||x + y|| ≤ ||x|| + ||y||. Proof: (x + y) · (x + y) = ||x||2 + ||y||2 + 2x · y ≤ ||x||2 + ||y||2 + 2||x||||y|| = (||x|| + ||y||)2 . ANGLE. The angle between two vectors x, y is α = arccos x · y ||x||||y|| . CORRELATION. cos(α) = x·y ||x||||y|| is called the correlation between x and y. It is a number in [−1, 1]. EXAMPLE. The angle between two orthogonal vectors is 90 degrees or 270 degrees. If x and y represent data showing the deviation from the mean, then x·y ||x||||y|| is called the statistical correlation of the data. QUESTION. Express the fact that x is in the kernel of a matrix A using orthogonality. ANSWER: Ax = 0 means that wk · x = 0 for every row vector wk of Rn . REMARK. We will call later the matrix AT , obtained by switching rows and columns of A the transpose of A. You see already that the image of AT is orthogonal to the kernel of A. QUESTION. Find a basis for the orthogonal complement of the linear space V spanned by     1 2 3 4     ,     4 5 6 7    . ANSWER: The orthogonality of     x y z u     to the two vectors means solving the linear system of equations x + 2y + 3z + 4w = 0, 4x + 5y + 6z + 7w = 0. An other way to solve it: the kernel of A = 1 2 3 4 4 5 6 7 is the orthogonal complement of V . This reduces the problem to an older problem.
ON THE RELEVANCE OF ORTHOGONALITY. 1) During the pyramid age in Egypt (from -2800 til -2300 BC), the Egyptians used ropes divided into length ratios 3 : 4 : 5 to build triangles. This allowed them to triangulate areas quite pre- cisely: for example to build irrigation needed because the Nile was reshaping the land constantly or to build the pyramids: for the great pyramid at Giza with a base length of 230 meters, the average error on each side is less then 20cm, an error of less then 1/1000. A key to achieve this was orthogonality. 2) During one of Thales (-624 til -548 BC) journeys to Egypt, he used a geometrical trick to measure the height of the great pyramid. He measured the size of the shadow of the pyramid. Using a stick, he found the relation between the length of the stick and the length of its shadow. The same length ratio applies to the pyramid (orthogonal triangles). Thales found also that triangles inscribed into a circle and having as the base as the diameter must have a right angle. 3) The Pythagoreans (-572 until -507) were interested in the dis- covery that the squares of a lengths of a triangle with two or- thogonal sides would add up as a2 + b2 = c2 . They were puzzled in assigning a length to the diagonal of the unit square, which is √ 2. This number is irrational because √ 2 = p/q would imply that q2 = 2p2 . While the prime factorization of q2 contains an even power of 2, the prime factorization of 2p2 contains an odd power of 2. 4) Eratosthenes (-274 until 194) realized that while the sun rays were orthogonal to the ground in the town of Scene, this did no more do so at the town of Alexandria, where they would hit the ground at 7.2 degrees). Because the distance was about 500 miles and 7.2 is 1/50 of 360 degrees, he measured the circumference of the earth as 25’000 miles - pretty close to the actual value 24’874 miles. 5) Closely related to orthogonality is parallelism. For a long time mathematicians tried to prove Euclid’s parallel axiom using other postulates of Euclid (-325 until -265). These attempts had to fail because there are geometries in which parallel lines always meet (like on the sphere) or geometries, where parallel lines never meet (the Poincar´e half plane). Also these geometries can be studied using linear algebra. The geometry on the sphere with rotations, the geometry on the half plane uses M¨obius transformations, 2 × 2 matrices with determinant one. 6) The question whether the angles of a right triangle are in reality always add up to 180 degrees became an issue when geometries where discovered, in which the measure- ment depends on the position in space. Riemannian geometry, founded 150 years ago, is the foundation of general relativity a theory which describes gravity geometrically: the presence of mass bends space-time, where the dot product can depend on space. Orthogonality becomes relative too. 7) In probability theory the notion of independence or decorrelation is used. For example, when throwing a dice, the number shown by the first dice is independent and decorrelated from the number shown by the second dice. Decorrelation is identical to orthogonality, when vectors are associated to the random variables. The correlation coefficient between two vectors v, w is defined as v · w/(|v|w|). It is the cosine of the angle between these vectors. 8) In quantum mechanics, states of atoms are described by functions in a linear space of functions. The states with energy −EB/n2 (where EB = 13.6eV is the Bohr energy) in a hydrogen atom. States in an atom are orthogonal. Two states of two different atoms which don’t interact are orthogonal. One of the challenges in quantum computing, where the computation deals with qubits (=vectors) is that orthogonality is not preserved during the computation. Different states can interact. This coupling is called decoherence.
GRAM SCHMIDT AND QR Math 21b, O. Knill GRAM-SCHMIDT PROCESS. Let v1, ..., vn be a basis in V . Let u1 = v1 and w1 = u1/||u1||. The Gram-Schmidt process recursively constructs from the already constructed orthonormal set w1, ..., wi−1 which spans a linear space Vi−1 the new vector ui = (vi − projVi−1 (vi)) which is orthogonal to Vi−1, and then normalizing ui to to get wi = ui/||ui||. Each vector wi is orthonormal to the linear space Vi−1. The vectors {w1, .., wn} form an orthonormal basis in V . EXAMPLE. Find an orthonormal basis for v1 =   2 0 0  , v2 =   1 3 0   and v3 =   1 2 5  . SOLUTION. 1. w1 = v1/||v1|| =   1 0 0  . 2. u2 = (v2 − projV1 (v2)) = v2 − (w1 · v2)w1 =   0 3 0  . w2 = u2/||u2|| =   0 1 0  . 3. u3 = (v3 − projV2 (v3)) = v3 − (w1 · v3)w1 − (w2 · v3)w2 =   0 0 5  , w3 = u3/||u3|| =   0 0 1  . QR FACTORIZATION. The formulas can be written as v1 = ||v1||w1 = r11w1 · · · vi = (w1 · vi)w1 + · · · + (wi−1 · vi)wi−1 + ||ui||wi = ri1w1 + · · · + riiwi · · · vn = (w1 · vn)w1 + · · · + (wn−1 · vn)wn−1 + ||un||wn = rn1w1 + · · · + rnnwn which means in matrix form A =   | | · | v1 · · · · vm | | · |   =   | | · | w1 · · · · wm | | · |     r11 r12 · r1m 0 r22 · r2m 0 0 · rmm   = QR , where A and Q are n × m matrices and R is a m × m matrix. Any matrix A with linearly independent columns can be decom- posed as A = QR, where Q has orthonormal column vectors and where R is an upper triangular square matrix.
BACK TO THE EXAMPLE. The matrix with the vectors v1, v2, v3 is A =   2 1 1 0 3 2 0 0 5  . v1 = ||v1||w1 v2 = (w1 · v2)w1 + ||u2||w2 v3 = (w1 · v3)w1 + (w2 · v3)w2 + ||u3||w3, so that Q =   1 0 0 0 1 0 0 0 1   and R =   2 1 1 0 3 2 0 0 5  . PRO MEMORIA. While building the matrix R we keep track of the vectors ui during the Gram-Schmidt procedure. At the end you have vectors ui, vi, wi and the matrix R has ||ui|| in the diagonal as well as the dot products wi · vj in the upper right triangle. PROBLEM. Make the QR decomposition of A = 0 −1 1 1 . w1 = 0 −1 . u2 = −1 1 − 0 1 = −1 0 . w2 = u2. A = 0 −1 1 0 1 1 0 1 = QR. WHY do we care to have an orthonormal basis? • An orthonormal basis looks like the standard basis v1 = (1, 0, ..., 0), . . ., vn = (0, 0, ..., 1). Actually, we will see that an orthonormal basis into a standard basis or a mirror of the standard basis. • The Gram-Schmidt process is tied to the factorization A = QR. The later helps to solve linear equations. In physical problems like in astrophysics, the numerical methods to simulate the problems one needs to invert huge matrices in every time step of the evolution. The reason why this is necessary sometimes is to assure the numerical method is stable implicit methods. Inverting A−1 = R−1 Q−1 is easy because R and Q are easy to invert. • For many physical problems like in quantum mechanics or dynamical systems, matrices are symmetric A∗ = A, where A∗ ij = Aji. For such matrices, there will a natural orthonormal basis. • The formula for the projection onto a linear subspace V simplifies with an orthonormal basis vj in V : projV (x) = (v1 · x)w1 + · · · + (wn · x)wn. • An orthonormal basis simplifies compuations due to the presence of many zeros wj · wi = 0. This is especially the case for problems with symmetry. • The Gram Schmidt process can be used to define and construct classes of classical polynomials, which are important in physics. Examples are Chebyshev polynomials, Laguerre polynomials or Hermite polynomi- als. • QR factorization allows fast computation of the determinant, least square solutions R−1 Q−1 b of overde- termined systems Ax = b or finding eigenvalues - all topics which will appear later. SOME HISTORY. The recursive formulae of the process were stated by Erhard Schmidt (1876-1959) in 1907. The essence of the formulae were already in a 1883 paper of J.P.Gram in 1883 which Schmidt mentions in a footnote. The process seems already have been used by Laplace (1749-1827) and was also used by Cauchy (1789-1857) in 1836. Gram Schmidt Laplace Cauchy
ORTHOGONAL MATRICES 10/27/2002 Math 21b, O. Knill TRANSPOSE The transpose of a matrix A is the matrix (AT )ij = Aji. If A is a n×m matrix, then AT is a m × n matrix. For square matrices, the transposed matrix is obtained by reflecting the matrix at the diagonal. EXAMPLES The transpose of a vector A =   1 2 3   is the row vector AT = 1 2 3 . The transpose of the matrix 1 2 3 4 is the matrix 1 3 2 4 . A PROPERTY OF THE TRANSPOSE. a) x · Ay = AT x · y. b) (AB)T = BT AT . c) (AT )T = A. PROOFS. a) Because x · Ay = j i xiAijyj and AT x · y = j i Ajixiyj the two expressions are the same by renaming i and j. b) (AB)kl = i AkiBil. (AB)T kl = i AliBik = AT BT . c) ((AT )T )ij = (AT )ji = Aij. ORTHOGONAL MATRIX. A n × n matrix A is called orthogonal if AT A = 1. The corresponding linear transformation is called orthogonal. INVERSE. It is easy to invert an orthogonal matrix: A−1 = AT . EXAMPLES. The rotation matrix A = cos(φ) sin(φ) − sin(φ) cos(φ) is orthogonal because its column vectors have length 1 and are orthogonal to each other. Indeed: AT A = cos(φ) sin(φ) − sin(φ) cos(φ) · cos(φ) − sin(φ) sin(φ) cos(φ) = 1 0 0 1 . A reflection at a line is an orthogonal transformation because the columns of the matrix A have length 1 and are orthogonal. Indeed: AT A = cos(2φ) sin(2φ) sin(2φ) − cos(2φ) · cos(2φ) sin(2φ) sin(2φ) − cos(2φ) = 1 0 0 1 . FACTS. An orthogonal transformation preserves the dot product: Ax · Ay = x · y Proof: this is a homework assignment: Hint: just look at the properties of the transpose. Orthogonal transformations preserve the length of vectors as well as the angles between them. Proof. We have ||Ax||2 = Ax · Ax = x · x ||x||2 . Let α be the angle between x and y and let β denote the angle between Ax and Ay and α the angle between x and y. Using Ax·Ay = x·y we get ||Ax||||Ay|| cos(β) = Ax·Ay = x · y = ||x||||y|| cos(α). Because ||Ax|| = ||x||, ||Ay|| = ||y||, this means cos(α) = cos(β). Because this property holds for all vectors we can rotate x in plane V spanned by x and y by an angle φ to get cos(α+φ) = cos(β +φ) for all φ. Differentiation with respect to φ at φ = 0 shows also sin(α) = sin(β) so that α = β. ORTHOGONAL MATRICES AND BASIS. A linear transformation A is orthogonal if and only if the column vectors of A form an orthonormal basis. (That is what AT A = 1n means.) COMPOSITION OF ORTHOGONAL TRANSFORMATIONS. The composition of two orthogonal transformations is orthogonal. The inverse of an orthogonal transformation is orthogonal. Proof. The properties of the transpose give (AB)T AB = BT AT AB = BT B = 1 and (A−1 )T A−1 = (AT )−1 A−1 = (AAT )−1 = 1. EXAMPLES. The composition of two reflections at a line is a rotation. The composition of two rotations is a rotation. The composition of a reflections at a plane with a reflection at an other plane is a rotation (the axis of rotation is the intersection of the planes).
ORTHOGONAL PROJECTIONS. The orthogonal projection P onto a linear space with orthonormal basis v1, . . . , vn is the matrix AAT , where A is the matrix with column vectors vi. To see this just translate the formula Px = (v1 · x)v1 + . . . + (vn · x)vn into the language of matrices: AT x is a vector with components bi = (vi · x) and Ab is the sum of the bivi, where vi are the column vectors of A. Orthogonal projections are no orthogonal transformations in general! EXAMPLE. Find the orthogonal projection P from R3 to the linear space spanned by v1 =   0 3 4   1 5 and v2 =   1 0 0  . Solution: AAT =   0 1 3/5 0 4/5 0   0 3/5 4/5 1 0 0 =   1 0 0 0 9/25 12/25 0 12/25 16/25  . WHY ARE ORTHOGONAL TRANSFORMATIONS USEFUL? • In Physics, Galileo transformations are compositions of translations with orthogonal transformations. The laws of classical mechanics are invariant under such transformations. This is a symmetry. • Many coordinate transformations are orthogonal transformations. We will see examples when dealing with differential equations. • In the QR decomposition of a matrix A, the matrix Q is orthogonal. Because Q−1 = Qt , this allows to invert A easier. • Fourier transformations are orthogonal transformations. We will see this transformation later in the course. In application, it is useful in computer graphics (i.e. JPG) and sound compression (i.e. MP3). • Quantum mechanical evolutions (when written as real matrices) are orthogonal transformations. WHICH OF THE FOLLOWING MAPS ARE ORTHOGONAL TRANSFORMATIONS?: Yes No Shear in the plane. Yes No Projection in three dimensions onto a plane. Yes No Reflection in two dimensions at the origin. Yes No Reflection in three dimensions at a plane. Yes No Dilation with factor 2. Yes No The Lorenz boost x → Ax in the plane with A = cosh(α) sinh(α) sinh(α) cosh(α) Yes No A translation. CHANGING COORDINATES ON THE EARTH. Problem: what is the matrix which rotates a point on earth with (latitude,longitude)=(a1, b1) to a point with (latitude,longitude)=(a2, b2)? Solution: The ma- trix which rotate the point (0, 0) to (a, b) a composition of two rotations. The first rotation brings the point into the right latitude, the second brings the point into the right longitude. Ra,b =  cos(b) − sin(b) 0 sin(b) cos(b) 0 0 0 1     cos(a) 0 − sin(a) 0 1 0 sin(a) 0 cos(a)  . To bring a point (a1, b1) to a point (a2, b2), we form A = Ra2,b2 R−1 a1,b1 . EXAMPLE: With Cambridge (USA): (a1, b1) = (42.366944, 288.893889)π/180 and Z¨urich (Switzerland): (a2, b2) = (47.377778, 8.551111)π/180, we get the matrix A =   0.178313 −0.980176 −0.0863732 0.983567 0.180074 −0.0129873 0.028284 −0.082638 0.996178  .
LEAST SQUARES AND DATA Math 21b, O. Knill GOAL. The best possible ”solution” of an inconsistent linear systems Ax = b will be called the least square solution. It is the orthogonal projection of b onto the image im(A) of A. The theory of the kernel and the image of linear transformations helps to understand this situation and leads to an explicit formula for the least square fit. Why do we care about non-consistent systems? Often we have to solve linear systems of equations with more constraints than variables. An example is when we try to find the best polynomial which passes through a set of points. This problem is called data fitting. If we wanted to accommodate all data, the degree of the polynomial would become too large, the fit would look too wiggly. Taking a smaller degree polynomial will not only be more convenient but also give a better picture. Especially important is regression, the fitting of data with lines. The above pictures show 30 data points which are fitted best with polynomials of degree 1, 6, 11 and 16. The first linear fit maybe tells most about the trend of the data. THE ORTHOGONAL COMPLEMENT OF im(A). Because a vector is in the kernel of AT if and only if it is orthogonal to the rows of AT and so to the columns of A, the kernel of AT is the orthogonal complement of im(A): (im(A))⊥ = ker(AT ) EXAMPLES. 1) A =   a b c  . The kernel V of AT = a b c consists of all vectors satisfying ax + by + cz = 0. V is a plane. The orthogonal complement is the image of A which is spanned by the normal vector   a b c   to the plane. 2) A = 1 1 0 0 . The image of A is spanned by 1 0 the kernel of AT is spanned by 0 1 . ORTHOGONAL PROJECTION. If b is a vector and V is a linear subspace, then projV (b) is the vector closest to b on V : given any other vector v on V , one can form the triangle b, v, projV (b) which has a right angle at projV (b) and invoke Pythagoras. THE KERNEL OF AT A. For any m × n matrix ker(A) = ker(AT A) Proof. ⊂ is clear. On the other hand AT Av = 0 means that Av is in the kernel of AT . But since the image of A is orthogonal to the kernel of AT , we have Av = 0, which means v is in the kernel of A. LEAST SQUARE SOLUTION. The least square so- lution of Ax = b is the vector x∗ such that Ax∗ is closest to b from all other vectors Ax. In other words, Ax∗ = projV (b), where V = im(V ). Because b − Ax∗ is in V ⊥ = im(A)⊥ = ker(AT ), we have AT (b − Ax∗ ) = 0. The last equation means that x∗ is a solution of AT Ax = AT b, the normal equation of Ax = b . If the kernel of A is triv- ial, then the kernel of AT A is trivial and AT A can be inverted. Therefore x∗ = (AT A)−1 AT b is the least square solution. V=im(A) T ker(A )= im(A) T b Ax* A (b-Ax)=0 T Ax=A(A A) A bT T-1* WHY LEAST SQUARES? If x∗ is the least square solution of Ax = b then ||Ax∗ − b|| ≤ ||Ax − b|| for all x. Proof. AT (Ax∗ − b) = 0 means that Ax∗ − b is in the kernel of AT which is orthogonal to V = im(A). That is projV (b) = Ax∗ which is the closest point to b on V .
ORTHOGONAL PROJECTION If v1, . . . , vn is a basis in V which is not necessarily orthonormal, then the orthogonal projection is x → A(AT A)−1 AT (x) where A = [v1, . . . , vn]. Proof. x = (AT A)−1 AT b is the least square solution of Ax = b. Therefore Ax = A(AT A)−1 AT b is the vector in im(A) closest to b. Special case: If w1, . . . , wn is an orthonormal basis in V , we had seen earlier that AAT with A = [w1, . . . , wn] is the orthogonal projection onto V (this was just rewriting Ax = (w1 · x)w1 + · · · + (wn · x)wn in matrix form.) This follows from the above formula because AT A = I in that case. EXAMPLE Let A =   1 0 2 0 0 1  . The orthogonal projection onto V = im(A) is b → A(AT A)−1 AT b. We have AT A = 5 0 2 1 and A(AT A)−1 AT =   1/5 2/5 0 2/5 4/5 0 0 0 1  . For example, the projection of b =   0 1 0   is x∗ =   2/5 4/5 0   and the distance to b is 1/ √ 5. The point x∗ is the point on V which is closest to b. Remember the formula for the distance of b to a plane V with normal vector n? It was d = |n · b|/||n||. In our case, we can take n = [−2, 1, 0] this formula gives the distance 1/ √ 5. Let’s check: the distance of x∗ and b is ||(2/5, −1/5, 0)|| = 1/ √ 5. EXAMPLE. Let A =     1 2 0 1    . Problem: find the matrix of the orthogonal projection onto the image of A. The image of A is a one-dimensional line spanned by the vector v = (1, 2, 0, 1). We calculate AT A = 6. Then A(AT A)−1 AT =     1 2 0 1     1 2 0 1 /6 =     1 2 0 1 2 4 0 2 0 0 0 0 1 2 0 1     /6 DATA FIT. Find a quadratic polynomial p(t) = at2 + bt + c which best fits the four data points (−1, 8), (0, 8), (1, 4), (2, 16). A =     1 −1 1 0 0 1 1 1 1 4 2 1     b =     8 8 4 16     T . AT A =   18 8 6 8 6 2 6 2 4   and x∗ = (AT A)−1 AT b =   3 −1 5  . Software packages like Mathematica have already built in the facility to fit numerical data: The series expansion of f showed that indeed, f(t) = 5 − t + 3t2 is indeed best quadratic fit. Actually, Mathematica does the same to find the fit then what we do: ”Solving” an inconsistent system of linear equations as best as possible. PROBLEM: Prove im(A) = im(AAT ). SOLUTION. The image of AAT is contained in the image of A because we can write v = AAT x as v = Ay with y = AT x. On the other hand, if v is in the image of A, then v = Ax. If x = y + z, where y in the kernel of A and z orthogonal to the kernel of A, then Ax = Az. Because z is orthogonal to the kernel of A, it is in the image of AT . Therefore, z = AT u and v = Az = AAT u is in the image of AAT .
THE DIMENSION OF THE MASS COAST LINE Math 21b, O.Knill To measure the dimension of an object, one can count the number f(n) of boxes of length 1/n needed to cover the object and see how f(n) grows. If f(n) grows like n2 , then the dimension is 2, if n(k) grows like n, the dimension is 1. For fractal objects, like coast lines, the number of boxes grows like ns for a number s between 1 and 2. The dimension is obtained by correlating yk = log2 f(k) with xk = log2(k). We measure: The Massachusetts coast line is a fractal of dimension 1.3. We measure the data: f(1) = 5, f(2) = 12, f(4) = 32 and f(8) = 72. A plot of the data (xk, yk) = (log2(k), log2 f(k)) together with a least square fit can be seen to the right. The slope of the line is the dimension of the coast. It is about 1.295. In order to measure the di- mension better, one would need better maps. xk yk 0 log2(5) 1 log2(12) 2 log2(32) 3 log2(72)
Finding the best linear fit y = ax + b is equivalent to find the least square solution of the system 0a + b = log2(5), 1a + b = log2(12), 2a + b = log2(32), 3a + b = log2(72) which is Ax = b with x = a b , b =     2.3 3.5 5 6     and A =     0 1 1 1 2 1 3 1    . We have AT A = 14 6 6 4 , (AT A)−1 = 2 −3 −3 7 /10 and B = (AT A)−1 AT = −3 −1 1 3 7 4 1 −2 /10. We get = a b = Bb = 1.29 2.32 . COMPARISON: THE DIMENSION OF THE CIRCLE Let us compare this with a smooth curve. To cover a circle with squares of size 2π/2n , we need about f(n) = 2n to cover the cir- cle. We measure that the circle is not a fractal. It has dimension 1. xk yk 3 log2(8) = 3 4 log2(16) = 4 5 log2(32) = 5 6 log2(64) = 6 COMPARISON: THE DIMENSION OF THE DISK For an other comparison, we take a disk. To cover the disk with squares of size 2π/2n , we need about f(n) = 22n squares. We measure that the disk is not a frac- tal. It has dimension 2. xk yk 3 log2(5) = 2.3 4 log2(13) = 3.7 5 log2(49) = 5.61 6 log2(213) = 7.73 REMARKS Calculating the dimension of coast lines is a classical illustration of ”fractal theory”. The coast of Brittain has been measured to have a dimension of 1.3 too. For natural objects, it is typical that measuring the length on a smaller scale gives larger results: one can measure empirically the dimension of mountains, clouds, plants, snowflakes, the lung etc. The dimension is an indicator how rough a curve or surface is.
DETERMINANTS I Math 21b, O. Knill PERMUTATIONS. A permutation of n elements {1, 2, . . . , n} is a rearrangement of {1, 2, . . . , n}. There are n! = n · (n − 1)... · 1 different permutations of {1, 2, . . . , n}: fixing the position of first element leaves (n − 1)! possibilities to permute the rest. EXAMPLE. There are 6 permutations of {1, 2, 3}: (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1). PATTERNS AND SIGN. The matrix A with zeros everywhere except Ai,π(i) = 1 is called a permutation matrix or the pattern of π. An inversion is a pair k < l such that σ(k) < σ(l). The sign of a permutation π, denoted by (−1)σ(π) is (−1) if there are an odd number of inversions in the pattern. Otherwise, the sign is defined to be +1. EXAMPLES. σ(1, 2) = 0, (−1)σ = 1, σ(2, 1) = 1, (−1)σ = −1. The permutations(1, 2, 3), (3, 2, 1), (2, 3, 1) have sign 1, the permutations (1, 3, 2), (3, 2, 1), (2, 1, 3) have sign −1. DETERMINANT The determinant of a n × n matrix A is the sum π(−1)σ(π) A1π(1)A2π(2) · · · Anπ(n), where π runs over all permutations of {1, 2, . . . , n} and σ(π) is the sign of the permutation. 2 × 2 CASE. The determinant of A = a b c d is ad − bc. There are two permutations of (1, 2). The identity permutation (1, 2) gives A11A12, the permutation (2, 1) gives A21A22. If you have seen some multi-variable calculus, you know that det(A) is the area of the parallelogram spanned by the column vectors of A. The two vectors form a basis if and only if det(A) = 0. 3 × 3 CASE. The determinant of A = a b c d e f g h i is aei + bfg + cdh − ceg − fha − bdi corresponding to the 6 permutations of (1, 2, 3). Geometrically, det(A) is the volume of the parallelepiped spanned by the column vectors of A. The three vectors form a basis if and only if det(A) = 0. EXAMPLE DIAGONAL AND TRIANGULAR MATRICES. The determinant of a diagonal or triangular matrix is the product of the diagonal elements. EXAMPLE PERMUTATION MATRICES. The determinant of a matrix which has everywhere zeros except Aiπ(j) = 1 is just the sign (−1)σ(π) of the permutation. HOW FAST CAN WE COMPUTE THE DETERMINANT?. The cost to find the determinant is the same as for the Gauss- Jordan elimination as we will see below The graph to the left shows some measurements of the time needed for a CAS to calculate the determinant in dependence on the size of the n × n matrix. The matrix size ranges from n=1 to n=300. We also see a best cubic fit of these data using the least square method from the last lesson. It is the cubic p(x) = a + bx + cx2 + dx3 which fits best through the 300 data points. WHY DO WE CARE ABOUT DETERMINANTS? • check invertibility of matrices • have geometric interpretation as volume • explicit algebraic expressions for inverting a ma- trix • as a natural functional on matrices it appears in formulas in particle or statistical physics • allow to define orientation in any dimensions • appear in change of variable formulas in higher dimensional integration. • proposed alternative concepts are unnatural, hard to teach and and harder to understand • determinants are fun
TRIANGULAR AND DIAGONAL MATRICES. The determinant of a diagonal or triangular ma- trix is the product of its diagonal elements. Example: det(     1 0 0 0 4 5 0 0 2 3 4 0 1 1 2 1    ) = 20. PARTITIONED MATRICES. The determinant of a partitioned matrix A 0 0 B is the product det(A)det(B). Example det(     3 4 0 0 1 2 0 0 0 0 4 −2 0 0 2 2    ) = 2 · 12 = 24. LINEARITY OF THE DETERMINANT. If the columns of A and B are the same except for the i’th column, det([v1, ..., v, ...vn]] + det([v1, ..., w, ...vn]] = det([v1, ..., v + w, ...vn]] . In general, one has det([v1, ..., kv, ...vn]] = k det([v1, ..., v, ...vn]] . The same holds for rows. These identities follow directly from the original definition of the determimant. PROPERTIES OF DETERMINANTS. det(AB) = det(A)det(B) . det(A−1 ) = det(A)−1 det(SAS−1 ) = det(A) . det(AT ) = det(A) de det(− If B is obtained from A by switching two rows, then det(B) = −det(A). If B is obtained by adding an other row to a given row, then this does not change the value of the determinant. PROOF OF det(AB) = det(A)det(B), one brings the n × n matrix [A|AB] into row reduced echelon form. Similar than the augmented matrix [A|b] was brought into the form [1|A−1 b], we end up with [1|A−1 AB] = [1|B]. By looking at the n × n matrix to the left during Gauss-Jordan elimination, the determinant has changed by a factor det(A). We end up with a matrix B which has determinant det(B). Therefore, det(AB) = det(A)det(B). PROOF OF det(AT ) = det(A). The transpose of a pattern is a pattern with the same signature. PROBLEM. Find the determi- nant of A =     0 0 0 2 1 2 4 5 0 7 2 9 0 0 6 4    . SOLUTION. Three row transpositions give B =     1 2 4 5 0 7 2 9 0 0 6 4 0 0 0 2     a ma- trix which has determinant 84. Therefore det(A) = (−1)3 det(B) = −84. PROBLEM. Determine det(A100 ), where A is the matrix 1 2 3 16 . SOLUTION. det(A) = 10, det(A100 ) = (det(A))100 = 10100 = 1 · gogool. This name as well as the gogoolplex = 1010100 are official. They are huge numbers: the mass of the universe for example is 1052 kg and 1/101051 is the chance to find yourself on Mars by quantum fluctuations. (R.E. Crandall, Scient. Amer., Feb. 1997). ROW REDUCED ECHELON FORM. Determining rref(A) also determines det(A). If A is a matrix and αi are the factors which are used to scale different rows and s is the number of times, two rows are switched, then det(A) = (−1)s α1 · · · αn det(rref(A)) . INVERTIBILITY. Because of the last formula: A n × n matrix A is invertible if and only if det(A) = 0. THE LAPLACE EXPANSION. In order to calculate by hand the determinant of n × n matrices A = aij for n > 3, the following expansion is useful. Choose a column i. For each entry aji in that column, take the (n − 1) × (n − 1) matrix Aij called minor which does not contain the i’th column and j’th row. Then det(A) = (−1)i+1 ai1det(Ai1) + · · · + (−1)i+n aindet(Ain) = n j=1(−1)i+j aijdet(Aij) ORTHOGONAL MATRICES. Because QT Q = 1, we have det(Q)2 = 1 and so |det(Q)| = 1 Rotations have determinant 1, reflections have determinant −1. QR DECOMPOSITION. If A = QR, then det(A) = det(Q)det(R). The determinant of Q is ±1, the determinant of R is the product of the diagonal elements of R.
DETERMINANTS II, Math 21b, O.Knill DETERMINANT AND VOLUME. If A is a n × n matrix, then |det(A)| is the volume of the n-dimensional parallelepiped En spanned by the n column vectors vj of A. Proof. Use the QR decomposition A = QR, where Q is orthogonal and R is upper triangular. From QQT = 1, we get 1 = det(Q)det(QT ) = det(Q)2 see that |det(Q)| = 1. Therefore, det(A) = ±det(R). The determinant of R is the product of the ||ui|| = ||vi − projVj−1 vi|| which was the distance from vi to Vj−1. The volume vol(Ej) of a j-dimensional parallelepiped Ej with base Ej−1 in Vj−1 and height ||ui|| is vol(Ej−1)||uj||. Inductively vol(Ej) = ||uj||vol(Ej−1) and therefore vol(En) = n j=1 ||uj|| = det(R). The volume of a k dimensional parallelepiped defined by the vectors v1, . . . , vk is det(AT A). Proof. QT Q = In gives AT A = (QR)T (QR) = RT QT QR = RT R. So, det(RT R) = det(R)2 = ( k j=1 ||uj||)2 . (Note that A is a n × k matrix and that AT A = RT R and R are k × k matrices.) ORIENTATION. Determinants allow to define the orientation of n vectors in n-dimensional space. This is ”handy” because there is no ”right hand rule” in hyperspace... To do so, define the matrix A with column vectors vj and define the orientation as the sign of det(A). In three dimensions, this agrees with the right hand rule: if v1 is the thumb, v2 is the pointing finger and v3 is the middle finger, then their orientation is positive. b i det x det(A) =i CRAMER’S RULE. This is an explicit formula for the solution of Ax = b. If Ai denotes the matrix, where the column vi of A is replaced by b, then xi = det(Ai)/det(A) Proof. det(Ai) = det([v1, . . . , b, . . . , vn] = det([v1, . . . , (Ax), . . . , vn] = det([v1, . . . , i xivi, . . . , vn] = xidet([v1, . . . , vi, . . . , vn]) = xidet(A) EXAMPLE. Solve the system 5x+3y = 8, 8x+5y = 2 using Cramers rule. This linear system with A = 5 3 8 5 and b = 8 2 . We get x = det 8 3 2 5 = 34y = det 5 8 8 2 = −54. GABRIEL CRAMER. (1704-1752), born in Geneva, Switzerland, he worked on geometry and analysis. Cramer used the rule named after him in a book ”Introduction `a l’analyse des lignes courbes alg´ebraique”, where he solved like this a system of equations with 5 unknowns. According to a short biography of Cramer by J.J O’Connor and E F Robertson, the rule had however been used already before by other mathematicians. Solving systems with Cramers formulas is slower than by Gaussian elimination. The rule is still important. For example, if A or b depends on a parameter t, and we want to see how x depends on the parameter t one can find explicit formulas for (d/dt)xi(t). THE INVERSE OF A MATRIX. Because the columns of A−1 are solutions of Ax = ei, where ej are basis vectors, Cramers rule together with the Laplace expansion gives the formula: [A−1 ]ij = (−1)i+j det(Aji)/det(A) Bij = (−1)i+j det(Aji) is called the classical adjoint of A. Note the change ij → ji. Don’t confuse the classical adjoint with the transpose AT which is sometimes also called the adjoint.
EXAMPLE. A =   2 3 1 5 2 4 6 0 7   has det(A) = −17 and we get A−1 =   14 −21 10 −11 8 −3 −12 18 −11   /(−17): B11 = (−1)1+1 det 2 4 0 7 = 14. B12 = (−1)1+2 det 3 1 0 7 = −21. B13 = (−1)1+3 det 3 1 2 4 = 10. B21 = (−1)2+1 det 5 4 6 7 = −11. B22 = (−1)2+2 det 2 1 6 7 = 8. B23 = (−1)2+3 det 2 1 5 4 = −3. B31 = (−1)3+1 det 5 2 6 0 = −12. B32 = (−1)3+2 det 2 3 6 0 = 18. B33 = (−1)3+3 det 2 3 5 2 = −11. THE ART OF CALCULATING DETERMINANTS. When confronted with a matrix, it is good to go through a checklist of methods to crack the determinant. Often, there are different possibilities to solve the problem, in many cases the solution is particularly simple using one method. • Is it a 2 × 2 or 3 × 3 matrix? • Is it a upper or lower triangular matrix? • Is it a partitioned matrix? • Is it a trick: like A1000 ? • Does geometry imply noninvertibility? • Do you see duplicated columns or rows? • Can you row reduce to a triangular case? • Are there only a few nonzero patters? • Laplace expansion with some row or column? • Later: Can you see the eigenvalues of A − λIn? EXAMPLES. 1) A =       1 2 3 4 5 2 4 6 8 10 5 5 5 5 4 1 3 2 7 4 3 2 8 4 9       Try row reduction. 2) A =       2 1 0 0 0 1 1 1 0 0 0 0 2 1 0 0 0 0 3 1 0 0 0 0 4       Laplace expansion. 3) A =       1 1 0 0 0 1 2 2 0 0 0 1 1 0 0 0 0 0 1 3 0 0 0 4 2       Partitioned matrix. 4) A =       1 6 10 1 15 2 8 17 1 29 0 0 3 8 12 0 0 0 4 9 0 0 0 0 5       Make it tridiagonal. APPLICATION HOFSTADTER BUTTERFLY. In solid state physics, one is interested in the function f(E) = det(L − EIn), where L =        λ cos(α) 1 0 · 0 1 1 λ cos(2α) 1 · · 0 0 1 · · · · · · · · 1 0 0 · · 1 λ cos((n − 1)α) 1 1 0 · 0 1 λ cos(nα)        describes an electron in a periodic crystal, E is the energy and α = 2π/n. The electron can move as a Bloch wave whenever the determinant is negative. These intervals form the spectrum of the quantum mechanical system. A physicist is interested in the rate of change of f(E) or its dependence on λ when E is fixed. . -4 -2 2 4 1 2 3 4 5 6 The graph to the left shows the function E → log(|det(L − EIn)|) in the case λ = 2 and n = 5. In the energy intervals, where this function is zero, the electron can move, otherwise the crystal is an in- sulator. The picture to the right shows the spectrum of the crystal depending on α. It is called the ”Hofs- tadter butterfly” made popular in the book ”G¨odel, Escher Bach” by Douglas Hofstadter.
COMPUTING EIGENVALUES Math 21b, O.Knill THE TRACE. The trace of a matrix A is the sum of its diagonal elements. EXAMPLES. The trace of A =   1 2 3 3 4 5 6 7 8   is 1 + 4 + 8 = 13. The trace of a skew symmetric matrix A is zero because there are zeros in the diagonal. The trace of In is n. CHARACTERISTIC POLYNOMIAL. The polynomial fA(λ) = det(λIn − A) is called the characteristic polynomial of A. EXAMPLE. The characteristic polynomial of A above is x3 − 13x2 + 15x. The eigenvalues of A are the roots of the characteristic polynomial fA(λ). Proof. If λ is an eigenvalue of A with eigenfunction v, then A − λ has v in the kernel and A − λ is not invertible so that fA(λ) = det(λ − A) = 0. The polynomial has the form fA(λ) = λn − tr(A)λn−1 + · · · + (−1)n det(A) THE 2x2 CASE. The characteristic polynomial of A = a b c d is fA(λ) = λ2 − (a + d)/2λ + (ad − bc). The eigenvalues are λ± = T/2 ± (T/2)2 − D, where T is the trace and D is the determinant. In order that this is real, we must have (T/2)2 ≥ D. Away from that parabola, there are two different eigenvalues. The map A contracts volume for |D| < 1. EXAMPLE. The characteristic polynomial of A = 1 2 0 2 is λ2 − 3λ + 2 which has the roots 1, 2: pA(λ) = (λ − 1)(λ − 2). THE FIBONNACCI RABBITS. The Fibonacci’s recursion un+1 = un + un−1 defines the growth of the rabbit population. We have seen that it can be rewritten as un+1 un = A un un−1 with A = 1 1 1 0 . The roots of the characteristic polynomial pA(x) = λ2 − λ − 1. are ( √ 5 + 1)/2, ( √ 5 − 1)/2. ALGEBRAIC MULTIPLICITY. If fA(λ) = (λ − λ0)k g(λ), where g(λ0) = 0 then λ is said to be an eigenvalue of algebraic multiplicity k. EXAMPLE:   1 1 1 0 1 1 0 0 2   has the eigenvalue λ = 1 with algebraic multiplicity 2 and the eigenvalue λ = 2 with algebraic multiplicity 1. HOW TO COMPUTE EIGENVECTORS? Because (λ − A)v = 0, the vector v is in the kernel of λ − A. We know how to compute the kernel. EXAMPLE FIBONNACCI. The kernel of λI2 − A = λ± − 1 −1 −1 λ± is spanned by v+ = (1 + √ 5)/2, 1 and v− = (1 − √ 5)/2, 1 . They form a basis B. SOLUTION OF FIBONNACCI. To obtain a formula for An v with v = 1 0 , we form [v]B = 1 −1 / √ 5. Now, un+1 un = An v = An (v+/ √ 5 − v−/ √ 5) = An v+/ √ 5− An (v−/ √ 5 = λn +v+/ √ 5+ λn −v+/ √ 5. We see that un = [(1+ √ 5 2 )n + (1− √ 5 2 )n ]/ √ 5. ROOTS OF POLYNOMIALS. For polynomials of degree 3 and 4 there exist explicit formulas in terms of radicals. As Galois (1811-1832) and Abel (1802-1829) have shown, it is not possible for equations of degree 5 or higher. Still, one can compute the roots numerically.
REAL SOLUTIONS. A (2n + 1) × (2n + 1) matrix A always has a real eigenvalue because the characteristic polynomial p(x) = x5 + ... + det(A) has the property that p(x) goes to ±∞ for x → ±∞. Because there exist values a, b for which p(a) < 0 and p(b) > 0, by the intermediate value theorem, there exists a real x with p(x) = 0. Application: A rotation in 11 dimensional space has all eigenvalues |λ| = 1. The real eigenvalue must have an eigenvalue 1 or −1. EIGENVALUES OF TRANSPOSE. We know (see homework) that the characteristic polynomials of A and AT agree. Therefore A and AT have the same eigenvalues. APPLICATION: MARKOV MATRICES. A matrix A for which each column sums up to 1 is called a Markov matrix. The transpose of a Markov matrix has the eigenvector     1 1 . . . 1     with eigenvalue 1. Therefore: A Markov matrix has an eigenvector v to the eigenvalue 1. This vector v defines an equilibrium point of the Markov process. EXAMPLE. If A = 1/3 1/2 2/3 1/2 . Then [3/7, 4/7] is the equilibrium eigenvector to the eigenvalue 1. BRETSCHERS HOMETOWN. Problem 28 in the book deals with a Markov problem in Andelfingen the hometown of Bretscher, where people shop in two shops. (Andelfingen is a beautiful village at the Thur river in the middle of a ”wine country”). Initially all shop in shop W. After a new shop opens, every week 20 percent switch to the other shop M. Missing something at the new place, every week, 10 percent switch back. This leads to a Markov matrix A = 8/10 1/10 2/10 9/10 . After some time, things will settle down and we will have certain percentage shopping in W and other percentage shopping in M. This is the equilibrium. . MARKOV PROCESS IN PROBABILITY. Assume we have a graph like a network and at each node i, the probability to go from i to j in the next step is [A]ij, where Aij is a Markov matrix. We know from the above result that there is an eigenvector p which satisfies Ap = p. It can be normalized that i pi = 1. The interpretation is that pi is the probability that the walker is on the node p. For example, on a triangle, we can have the probabilities: P(A → B) = 1/2, P(A → C) = 1/4, P(A → A) = 1/4, P(B → A) = 1/3, P(B → B) = 1/6, P(B → C) = 1/2, P(C → A) = 1/2, P(C → B) = 1/3, P(C → C) = 1/6. The corresponding matrix is A =   1/4 1/3 1/2 1/2 1/6 1/3 1/4 1/2 1/6   . In this case, the eigenvector to the eigenvalue 1 is p = [38/107, 36/107, 33/107]T . A B C
CALCULATING EIGENVECTORS Math 21b, O.Knill NOTATION. We often just write 1 instead of the identity matrix 1n. COMPUTING EIGENVALUES. Recall: because λ−A has v in the kernel if λ is an eigenvalue the characteristic polynomial fA(λ) = det(λ − A) = 0 has eigenvalues as roots. 2×2 CASE. Recall: The characteristic polynomial of A = a b c d is fA(λ) = λ2 −(a+d)/2λ+(ad−bc). The eigenvalues are λ± = T/2 ± (T/2)2 − D, where T = a + d is the trace and D = ad − bc is the determinant of A. If (T/2)2 ≥ D, then the eigenvalues are real. Away from that parabola in the (T, D) space, there are two different eigenvalues. The map A contracts volume for |D| < 1. NUMBER OF ROOTS. Recall: There are examples with no real eigenvalue (i.e. rotations). By inspecting the graphs of the polynomials, one can deduce that n × n matrices with odd n always have a real eigenvalue. Also n × n matrixes with even n and a negative determinant always have a real eigenvalue. IF ALL ROOTS ARE REAL. fA(λ) = λn − tr(A)λn−1 + ... + (−1)n det(A) = (λ − λ1)...(λ − λn), we see that i λi = trace(A) and i λi = det(A). HOW TO COMPUTE EIGENVECTORS? Because (λ − A)v = 0, the vector v is in the kernel of λ − A. EIGENVECTORS of a b c d are v± with eigen- value λ±. If c = 0 and d = 0, then 1 0 and 0 1 are eigen- vectors. If c = 0, then the eigenvectors to λ± are v± = λ± − d c . If c = 0 and a = d, then the eigenvectors to a and d are 1 0 and b/(d − a) 1 . ALGEBRAIC MULTIPLICITY. If fA(λ) = (λ−λ0)k g(λ), where g(λ0) = 0, then f has algebraic multiplicity k. The algebraic multiplicity counts the number of times, an eigenvector occurs. EXAMPLE:   1 1 1 0 1 1 0 0 2   has the eigenvalue λ = 1 with algebraic multiplicity 2. GEOMETRIC MULTIPLICITY. The dimension of the eigenspace Eλ of an eigenvalue λ is called the geometric multiplicity of λ. EXAMPLE: the matrix of a shear is 1 1 0 1 . It has the eigenvalue 1 with algebraic multiplicity 2. The kernel of A − 1 = 0 1 0 0 is spanned by 1 0 and the geometric multiplicity is 1. It is different from the algebraic multiplicity. EXAMPLE: The matrix   1 1 1 0 0 1 0 0 1   has eigenvalue 1 with algebraic multiplicity 2 and the eigenvalue 0 with multiplicity 1. Eigenvectors to the eigenvalue λ = 1 are in the kernel of A − 1 which is the kernel of  0 1 1 0 −1 1 0 0 0   and spanned by   1 0 0  . The geometric multiplicity is 1. RELATION BETWEEN ALGEBRAIC AND GEOMETRIC MULTIPLICIY. (Proof later in the course). The geometric multiplicity is smaller or equal than the algebraic multiplicity. PRO MEMORIA. Remember that the geometric mean √ ab of two numbers is smaller or equal to the alge- braic mean (a + b)/2? (This fact is totally∗ unrelated to the above fact and a mere coincidence of expressions, but it helps to remember it). Quite deeply buried there is a connection in terms of convexity. But this is rather philosophical. .
EXAMPLE. What are the algebraic and geometric multiplicities of A =       2 1 0 0 0 0 2 1 0 0 0 0 2 1 0 0 0 0 2 1 0 0 0 0 2       ? SOLUTION. The algebraic multiplicity of the eigenvalue 2 is 5. To get the kernel of A−2, one solves the system of equations x4 = x3 = x2 = x1 = 0 so that the geometric multiplicity of the eigenvalues 2 is 4. CASE: ALL EIGENVALUES ARE DIFFERENT. If all eigenvalues are different, then all eigenvectors are linearly independent and all geometric and alge- braic multiplicities are 1. PROOF. Let λi be an eigenvalue different from 0 and assume the eigenvectors are linearly dependent. We have vi = j=i ajvj and λivi = Avi = A( j=i ajvj) = j=i ajλjvj so that vi = j=i bjvj with bj = ajλj/λi. If the eigenvalues are different, then aj = bj and by subtracting vi = j=i ajvj from vi = j=i bjvj, we get 0 = j=i(bj − aj)vj = 0. Now (n − 1) eigenvectors of the n eigenvectors are linearly dependent. Use induction. CONSEQUENCE. If all eigenvalues of a n × n matrix A are different, there is an eigenbasis, a basis consisting of eigenvectors. EXAMPLE. A = 1 1 0 3 has eigenvalues 1, 3 to the eigenvectors 1 0 −1 1 . EXAMPLE. (See homework problem 40 in the book). Photos of the Swiss lakes in the text. The pollution story is fiction fortunately. The vector An (x)b gives the pollution levels in the three lakes (Silvaplana, Sils, St Moritz) after n weeks, where A =   0.7 0 0 0.1 0.6 0 0 0.2 0.8   and b =   100 0 0   is the initial pollution. 4 6 8 10 20 40 60 80 100 There is an eigenvector e3 = v3 =   0 0 1   to the eigenvalue λ3 = 0.8. There is an eigenvector v2 =   0 1 −1   to the eigenvalue λ2 = 0.6. There is further an eigenvector v1 =   1 1 −2   to the eigenvalue λ1 = 0.7. We know An v1, An v2 and An v3 explicitly. How do we get the explicit solution An b? Because b = 100 · e1 = 100(v1 − v2 + 3v3), we have An (b) = 100An (v1 − v2 + 3v3) = 100(λn 1 v1 − λn 2 v2 + 3λn 3 v3) = 100  0.7n   1 1 −2   + 0.6n   0 1 −1   + 3 · 0.8n   0 0 1     =   100(0.7)n 100(0.7n + 0.6n ) 100(−2 · 0.7n − 0.6n + 3 · 0.8n )  
DIAGONALIZATION Math 21b, O.Knill SUMMARY. A n × n matrix, Av = λv with eigenvalue λ and eigenvector v. The eigenvalues are the roots of the characteristic polynomial fA(λ) = det(λ − A) = λn − tr(A)λn−1 + ... + (−1)n det(A). The eigenvectors to the eigenvalue λ are in ker(λ − A). The number of times, an eigenvalue λ occurs in the full list of n roots of fA(λ) is called algebraic multiplicity. It is bigger or equal than the geometric multiplicity: dim(ker(λ − A). EXAMPLE. The eigenvalues of a b c d are λ± = T/2 + T2/4 − D, where T = a + d is the trace and D = ad − bc is the determinant of A. If c = 0, the eigenvectors are v± = λ± − d c if c = 0, then a, d are eigenvalues to the eigenvectors a 0 and −b a − d . If a = d, then the second eigenvector is parallel to the first and the geometric multiplicity of the eigenvalue a = d is 1. EIGENBASIS. If A has n different eigenvalues, then A has an eigenbasis, consisting of eigenvectors of A. DIAGONALIZATION. How does the matrix A look in an eigenbasis? If S is the matrix with the eigenvectors as columns, then we know B = S−1 AS . We have Sei = vi and ASei = λivi we know S−1 ASei = λiei. Therefore, B is diagonal with diagonal entries λi. EXAMPLE. A = 2 3 1 2 has the eigenvalues λ1 = 2 + √ 3 with eigenvector v1 = [ √ 3, 1] and the eigenvalues λ2 = 2 − √ 3 with eigenvector v2 = [− √ 3, 1]. Form S = √ 3 − √ 3 1 1 and check S−1 AS = D is diagonal. APPLICATION: FUNCTIONAL CALCULUS. Let A be the matrix in the above example. What is A100 + A37 −1? The trick is to diagonalize A: B = S−1 AS, then Bk = S−1 Ak S and We can compute A100 +A37 −1 = S(B100 + B37 − 1)S−1 . APPLICATION: SOLVING LINEAR SYSTEMS. x(t+1) = Ax(t) has the solution x(n) = An x(0). To compute An , we diagonalize A and get x(n) = SBn S−1 x(0). This is an explicit formula. SIMILAR MATRICES HAVE THE SAME EIGENVALUES. One can see this in two ways: 1) If B = S−1 AS and v is an eigenvector of B to the eigenvalue λ, then Sv is an eigenvector of A to the eigenvalue λ. 2) From det(S−1 AS) = det(A), we know that the characteristic polynomials fB(λ) = det(λ − B) = det(λ − S−1 AS) = det(S−1 (λ − AS) = det((λ − A) = fA(λ) are the same. CONSEQUENCES. 1) Because the characteristic polynomials of similar matrices agree, the trace tr(A) of similar matrices agrees. 2) The trace is the sum of the eigenvalues of A. (Compare the trace of A with the trace of the diagonalize matrix.) THE CAYLEY HAMILTON THEOREM. If A is diagonalizable, then fA(A) = 0. PROOF. The DIAGONALIZATION B = S−1 AS has the eigenvalues in the diagonal. So fA(B), which contains fA(λi) in the diagonal is zero. From fA(B) = 0 we get SfA(B)S−1 = fA(A) = 0. By the way: the theorem holds for all matrices: the coefficients of a general matrix can be changed a tiny bit so that all eigenvalues are different. For any such perturbations one has fA(A) = 0. Because the coefficients of fA(A) depend continuously on A, they are zero 0 in general. CRITERIA FOR SIMILARITY. • If A and B have the same characteristic polynomial and diagonalizable, then they are similar. • If A and B have a different determinant or trace, they are not similar. • If A has an eigenvalue which is not an eigenvalue of B, then they are not similar.
WHY DO WE WANT TO DIAGONALIZE? 1) FUNCTIONAL CALCULUS. If p(x) = 1 + x + x2 + x3 /3! + x4 /4! be a polynomial and A is a matrix, then p(A) = 1 + A + A2 /2! + A3 /3! + A4 /4! is a matrix. If B = S−1 AS is diagonal with diagonal entries λi, then p(B) is diagonal with diagonal entries p(λi). And p(A) = S p(B)S−1 . This speeds up the calculation because matrix multiplication costs much. The matrix p(A) can be written down with three matrix multiplications, because p(B) is diagonal. 2) SOLVING LINEAR DIFFERENTIAL EQUATIONS. A differential equation ˙v = Av is solved by v(t) = eAt v(0), where eAt = 1 + At + A2 t2 /2! + A3 t3 /3!...)x(0). (Differentiate this sum with respect to t to get AeAt v(0) = Ax(t).) If we write this in an eigenbasis of A, then y(t) = eBt y(0) with the diagonal matrix B =   λ1 0 . . . 0 0 λ2 . . . 0 0 0 . . . λn  . In other words, we have then explicit solutions yj(t) = eλj t yj(0). Linear differential equations later in this course. It is important motivation. 3) STOCHASTIC DYNAMICS (i.e MARKOV PROCESSES). Complicated systems can be modeled by putting probabilities on each possible event and computing the probabilities that an event switches to any other event. This defines a transition matrix. Such a matrix always has an eigenvalue 1. The corresponding eigenvector is the stable probability distribution on the states. If we want to understand, how fast things settle to this equilibrium, we need to know the other eigenvalues and eigenvectors. MOLECULAR VIBRATIONS. While quantum mechanics describes the motion of atoms in molecules, the vibrations can be described classically, when treating the atoms as ”balls” connected with springs. Such ap- proximations are necessary when dealing with large atoms, where quantum mechanical computations would be too costly. Examples of simple molecules are white phosphorus P4, which has tetrahedronal shape or methan CH4 the simplest organic compound or freon, CF2Cl2 which is used in refrigerants. Caffeine or aspirin form more complicated molecules. Freon CF2Cl2 Caffeine C8H10N4O2 Aspirin C9H8O4 WHITE PHOSPHORUS VIBRATIONS. (Differential equations appear later, the context is motivation at this stage). Let x1, x2, x3, x4 be the positions of the four phosphorus atoms (each of them is a 3-vector). The inter- atomar forces bonding the atoms is modeled by springs. The first atom feels a force x2 − x1 + x3 − x1 + x4 − x1 and is accelerated in the same amount. Let’s just chose units so that the force is equal to the acceleration. Then ¨x1 = (x2 − x1) + (x3 − x1) + (x4 − x1) ¨x2 = (x3 − x2) + (x4 − x2) + (x1 − x2) ¨x3 = (x4 − x3) + (x1 − x3) + (x2 − x3) ¨x4 = (x1 − x4) + (x2 − x4) + (x3 − x4) which has the form ¨x = Ax, where the 4 × 4 ma- trix A =     −3 1 1 1 1 −3 1 1 1 1 −3 1 1 1 1 −3     v1 =     1 1 1 1     , v2 =     −1 0 0 1     , v3 =     −1 0 1 0     , v4 =     −1 1 0 0     are the eigenvectors to the eigenvalues λ1 = 0, λ2 = −4, λ3 = −4, λ3 = −4. With S = [v1v2v3v4], the matrix B = S−1 BS is diagonal with entries 0, −4, −4, −4. The coordinates yi = Sxi satisfy ¨y1 = 0, ¨y2 = −4y2, ¨y3 = −4y3, ¨y4 = −4y4 which we can solve y0 which is the center of mass satisfies y0 = a + bt (move molecule with constant speed). The motions yi = ai cos(2t) + bi sin(2t) of the other eigenvectors are oscillations, called normal modes. The general motion of the molecule is a superposition of these modes.
COMPLEX EIGENVALUES Math 21b, O. Knill NOTATION. Complex numbers are written as z = x + iy = r exp(iφ) = r cos(φ) + ir sin(φ). The real number r = |z| is called the absolute value of z, the value φ is the argument and denoted by arg(z). Complex numbers contain the real numbers z = x+i0 as a subset. One writes Re(z) = x and Im(z) = y if z = x + iy. ARITHMETIC. Complex numbers are added like vectors: x + iy + u + iv = (x + u) + i(y + v) and multiplied as z ∗ w = (x + iy)(u + iv) = xu − yv + i(yu − xv). If z = 0, one can divide 1/z = 1/(x + iy) = (x − iy)/(x2 + y2 ). ABSOLUTE VALUE AND ARGUMENT. The absolute value |z| = x2 + y2 satisfies |zw| = |z| |w|. The argument satisfies arg(zw) = arg(z) + arg(w). These are direct consequences of the polar representation z = r exp(iφ), w = s exp(iψ), zw = rs exp(i(φ + ψ)). GEOMETRIC INTERPRETATION. If z = x + iy is written as a vector x y , then multiplication with an other complex number w is a dilation-rotation: a scaling by |w| and a rotation by arg(w). THE DE MOIVRE FORMULA. zn = exp(inφ) = cos(nφ) + i sin(nφ) = (cos(φ) + i sin(φ))n follows directly from z = exp(iφ) but it is magic: it leads for example to formulas like cos(3φ) = cos(φ)3 − 3 cos(φ) sin2 (φ) which would be more difficult to come by using geometrical or power series arguments. This formula is useful for example in integration problems like cos(x)3 dx, which can be solved by using the above deMoivre formula. THE UNIT CIRCLE. Complex numbers of length 1 have the form z = exp(iφ) and are located on the unit circle. The characteristic polynomial fA(λ) = λ5 − 1 of the matrix       0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0       has all roots on the unit circle. The roots exp(2πki/5), for k = 0, . . . , 4 lye on the unit circle. THE LOGARITHM. log(z) is defined for z = 0 as log |z| + iarg(z). For example, log(2i) = log(2) + iπ/2. Riddle: what is ii ? (ii = ei log(i) = eiiπ/2 = e−π/2 ). The logarithm is not defined at 0 and the imaginary part is define only up to 2π. For example, both iπ/2 and 5iπ/2 are equal to log(i). HISTORY. The struggle with √ −1 is historically quite interesting. Nagging questions appeared for example when trying to find closed solutions for roots of polynomials. Cardano (1501-1576) was one of the mathemati- cians who at least considered complex numbers but called them arithmetic subtleties which were ”as refined as useless”. With Bombelli (1526-1573), complex numbers found some practical use. Descartes (1596-1650) called roots of negative numbers ”imaginary”. Although the fundamental theorem of algebra (below) was still not proved in the 18th century, and complex numbers were not fully understood, the square root of minus one √ −1 was used more and more. Euler (1707- 1783) made the observation that exp(ix) = cos x + i sin x which has as a special case the magic formula eiπ + 1 = 0 which relate the constants 0, 1, π, e in one equation. For decades, many mathematicians still thought complex numbers were a waste of time. Others used complex numbers extensively in their work. In 1620, Girard suggested that an equation may have as many roots as its degree in 1620. Leibniz (1646-1716) spent quite a bit of time trying to apply the laws of algebra to complex numbers. He and Johann Bernoulli used imaginary numbers as integration aids. Lambert used complex numbers for map projections, d’Alembert used them in hydrodynamics, while Euler, D’Alembert and Lagrange used them in their incorrect proofs of the fundamental theorem of algebra. Euler write first the symbol i for √ −1.
Gauss published the first correct proof of the fundamental theorem of algebra in his doctoral thesis, but still claimed in 1825 that the true metaphysics of the square root of −1 is elusive as late as 1825. By 1831 Gauss overcame his uncertainty about complex numbers and published his work on the geometric representation of complex numbers as points in the plane. In 1797, a Norwegian Caspar Wessel (1745-1818) and in 1806 a Swiss clerk named Jean Robert Argand (1768-1822) (who stated the theorem the first time for polynomials with complex coefficients) did similar work. But these efforts went unnoticed. William Rowan Hamilton (1805-1865) (who would also discover the quaternions while walking over a bridge) expressed in 1833 complex numbers as vectors. Complex numbers continued to develop to complex function theory or chaos theory, a branch of dynamical systems theory. Complex numbers are helpful in geometry in number theory or in quantum mechanics. Once believed fictitious they are now most ”natural numbers” and the ”natural numbers” themselves are in fact the most ”complex”. A philospher who asks ”does √ −1 really exist?” might be shown the representation of x + iy as x −y y x . When adding or multiplying such dilation-rotation matrices, they behave like complex numbers: for example 0 −1 1 0 plays the role of i. FUNDAMENTAL THEOREM OF ALGEBRA. (Gauss 1799) A polynomial of degree n has exactly n roots. CONSEQUENCE: A n × n MATRIX HAS n EIGENVALUES. The characteristic polynomial fA(λ) = λn + an−1λn−1 + . . . + a1λ + a0 satisfies fA(λ) = (λ − λ1) . . . (λ − λn), where λi are the roots of f. TRACE AND DETERMINANT. Comparing fA(λ) = (λ − λn) . . . (λ − λn) with λn − tr(A) + .. + (−1)n det(A) gives tr(A) = λ1 + · · · + λn, det(A) = λ1 · · · λn. COMPLEX FUNCTIONS. The characteristic polynomial is an example of a function f from C to C. The graph of this function would live in lC × lC which corresponds to a four dimensional real space. One can visualize the function however with the real-valued function z → |f(z)|. The figure to the left shows the contour lines of such a function z → |f(z)|, where f is a polynomial. ITERATION OF POLYNOMIALS. A topic which is off this course (it would be a course by itself) is the iteration of polynomials like fc(z) = z2 + c. The set of parameter values c for which the iterates fc(0), f2 c (0) = fc(fc(0)), . . . , fn c (0) stay bounded is called the Mandelbrot set. It is the fractal black region in the picture to the left. The now already dusty object appears everywhere, from photoshop plugins to decorations. In Mathematica, you can compute the set very quickly (see http://www.math.harvard.edu/computing/math/mandelbrot.m). COMPLEX NUMBERS IN MATHEMATICA OR MAPLE. In both computer algebra systems, the letter I is used for i = √ −1. In Maple, you can ask log(1 + I), in Mathematica, this would be Log[1 + I]. Eigenvalues or eigenvectors of a matrix will in general involve complex numbers. For example, in Mathematica, Eigenvalues[A] gives the eigenvalues of a matrix A and Eigensystem[A] gives the eigenvalues and the corresponding eigenvectors. EIGENVALUES AND EIGENVECTORS OF A ROTATION. The rotation matrix A = cos(φ) sin(φ) − sin(φ) cos(φ) has the characteristic polynomial λ2 − 2 cos(φ) + 1. The eigenvalues are cos(φ) ± cos2(φ) − 1 = cos(φ) ± i sin(φ) = exp(±iφ). The eigenvector to λ1 = exp(iφ) is v1 = −i 1 and the eigenvector to the eigenvector λ2 = exp(−iφ) is v2 = i 1 .
STABILITY Math 21b, O. Knill LINEAR DYNAMICAL SYSTEM. A linear map x → Ax defines a dynamical system. Iterating the map produces an orbit x0, x1 = Ax, x2 = A2 = AAx, .... The vector xn = An x0 describes the situation of the system at time n. Where does xn go when time evolves? Can one describe what happens asymptotically when time n goes to infinity? In the case of the Fibonacci sequence xn which gives the number of rabbits in a rabbit population at time n, the population grows essentially exponentially. Such a behavior would be called unstable. On the other hand, if A is a rotation, then An v stays bounded which is a type of stability. If A is a dilation with a dilation factor < 1, then An v → 0 for all v, a thing which we will call asymptotic stability. The next pictures show experiments with some orbits An v with different matrices. 0.99 −1 1 0 stable (not asymptotic) 0.54 −1 0.95 0 asymptotic stable 0.99 −1 0.99 0 asymptotic stable 0.54 −1 1.01 0 unstable 2.5 −1 1 0 unstable 1 0.1 0 1 unstable ASYMPTOTIC STABILITY. The origin 0 is invariant under a linear map T(x) = Ax. It is called asymptot- ically stable if An (x) → 0 for all x ∈ IRn . EXAMPLE. Let A = p −q q p be a dilation rotation matrix. Because multiplication wich such a matrix is analogue to the multiplication with a complex number z = p+iq, the matrix An corresponds to a multiplication with (p+iq)n . Since |(p+iq)|n = |p+iq|n , the origin is asymptotically stable if and only if |p+iq| < 1. Because det(A) = |p+iq|2 = |z|2 , rotation-dilation matrices A have an asymptotic stable origin if and only if |det(A)| < 1. Dilation-rotation matrices p −q q p have eigenvalues p ± iq and can be diagonalized in the complex. EXAMPLE. If a matrix A has an eigenvalue |λ| ≥ 1 to an eigenvector v, then An v = λn v, whose length is |λn | times the length of v. So, we have no asymptotic stability if an eigenvalue satisfies |λ| ≥ 1. STABILITY. The book also writes ”stable” for ”asymptotically stable”. This is ok to abbreviate. Note however that the commonly used term ”stable” also includes linear maps like rotations, reflections or the identity. It is therefore preferable to leave the attribute ”asymptotic” in front of ”stable”. ROTATIONS. Rotations cos(φ) − sin(φ) sin(φ) cos(φ) have the eigenvalue exp(±iφ) = cos(φ) + i sin(φ) and are not asymptotically stable. DILATIONS. Dilations r 0 0 r have the eigenvalue r with algebraic and geometric multiplicity 2. Dilations are asymptotically stable if |r| < 1. CRITERION. A linear dynamical system x → Ax has an asymptotically stable origin if and only if all its eigenvalues have an absolute value < 1. PROOF. We have already seen in Example 3, that if one eigenvalue satisfies |λ| > 1, then the origin is not asymptotically stable. If |λi| < 1 for all i and all eigenvalues are different, there is an eigenbasis v1, . . . , vn. Every x can be written as x = n j=1 xjvj. Then, An x = An ( n j=1 xjvj) = n j=1 xjλn j vj and because |λj|n → 0, there is stability. The proof of the general (nondiagonalizable) case will be accessible later.
THE 2-DIMENSIONAL CASE. The characteristic polynomial of a 2 × 2 matrix A = a b c d is fA(λ) = λ2 − tr(A)λ + det(A). If c = 0, the eigenvalues are λ± = tr(A)/2 ± (tr(A)/2)2 − det(A). If the discriminant (tr(A)/2)2 − det(A) is nonnegative, then the eigenvalues are real. This happens below the parabola, where the discriminant is zero. CRITERION. In two dimensions we have asymptotic stability if and only if (tr(A), det(A)) is contained in the stability triangle bounded by the lines det(A) = 1, det(A) = tr(A)−1 and det(A) = −tr(A)−1. PROOF. Write T = tr(A)/2, D = det(A). If |D| ≥ 1, there is no asymptotic stability. If λ = T + √ T2 − D = ±1, then T2 − D = (±1 − T)2 and D = 1 ± 2T. For D ≤ −1 + |2T| we have a real eigenvalue ≥ 1. The conditions for stability is therefore D > |2T| − 1. It implies automatically D > −1 so that the triangle can be described shortly as |tr(A)| − 1 < det(A) < 1 . EXAMPLES. 1) The matrix A = 1 1/2 −1/2 1 has determinant 5/4 and trace 2 and the origin is unstable. It is a dilation- rotation matrix which corresponds to the complex number 1 + i/2 which has an absolute value > 1. 2) A rotation A is never asymptotically stable: det(A)1 and tr(A) = 2 cos(φ). Rotations are the upper side of the stability triangle. 3) A dilation is asymptotically stable if and only if the scaling factor has norm < 1. 4) If det(A) = 1 and tr(A) < 2 then the eigenvalues are on the unit circle and there is no asymptotic stability. 5) If det(A) = −1 (like for example Fibonacci) there is no asymptotic stability. For tr(A) = 0, we are a corner of the stability triangle and the map is a reflection, which is not asymptotically stable neither. SOME PROBLEMS. 1) If A is a matrix with asymptotically stable origin, what is the stability of 0 with respect to AT ? 2) If A is a matrix which has an asymptotically stable origin, what is the stability with respect to to A−1 ? 3) If A is a matrix which has an asymptotically stable origin, what is the stability with respect to to A100 ? ON THE STABILITY QUESTION. For general dynamical systems, the question of stability can be very difficult. We deal here only with linear dynamical systems, where the eigenvalues determine everything. For nonlinear systems, the story is not so simple even for simple maps like the Henon map. The questions go deeper: it is for example not known, whether our solar system is stable. We don’t know whether in some future, one of the planets could get expelled from the solar system (this is a mathematical question because the escape time would be larger than the life time of the sun). For other dynamical systems like the atmosphere of the earth or the stock market, we would really like to know what happens in the near future ... A pioneer in stability theory was Alek- sandr Lyapunov (1857-1918). For nonlin- ear systems like xn+1 = gxn − x3 n − xn−1 the stability of the origin is nontrivial. As with Fibonacci, this can be written as (xn+1, xn) = (gxn − x2 n − xn−1, xn) = A(xn, xn−1) called cubic Henon map in the plane. To the right are orbits in the cases g = 1.5, g = 2.5. The first case is stable (but proving this requires a fancy theory called KAM theory), the second case is unstable (in this case actually the linearization at 0 determines the picture).
SYMMETRIC MATRICES Math 21b, O. Knill SYMMETRIC MATRICES. A matrix A with real entries is symmetric, if AT = A. EXAMPLES. A = 1 2 2 3 is symmetric, A = 1 1 0 3 is not symmetric. EIGENVALUES OF SYMMETRIC MATRICES. Symmetric matrices A have real eigenvalues. PROOF. The dot product is extend to complex vectors as (v, w) = i viwi. For real vectors it satisfies (v, w) = v · w and has the property (Av, w) = (v, AT w) for real matrices A and (λv, w) = λ(v, w) as well as (v, λw) = λ(v, w). Now λ(v, v) = (λv, v) = (Av, v) = (v, AT v) = (v, Av) = (v, λv) = λ(v, v) shows that λ = λ because (v, v) = 0 for v = 0. EXAMPLE. A = p −q q p has eigenvalues p + iq which are real if and only if q = 0. EIGENVECTORS OF SYMMETRIC MATRICES. Symmetric matrices have an orthonormal eigenbasis PROOF. If Av = λv and Aw = µw. The relation λ(v, w) = (λv, w) = (Av, w) = (v, AT w) = (v, Aw) = (v, µw) = µ(v, w) is only possible if (v, w) = 0 if λ = µ. WHY ARE SYMMETRIC MATRICES IMPORTANT? In applications, matrices are often symmetric. For ex- ample in geometry as generalized dot products v·Av, or in statistics as correlation matrices Cov[Xk, Xl] or in quantum mechanics as observables or in neural networks as learning maps x → sign(Wx) or in graph theory as adjacency matrices etc. etc. Symmetric matrices play the same role as real numbers do among the complex numbers. Their eigenvalues often have physical or geometrical interpretations. One can also calculate with symmetric matrices like with numbers: for example, we can solve B2 = A for B if A is symmetric matrix and B is square root of A.) This is not possible in general: try to find a matrix B such that B2 = 0 1 0 0 ... RECALL. We have seen when an eigenbasis exists, a matrix A can be transformed to a diagonal matrix B = S−1 AS, where S = [v1, ..., vn]. The matrices A and B are similar. B is called the diagonalization of A. Similar matrices have the same characteristic polynomial det(B − λ) = det(S−1 (A − λ)S) = det(A − λ) and have therefore the same determinant, trace and eigenvalues. Physicists call the set of eigenvalues also the spectrum. They say that these matrices are isospectral. The spectrum is what you ”see” (etymologically the name origins from the fact that in quantum mechanics the spectrum of radiation can be associated with eigenvalues of matrices.) SPECTRAL THEOREM. Symmetric matrices A can be diagonalized B = S−1 AS with an orthogonal S. PROOF. If all eigenvalues are different, there is an eigenbasis and diagonalization is possible. The eigenvectors are all orthogonal and B = S−1 AS is diagonal containing the eigenvalues. In general, we can change the matrix A to A = A + (C − A)t where C is a matrix with pairwise different eigenvalues. Then the eigenvalues are different for all except finitely many t. The orthogonal matrices St converges for t → 0 to an orthogonal matrix S and S diagonalizes A. WAIT A SECOND ... Why could we not perturb a general matrix At to have disjoint eigenvalues and At could be diagonalized: S−1 t AtSt = Bt? The problem is that St might become singular for t → 0. See problem 5) first practice exam. EXAMPLE 1. The matrix A = a b b a has the eigenvalues a + b, a − b and the eigenvectors v1 = 1 1 / √ 2 and v2 = −1 1 / √ 2. They are orthogonal. The orthogonal matrix S = v1 v2 diagonalized A.
EXAMPLE 2. The 3 × 3 matrix A =   1 1 1 1 1 1 1 1 1   has 2 eigenvalues 0 to the eigenvectors 1 −1 0 , 1 0 −1 and one eigenvalue 3 to the eigenvector 1 1 1 . All these vectors can be made orthogonal and a diagonalization is possible even so the eigenvalues have multiplicities. SQUARE ROOT OF A MATRIX. How do we find a square root of a given symmetric matrix? Because S−1 AS = B is diagonal and we know how to take a square root of the diagonal matrix B, we can form C = S √ BS−1 which satisfies C2 = S √ BS−1 S √ BS−1 = SBS−1 = A. RAYLEIGH FORMULA. We write also (v, w) = v ·w. If v(t) is an eigenvector of length 1 to the eigenvalue λ(t) of a symmetric matrix A(t) which depends on t, differentiation of (A(t) − λ(t))v(t) = 0 with respect to t gives (A −λ )v+(A−λ)v = 0. The symmetry of A−λ implies 0 = (v, (A −λ )v)+(v, (A−λ)v ) = (v, (A −λ )v). We see that the Rayleigh quotient λ = (A v, v) is a polynomial in t if A(t) only involves terms t, t2 , . . . , tm . The formula shows how λ(t) changes, when t varies. For example, A(t) = 1 t2 t2 1 has for t = 2 the eigenvector v = [1, 1]/ √ 2 to the eigenvalue λ = 5. The formula tells that λ (2) = (A (2)v, v) = ( 0 4 4 0 v, v) = 4. Indeed, λ(t) = 1 + t2 has at t = 2 the derivative 2t = 4. EXHIBITION. ”Where do symmetric matrices occur?” Some informal motivation: I) PHYSICS: In quantum mechanics a system is described with a vector v(t) which depends on time t. The evolution is given by the Schroedinger equation ˙v = i¯hLv, where L is a symmetric matrix and ¯h is a small number called the Planck constant. As for any linear differential equation, one has v(t) = ei¯hLt v(0). If v(0) is an eigenvector to the eigenvalue λ, then v(t) = eit¯hλ v(0). Physical observables are given by symmetric matrices too. The matrix L represents the energy. Given v(t), the value of the observable A(t) is v(t) · Av(t). For example, if v is an eigenvector to an eigenvalue λ of the energy matrix L, then the energy of v(t) is λ. This is called the Heisenberg picture. In order that v · A(t)v = v(t) · Av(t) = S(t)v · AS(t)v we have A(t) = S(T)∗ AS(t), where S∗ = ST is the correct generalization of the adjoint to complex matrices. S(t) satisfies S(t)∗ S(t) = 1 which is called unitary and the complex analogue of orthogonal. The matrix A(t) = S(t)∗ AS(t) has the same eigenvalues as A and is similar to A. II) CHEMISTRY. The adjacency matrix A of a graph with n vertices determines the graph: one has Aij = 1 if the two vertices i, j are connected and zero otherwise. The matrix A is symmetric. The eigenvalues λj are real and can be used to analyze the graph. One interesting question is to what extent the eigenvalues determine the graph. In chemistry, one is interested in such problems because it allows to make rough computations of the electron density distribution of molecules. In this so called H¨uckel theory, the molecule is represented as a graph. The eigenvalues λj of that graph approximate the energies an electron on the molecule. The eigenvectors describe the electron density distribution. The Freon molecule for example has 5 atoms. The adjacency matrix is       0 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0       . This matrix A has the eigenvalue 0 with multiplicity 3 (ker(A) is obtained im- mediately from the fact that 4 rows are the same) and the eigenvalues 2, −2. The eigenvector to the eigenvalue ±2 is ±2 1 1 1 1 T . III) STATISTICS. If we have a random vector X = [X1, · · · , Xn] and E[Xk] denotes the expected value of Xk, then [A]kl = E[(Xk − E[Xk])(Xl − E[Xl])] = E[XkXl] − E[Xk]E[Xl] is called the covariance matrix of the random vector X. It is a symmetric n × n matrix. Diagonalizing this matrix B = S−1 AS produces new random variables which are uncorrelated. For example, if X is is the sum of two dice and Y is the value of the second dice then E[X] = [(1 + 1) + (1 + 2) + ... + (6 + 6)]/36 = 7, you throw in average a sum of 7 and E[Y ] = (1 + 2 + ... + 6)/6 = 7/2. The matrix entry A11 = E[X2 ] − E[X]2 = [(1 + 1) + (1 + 2) + ... + (6 + 6)]/36 − 72 = 35/6 known as the variance of X, and A22 = E[Y 2 ] − E[Y ]2 = (12 + 22 + ... + 62 )/6 − (7/2)2 = 35/12 known as the variance of Y and A12 = E[XY ] − E[X]E[Y ] = 35/12. The covariance matrix is the symmetric matrix A = 35/6 35/12 35/12 35/12 .
CONTINUOUS DYNAMICAL SYSTEMS I Math 21b, O. Knill CONTINUOUS DYNAMICAL SYSTMES. A differ- ential equation d dt x = f(x) defines a dynamical sys- tem. The solutions is a curve x(t) which has the velocity vector f(x(t)) for all t. We often write ˙x for d dt x. ONE DIMENSION. A system ˙x = g(x, t) (written in the form ˙x = g(x, t), ˙t = 1) and often has explicit solutions • If ˙x = g(t), then x(t) = t 0 g(t) dt. • If ˙x = h(x), then dx/h(x) = dt and so t = x 0 dx/h(x) = H(x) so that x(t) = H−1 (t). • If ˙x = g(t)/h(x), then H(x) = x 0 h(x) dx = t 0 g(t) dt = G(t) so that x(t) = H−1 (G(t)). In general, we have no closed form solutions in terms of known functions. The solution x(t) = t 0 e−t2 dt of ˙x = e−t2 for example can not be expressed in terms of functions exp, sin, log, √ · etc but it can be solved using Taylor series: because e−t2 = 1 − t2 + t4 /2! − t6 /3! + . . . taking coefficient wise the anti-derivatives gives: x(t) = t − t3 /3 + t4 /(32!) − t7 /(73!) + . . .. HIGHER DIMENSIONS. In higher dimensions, chaos can set in and the dynamical system can become unpredictable. The nonlinear Lorentz system to the right ˙x(t) = 10(y(t) − x(t)), ˙y(t) = −x(t)z(t) + 28x(t) − y(t), ˙z(t) = x(t) ∗ y(t) − 8z(t)/3 shows a ”strange attractor”. Evenso completely deterministic: (from x(0) all the path x(t) is determined), there are observables which can be used as a ran- dom number generator. The Duffing sys- tem ¨x + ˙x.10 − x + x3 − 12 cos(t) = 0 to the left can be written in the form ˙v = f(v) with a vector v = (x, ˙x, t). 1D LINEAR DIFFERENTIAL EQUATIONS. A linear differential equation in one dimension ˙x = λx has the solution is x(t) = eλt x(0) . This differential equation appears • as population models with λ > 0: birth rate of the popolation is proportional to its size. • as a model for radioactive decay with λ < 0: the rate of decay is proportional to the number of atoms. LINEAR DIFFERENTIAL EQUATIONS IN HIGHER DIMENSIONS. Linear dynamical systems have the form ˙x = Ax, where A is a matrix. Note that the origin 0 is an equilibrium point: if x(0) = 0, then x(t) = 0 for all t. The general solution is x(t) = eAt = 1 + At + A2 t2 /2! + . . . because ˙x(t) = A + 2A2 t/2! + . . . = A(1 + At + A2 t2 /2! + . . .) = AeAt = Ax(t). If B = S−1 AS is diagonal with the eigenvalues λj = aj + ibj in the diagonal, then y = S−1 x satisfies y(t) = eBt and therefore yj(t) = eλj t yj(0) = eaj t eibj t yj(0). The solutions in the original coordinates are x(t) = Sy(t). PHASE PORTRAITS. For differential equations ˙x = f(x) in 2D one can draw the vector field x → f(x). The solution x(t) is tangent to the vector f(x(t)) everywhere. The phase portraits together with some solution
-1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 UNDERSTANDING A DIFFERENTIAL EQUATION. The closed form solution like x(t) = eAt x(0) for ˙x = Ax is actually quite useless. One wants to understand the solution quantitatively. Questions one wants to answer are: what happens in the long term? Is the origin stable, are there periodic solutions. Can one decompose the system into simpler subsystems? We will see that diagonalisation allows to understand the system: by decomposing it into one-dimensional linear systems, which can be analyzed seperately. In general, ”understanding” can mean different things: Plotting phase portraits. Computing solutions numerically and esti- mate the error. Finding special solutions. Predicting the shape of some orbits. Finding regions which are invariant. Finding special closed form solutions x(t). Finding a power series x(t) = n antn in t. Finding quantities which are unchanged along the flow (called ”Integrals”). Finding quantities which increase along the flow (called ”Lyapunov functions”). LINEAR STABILITY. A linear dynamical system ˙x = Ax with diagonalizable A is linearly stable if and only if aj = Re(λj) < 0 for all eigenvalues λj of A. PROOF. We see that from the explicit solutions yj(t) = eaj t eibj t yj(0) in the basis consisting of eigenvectors. Now, y(t) → 0 if and only if aj < 0 for all j and x(t) = Sy(t) → 0 if and only if y(t) → 0. RELATION WITH DISCRETE TIME SYSTEMS. From ˙x = Ax, we obtain x(t + 1) = Bx(t), with the matrix B = eA . The eigenvalues of B are µj = eλj . Now |µj| < 1 if and only if Reλj < 0. The criterium for linear stability of discrete dynamical systems is compatible with the criterium for linear stability of ˙x = Ax. EXAMPLE 1. The system ˙x = y, ˙y = −x can in vector form v = (x, y) be written as ˙v = Av, with A = 0 −1 1 0 . The matrix A has the eigenvalues i, −i. After a coordi- nate transformation w = S−1 v we get with w = (a, b) the differential equations ˙a = ia, ˙b = −i = b which has the solutions a(t) = eit a(0), b(t) = e−it b(0). The original coordates satisfy x(t) = cos(t)x(0) − sin(t)y(0), y(t) = sin(t)x(0) + cos(t)y(0). Indeed eAt is a rotation in the plane. -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 EXAMPLE 2. A harmonic oscillator ¨x = −x can be written with y = ˙x as ˙x = y, ˙y = −x (see Example 1). The general solution is therefore x(t) = cos(t)x(0) − sin(t) ˙x(0). EXAMPLE 3. We take two harmonic oscillators and couple them: ¨x1 = −x1 − (x2 − x1), ¨x2 = −x2 + (x2 − x1). For small xi one can simulate this with two coupled penduli. The system can be written as ¨v = Av, with A = −1 + − − −1 + . The matrix A has an eigenvalue λ1 = −1 to the eigenvector 1 1 and an eigenvalue λ2 = −1 + 2 ∗ to the eigenvector 1 −1 . The coordinate change S is S = 1 1 1 −1 . It has the inverse S−1 = 1 1 −1 1 /2. In the coordinates w = S−1 v = [y1, y2], we have oscillations ¨y1 = −y1 corresponding to the case x1 − x2 = 0 (the pendula swing synchroneous) and ¨y2 = −(1 − 2 )y2 corresponding to x1 + x2 = 0 (the pendula swing against each other).
CONTINUOUS DYNAMICAL SYSTEMS II Math 21b, O. Knill COMPLEX LINEAR 1D CASE. ˙x = λx for λ = a + ib has solution x(t) = eat eibt x(0) and norm ||x(t)|| = eat ||x(0)||. OSCILLATOR: The system ¨x = −λx has the solution x(t) = cos( √ λt)x(0) + sin( √ λt) ˙x(0)/ √ λ. DERIVATION. ˙x = y, ˙y = −λx and in matrix form as ˙x ˙y = 0 −1 λ 0 x y = A x y and because A has eigenvalues ±i √ λ, the new coordinates move as a(t) = ei √ λt a(0) and b(t) = e−i √ λt b(0). Writing this in the original coordinaes x(t) y(t) = S a(t) b(t) and fixing the constants gives x(t), y(t). EXAMPLE. THE SPINNER. The spinner is a rigid body attached to a spring aligned around the z-axes. The body can rotate around the z-axes and bounce up and down. The two motions are coupled in the following way: when the spinner winds up in the same direction as the spring, the spring gets tightend and the body gets a lift. If the spinner winds up to the other direction, the spring becomes more relaxed and the body is lowered. Instead of reducing the system to a 4D first order system, system d dt x = Ax, we will keep the second time derivative and diagonalize the 2D system d2 dt2 x = Ax, where we know how to solve the one dimensional case d2 dt2 v = −λv as v(t) = A cos( √ λt) + B sin( √ λt) with constants A, B depending on the initial conditions, v(0), ˙v(0). SETTING UP THE DIFFERENTIAL EQUATION. x is the angle and y the height of the body. We put the coordinate system so that y = 0 is the point, where the body stays at rest if x = 0. We assume that if the spring is winded up with an an- gle x, this produces an upwards force x and a momentum force −3x. We furthermore assume that if the body is at position y, then this produces a momentum y onto the body and an upwards force y. The differential equations ¨x = −3x + y ¨y = −y + x can be written as ¨v = Av = −3 1 1 −1 v. FINDING GOOD COORDINATES w = S−1 v is obtained with getting the eigenvalues and eigenvectors of A: λ1 = −2 − √ 2, λ2 = −2 + √ 2 v1 = −1 − √ 2 1 , v1 = −1 + √ 2 1 so that S = −1 − √ 2 −1 + √ 2 1 1 . SOLVE THE SYSTEM ¨a = λ1a,¨b = λ2b IN THE GOOD COORDINATES a b = S−1 x y . a(t) = A cos(ω1t) + B sin(ω1)t, ω1 = √ −λ1, b(t) = C cos(ω2t) + D sin(ω2)t, ω2 = √ −λ2. THE SOLUTION IN THE ORIGINAL COORDINATES. x(t) y(t) = S a(t) b(t) . At t = 0 we know x(0), y(0), ˙x(0), ˙y(0). This fixes the constants in x(t) = A1 cos(ω1t) + B1 sin(ω1t) +A2 cos(ω2t) + B2 sin(ω2t). The curve (x(t), y(t)) traces a Lyssajoux curve: -0.6 -0.4 -0.2 0.2 0.4 0.6 x[t] -1 -0.5 0.5 1 y[t] ASYMPTOTIC STABILITY. A linear system ˙x = Ax in the 2D plane is asymptotically stable if and only if det(A) > 0 and tr(A) < 0. PROOF. If the eigenvalues λ1, λ2 of A are real then both beeing negative is equivalent with λ1λ2 = det(A) > 0 and tr(A) = λ1 + λ2 < 0. If λ1 = a + ib, λ2 = a − ib, then a negative a is equivalent to λ1 + λ2 = 2a < 0 and λ1λ2 = a2 + b2 > 0.
ASYMPTOTIC STABILITY COMPARISON OF DISCRETE AND CONTINUOUS SITUATION. The trace and the determimant are independent of the basis, they can be computed fast, and are real if A is real. It is therefore convenient to determine the region in the tr − det-plane, where continuous or discrete dynamical systems are asymptotically stable. While the continuous dynamical system is related to a discrete system, it is important not to mix these two situations up. Continuous dynamical system. Discrete dynamical system. Stability of ˙x = Ax (x(t + 1) = eA x(t)). Stability of x(t + 1) = Ax Stability in det(A) > 0, tr(A) > 0 Stability in |tr(A)| − 1 < det(A) < 1 Stability if Re(λ1) < 0, Re(λ2) < 0. Stability if |λ1| < 1,|λ2| < 1. PHASEPORTRAITS. (In two dimensions we can plot the vector field, draw some trajectories) -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 < 0 λ2 < 0, i.e A = −2 0 0 −3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 < 0 λ2 > 0, i.e A = −2 0 0 3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 > 0 λ2 > 0, i.e A = 2 0 0 3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = 0 λ2 < 0, i.e A = 0 0 0 −3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = 0 λ2 = 0, i.e A = 0 0 1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = a + ib, a < 0 λ2 = a − ib, i.e A = −1 1 −1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = a + ib, a > 0 λ2 = a − ib, i.e A = 1 1 −1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = ib λ2 = −ib, i.e A = 0 1 −1 0
NONLINEAR DYNAMICAL SYSTEMS Math 21b, O. Knill SUMMARY. For linear systems ˙x = Ax, the eigenvalues of A determine the behavior completely. For nonlinear systems explicit formulas for solutions are no more available in general. It even also happen that orbits go go off to infinity in finite time like in ˙x = x2 with solution x(t) = −1/(t − x(0)). With x(0) = 1 it reaches infinity at time t = 1. Linearity is often too crude. The exponential growth ˙x = ax of a bacteria colony for example is slowed down due to lack of food and the logistic model ˙x = ax(1 − x/M) would be more accurate, where M is the population size for which bacteria starve so much that the growth has stopped: x(t) = M, then ˙x(t) = 0. Nonlinear systems can be investigated with qualitative methods. In 2 dimensions ˙x = f(x, y), ˙y = g(x, y), where chaos does not happen, the analysis of equilibrium points and linear approximation at those points in general allows to understand the system. Also in higher dimensions, where ODE’s can have chaotic solutions, the analysis of equilibrium points and linear approximation at those points is a place, where linear algebra becomes useful. EQUILIBRIUM POINTS. A point x0 is called an equilibrium point of ˙x = f(x) if f(x0) = 0. If x(0) = x0 then x(t) = x0 for all times. The system ˙x = x(6−2x−y), ˙y = y(4−x−y) for example has the four equilibrium points (0, 0), (3, 0), (0, 4), (2, 2). JACOBIAN MATRIX. If x0 is an equilibrium point for ˙x = f(x) then [A]ij = ∂ ∂xj fi(x) is called the Jacobian at x0. For two dimensional systems ˙x = f(x, y) ˙y = g(x, y) this is the 2 × 2 matrix A= ∂f ∂x (x, y) ∂f ∂y (x, y) ∂g ∂x (x, y) ∂g ∂y (x, y) . The linear ODE ˙y = Ay with y = x − x0 approximates the nonlinear system well near the equilibrium point. The Jacobian is the linear approximation of F = (f, g) near x0. VECTOR FIELD. In two dimensions, we can draw the vector field by hand: attaching a vector (f(x, y), g(x, y)) at each point (x, y). To find the equilibrium points, it helps to draw the nullclines {f(x, y) = 0}, {g(x, y) = 0}. The equilibrium points are located on intersections of nullclines. The eigenvalues of the Jacobeans at equilibrium points allow to draw the vector field near equilibrium points. This information is sometimes enough to draw the vector field by hand. MURRAY SYSTEM (see handout) ˙x = x(6 − 2x − y), ˙y = y(4 − x − y) has the nullclines x = 0, y = 0, 2x + y = 6, x + y = 5. There are 4 equilibrium points (0, 0), (3, 0), (0, 4), (2, 2). The Jacobian matrix of the system at the point (x0, y0) is 6 − 4x0 − y0 −x0 −y0 4 − x0 − 2y0 . Note that without interaction, the two systems would be logistic systems ˙x = x(6 − 2x), ˙y = y(4 − y). The additional −xy is the competition. Equilibrium Jacobean Eigenvalues Nature of equilibrium (0,0) 6 0 0 4 λ1 = 6, λ2 = 4 Unstable source (3,0) −6 −3 0 1 λ1 = −6, λ2 = 1 Hyperbolic saddle (0,4) 2 0 −4 −4 λ1 = 2, λ2 = −4 Hyperbolic saddle (2,2) −4 −2 −2 −2 λi = −3 ± √ 5 Stable sink USING TECHNOLOGY (Example: Mathematica). Plot the vector field: Needs["Graphics‘PlotField‘"] f[x_,y_]:={x(6-2x-y),y(5-x-y)};PlotVectorField[f[x,y],{x,0,4},{y,0,4}] Find the equilibrium solutions: Solve[{x(6-2x-y)==0,y(5-x-y)==0},{x,y}] Find the Jacobian and its eigenvalues at (2, 2): A[{x_,y_}]:={{6-4x,-x},{-y,5-x-2y}};Eigenvalues[A[{2,2}]] Plotting an orbit: S[u_,v_]:=NDSolve[{x’[t]==x[t](6-2x[t]-y[t]),y’[t]==y[t](5-x[t]-y[t]),x[0]==u,y[0]==v},{x,y},{t,0,1}] ParametricPlot[Evaluate[{x[t],y[t]}/.S[0.3,0.5]],{t,0,1},AspectRatio->1,AxesLabel->{"x[t]","y[t]"}]
VOLTERRA-LODKA SYSTEMS are systems of the form ˙x = 0.4x − 0.4xy ˙y = −0.1y + 0.2xy This example has equilibrium points (0, 0) and (1/2, 1). It describes for example a tuna shark population. The tuna pop- ulation x(t) becomes smaller with more sharks. The shark population grows with more tuna. Volterra ex- plained so first the oscillation of fish populations in the Mediterrian sea. EXAMPLE: HAMILTONIAN SYS- TEMS are systems of the form ˙x = ∂yH(x, y) ˙y = −∂xH(x, y) where H is called the energy. Usu- ally, x is the position and y the mo- mentum. THE PENDULUM: H(x, y) = y2 /2 − cos(x). ˙x = y ˙y = − sin(x) x is the angle between the pendulum and y-axes, y is the angular velocity, sin(x) is the potential. (See homework). Hamiltonian systems preserve energy H(x, y) because d dt H(x(t), y(t)) = ∂xH(x, y) ˙x + ∂yH(x, y) ˙y = ∂xH(x, y)∂yH(x, y) − ∂yH(x, y)∂xH(x, y) = 0. Orbits stay on level curves of H. EXAMPLE: LIENHARD SYSTEMS are differential equations of the form ¨x + ˙xF (x) + G (x) = 0. With y = ˙x + F(x), G (x) = g(x), this gives ˙x = y − F(x) ˙y = −g(x) VAN DER POL EQUATION ¨x+(x2 − 1) ˙x + x = 0 appears in electrical engineering, biology or biochemistry. Since F(x) = x3 /3 − x, g(x) = x. ˙x = y − (x3 /3 − x) ˙y = −x Lienhard systems have limit cycles. A trajectory always ends up on that limit cycle. This is useful for engineers, who need oscillators which are stable under changes of parameters. One knows: if g(x) > 0 for x > 0 and F has exactly three zeros 0, a, −a, F (0) < 0 and F (x) ≥ 0 for x > a and F(x) → ∞ for x → ∞, then the corresponding Lienhard system has exactly one stable limit cycle. CHAOS can occur for systems ˙x = f(x) in three dimensions. For example, ¨x = f(x, t) can be written with (x, y, z) = (x, ˙x, t) as ( ˙x, ˙y, ˙z) = (y, f(x, z), 1). The system ¨x = f(x, ˙x) becomes in the coordinates (x, ˙x) the ODE ˙x = f(x) in four dimensions. The term chaos has no uniform definition, but usually means that one can find a copy of a random number generator embedded inside the system. Chaos theory is more than 100 years old. Basic insight had been obtained by Poincar´e. During the last 30 years, the subject exploded to its own branch of physics, partly due to the availability of computers. ROESSLER SYSTEM ˙x = −(y + z) ˙y = x + y/5 ˙z = 1/5 + xz − 5.7z LORENTZ SYSTEM ˙x = 10(y − x) ˙y = −xz + 28x − y ˙z = xy − 8z 3 These two systems are examples, where one can observe strange attractors. THE DUFFING SYSTEM ¨x + ˙x 10 − x + x3 − 12 cos(t) = 0 ˙x = y ˙y = −y/10 − x + x3 − 12 cos(z) ˙z = 1 The Duffing system models a metal- lic plate between magnets. Other chaotic examples can be obtained from mechanics like the driven pendulum ¨x + sin(x) − cos(t) = 0.
FUNCTION SPACES/LINEAR MAPS, Math 21b, O. Knill FROM VECTORS TO FUNCTIONS. Vectors can be displayed in different ways:        1 2 5 3 2 6         1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 Listing the (i, vi) can also be interpreted as the graph of a function f : 1, 2, 3, 4, 5, 6 → R, where f(i) = vi. LINEAR SPACES. A space X in which we can add, scalar multiplications and where basic laws like commutativity, distributivity and associativity hold is called a linear space. Examples: • Lines, planes and more generally, the n-dimensional Euclidean space. • Pn, the space of all polynomials of degree n. • The space P of all polynomials. • C∞ , the space of all smooth functions on the line • C0 , the space of all continuous functions on the line. • C1 , the space of all differentiable functions on the line. • C∞ (R3 ) the space of all smooth functions in space. • L2 the space of all functions on the line for which f2 is integrable and ∞ ∞ f2 (x) dx < ∞. In all these function spaces, the function f(x) = 0 which is constantly 0 is the zero function. WHICH OF THE FOLLOWING ARE LINEAR SPACES? The space X of all polynomials of the form f(x) = ax3 + bx4 + cx5 The space X of all continuous functions on the unit interval [−1, 1] which vanish at −1 and 1. It contains for example f(x) = x2 − |x|. The space X of all smooth periodic functions f(x+1) = f(x). Example f(x) = sin(2πx)+ cos(6πx). The space X = sin(x) + C∞ (R) of all smooth functions f(x) = sin(x) + g, where g is a smooth function. The space X of all trigonometric polynomials f(x) = a0 + a1 sin(x) + a2 sin(2x) + ... + an sin(nx). The space X of all smooth functions on R which satisfy f(1) = 1. It contains for example f(x) = 1 + sin(x) + x. The space X of all continuous functions on R which satisfy f(2) = 0 and f(10) = 0. The space X of all smooth functions on R which satisfy lim|x|→∞ f(x) = 0. The space X of all continuous functions on R which satisfy lim|x|→∞ f(x) = 1. The space X of all smooth functions on R of compact support: for every f, there exists an interval I such that f(x) = 0 outside that interval. The space X of all smooth functions on R2 .
LINEAR TRANSFORMATIONS. A map T between linear spaces is called lin- ear if T(x + y) = T(x) + T(y), T(λx) = λT(x), T(0) = 0. Examples: • Df(x)=f (x) on C∞ • Tf(x) = x 0 f(x) dx on C0 • Tf(x) = (f(0), f(1), f(2), f(3)) on C∞ . • Tf(x) = sin(x)f(x) on C∞ • Tf(x) = ( 1 0 f(x)g(x) dx)g(x) on C0 [0, 1]. WHICH OF THE FOLLOWING MAPS ARE LINEAR TRANSFORMATIONS? The map T(f) = f (x) on X = C∞ (T). The map T(f) = 1 + f (x) on X = C∞ (R). The map T(f)(x) = sin(x)f(x) on X = C∞ (R). The map T(f)(x) = f(x)/x on X = C∞ (R). The map T(f)(x) = x 0 f(x) dx on X = C([0, 1]). The map T(f)(x) = f(x + √ 2) on X = C∞ (R). The map T(f)(x) = ∞ ∞ f(x − s) sin(s) ds on C0 . The map T(f)(x) = f (x) + f(2) on C∞ (R). The map T(f)(x) = f (x) + f(2) + 1 on C∞ (R). The map T(f)(x, y, z) = fxx(x, y, z) + fyy(x, y, z) + fzz(x, y, z) + 1/(|x|)f(x, y, z) The map T(f)(x) = f(x2 ). The map T(f)(x) = f (x) − x2 f(x). The map T(f)(x) = f2 (x) on C∞ (R). EIGENVALUES, BASIS, KERNEL, IMAGE. Many concepts work also here. X linear space f, g ∈ X, f + g ∈ X, λf ∈ X, 0 ∈ X. T linear transformation T(f + g) = T(f) + T(g), T(λf) = λT(f), T(0) = 0. f1, f2, ..., fn linear independent i cifi = 0 implies fi = 0. f1, f2, ..., fn span X Every f is of the form i cifi. f1, f2, ..., fn basis of X linear independent and span. T has eigenvalue λ Tf = λf kernel of T {Tf = 0} image of T {Tf |f ∈ X}. Some concepts do not work without modification. Example: det(T) or tr(T) are not always defined for linear transformations in infinite dimensions. The concept of a basis in infinite dimensions has to be defined properly. INNER PRODUCT. The analogue of the dot product i figi for vectors (f1, f2, ..., fn), (g1, g2, ..., gn) is the integral 1 0 f(x)g(x) dx for functions on the interval [0, 1]. One writes (f, g) and ||f|| = (f, f). This inner product is defined on L2 or C∞ ([0, 1]). Example: (x3 , x2 ) = 1 0 x5 dx = 1/5. Having an inner product allows to define ”length”, ”angle” in infinite dimensions. One can use it to define projections, Gram-Schmidt orthogonalization etc.
DIFFERENTIAL EQUATIONS, Math 21b, O. Knill LINEAR DIFFERENTIAL EQUATIONS WITH CONSTANT COEFFICIENTS. Df = Tf = f is a linear map on the space of smooth functions C∞ . If p(x) = a0 + a1x + ... + anxn is a polynomial, then p(D) = a0 + a1D + ... + anDn is a linear map on C∞ (R) too. We will see here how to find the general solution of p(D)f = g. EXAMPLE. For p(x) = x2 − x + 6 and g(x) = cos(x) the problem p(D)f = g is the differential equation f (x) − f (x) − 6f(x) = cos(x). It has the solution c1e−2x + c2e3x − (sin(x) + 7 cos(x))/50, where c1, c2 are arbitrary constants. How do we find these solutions? THE IDEA. In general, a differential equation p(D)f = g has many solution. For example, for p(D) = D3 , the equation D3 f = 0 has solutions (c0 +c1x+c2x2 ). The constants come from integrating three times. Integrating means applying D−1 but since D has as a kernel the constant functions, integration gives a one dimensional space of anti-derivatives (we can add a constant to the result and still have an anti-derivative). In order to solve D3 f = g, we integrate g three times. We will generalize this idea by writing T = p(D) as a product of simpler transformations which we can invert. These simpler transformations have the form (D − λ)f = g. FINDING THE KERNEL OF A POLYNOMIAL IN D. How do we find a basis for the kernel of Tf = f +2f +f? The linear map T can be written as a polynomial in D which means T = D2 − D − 2 = (D + 1)(D − 2). The kernel of T contains the kernel of D − 2 which is one-dimensional and spanned by f1 = e2x . The kernel of T = (D − 2)(D + 1) also contains the kernel of D + 1 which is spanned by f2 = e−x . The kernel of T is therefore two dimensional and spanned by e2x and e−x . THEOREM: If T = p(D) = Dn + an−1Dn−1 + ... + a1D + a0 on C∞ then dim(ker(T)) = n. PROOF. T = p(D) = (D − λj), where λj are the roots of the polynomial p. The kernel of T contains the kernel of D − λj which is spanned by fj(t) = eλj t . In the case when we have a factor (D − λj)k of T, then we have to consider the kernel of (D − λj)k which is q(t)eλt , where q is a polynomial of degree k − 1. For example, the kernel of (D − 1)3 consists of all functions (a + bt + ct2 )et . SECOND PROOF. Write this as A˙g = 0, where A is a n × n matrix and g = f, ˙f, · · · , f(n−1) T , where f(k) = Dk f is the k’th derivative. The linear map T = AD acts on vectors of functions. If all eigenvalues λj of A are different (they are the same λj as before), then A can be diagonalized. Solving the diagonal case BD = 0 is easy. It has a n dimensional kernel of vectors F = [f1, . . . , fn]T , where fi(t) = t. If B = SAS−1 , and F is in the kernel of BD, then SF is in the kernel of AD. REMARK. The result can be generalized to the case, when aj are functions of x. Especially, Tf = g has a solution, when T is of the above form. It is important that the function in front of the highest power Dn is bounded away from 0 for all t. For example xDf(x) = ex has no solution in C∞ , because we can not integrate ex /x. An example of a ODE with variable coefficients is the Sturm-Liouville eigenvalue problem T(f)(x) = a(x)f (x) + a (x)f (x) + q(x)f(x) = λf(x) like for example the Legendre differential equation (1 − x2 )f (x) − 2xf (x) + n(n + 1)f(x) = 0. BACKUP • Equations Tf = 0, where T = p(D) form linear differential equations with constant coefficients for which we want to understand the solution space. Such equations are called homogeneous. Solving differential equations often means finding a basis of the kernel of T. In the above example, a general solution of f + 2f + f = 0 can be written as f(t) = a1f1(t) + a2f2(t). If we fix two values like f(0), f (0) or f(0), f(1), the solution is unique. • If we want to solve Tf = g, an inhomogeneous equation then T −1 is not unique because we have a kernel. If g is in the image of T there is at least one solution f. The general solution is then f + ker(T). For example, for T = D2 , which has C∞ as its image, we can find a solution to D2 f = t3 by integrating twice: f(t) = t5 /20. The kernel of T consists of all linear functions at+b. The general solution to D2 = t3 is at + b + t5 /20. The integration constants parameterize actually the kernel of a linear map.
THE SYSTEM Tf = (D − λ)f = g has the general solution ceλx + eλx x 0 e−λt g(t) dt . The solution f = (D − λ)−1 g is the sum of a function in the kernel and a special function. THE SOLUTION OF (D − λ)k f = g is obtained by applying (D − λ)−1 several times on g. In particular, for g = 0, we get: the kernel of (D − λ)k as (c0 + c1x + ... + ck−1xk−1 )eλx . THEOREM. The inhomogeneous p(D)f = g has an n-dimensional space of solutions in C∞ (R). PROOF. To solve Tf = p(D)f = g, we write the equation as (D − λ1)k1 (D − λ2)k2 · · · (D − λn)kn f = g. Since we know how to invert each Tj = (D − λj)kj , we can construct the general solution by inverting one factor Tj of T one after another. Often we can find directly a special solution f1 of p(D)f = g and get the general solution as f1 + fh, where fh is in the n-dimensional kernel of T. EXAMPLE 1) Tf = e3x , where T = D2 − D = D(D − 1). We first solve (D − 1)f = e3x . It has the solution f1 = cex + ex x 0 e−t e3t dt = c2ex + e3x /2. Now solve Df = f1. It has the solution c1 + c2ex + e3x /6 . EXAMPLE 2) Tf = sin(x) with T = (D2 − 2D + 1) = (D − 1)2 . We see that cos(x)/2 is a special solution. The kernel of T = (D − 1)2 is spanned by xex and ex so that the general solution is (c1 + c2x)ex + cos(x)/2 . EXAMPLE 3) Tf = x with T = D2 +1 = (D−i)(D+i) has the special solution f(x) = x. The kernel is spanned by eix and e−ix or also by cos(x), sin(x). The general solution can be written as c1 cos(x) + c2 sin(x) + x . EXAMPLE 4) Tf = x with T = D4 + 2D2 + 1 = (D − i)2 (D + i)2 has the special solution f(x) = x. The kernel is spanned by eix , xeix , e−ix , x−ix or also by cos(x), sin(x), x cos(x), x sin(x). The general solution can be written as (c0 + c1x) cos(x) + (d0 + d1x) sin(x) + x . THESE EXAMPLES FORM 4 TYPICAL CASES. CASE 1) p(D) = (D − λ1)...(D − λn) with real λi. The general solution of p(D)f = g is the sum of a special solution and c1eλ1x + ... + cneλnx CASE 2) p(D) = (D − λ)k . The general solution is the sum of a special solution and a term (c0 + c1x + ... + ck−1xk−1 )eλx CASE 3) p(D) = (D − λ)(D − λ) with λ = a + ib. The general solution is a sum of a special solution and a term c1eax cos(bx) + c2eax sin(bx) CASE 4) p(D) = (D − λ)k (D − λ)k with λ = a + ib. The general solution is a sum of a special solution and (c0 + c1x + ... + ck−1xk−1 )eax cos(bx) + (d0 + d1x + ... + dk−1xk−1 )eax sin(bx) We know this also from the eigenvalue problem for a matrix. We either have distinct real eigenvalues, or we have some eigenvalues with multiplicity, or we have pairs of complex conjugate eigenvalues which are distinct, or we have pairs of complex conjugate eigenvalues with some multiplicity. CAS SOLUTION OF ODE’s: Example: DSolve[f [x] − f [x] == Exp[3x], f[x], x] REMARK. (informal) Operator methods are also useful for ODEs with variable coefficients. For ex- ample, T = H − 1 = D2 − x2 − 1, the quantum harmonic oscillator, can be written as T = A∗ A = AA∗ + 2 with a creation operator A∗ = (D − x) and annihilation operator A = (D + x). (Hint: use the commutation relation Dx − xD = 1.) The kernel f0 = Ce−x2 /2 of A = (D + x) is also the kernel of T and so an eigenvector of T and H. It is called the vacuum. If f is an eigenvector of H with Hf = λf, then A∗ f is an eigenvector with eigenvalue λ + 2 . Proof. Because HA∗ − A∗ H = [H, A∗ ] = 2A∗ , we have H(A∗ f) = A∗ Hf + [H, A∗ ]f = A∗ λf + 2A∗ f = (λ + 2)(A∗ f). We obtain all eigenvectors fn = A∗ fn−1 of eigenvalue 1 + 2n by applying iteratively the creation operator A∗ on the vacuum f0. Because every function f with f2 dx < ∞ can be written uniquely as f = ∞ n=0 anfn, we can diagonalize H and solve Hf = g with f = n bn/(1 + 2n)fn, where g = n bnfn.
FOURIER SERIES, Math 21b, O. Knill HOW DOES IT WORK? Piecewise smooth functions f(x) on [−π, π] form a linear space. The numbers ck = 1 2π π −π f(x)e−ikx dx are called the Fourier coefficients of f. We can recover f from these coefficients by f(x) = k ckeikx . When we write f = feven + fodd, where feven(x) = (f(x) + f(−x))/2 and fodd(x) = ((f(x) − f(−x))/2, then feven(x) = a0/ √ 2 + k=1 ak cos(kx) and fodd(x) = ∞ k=1 bk sin(kx). The sum of the cos-series for feven and the sin-series for fodd up gives the real Fourier series of f. Those coefficients are ak = 1 π π −π f(x) cos(kx) dx, a0 = 1 π π −π f(x)/ √ 2 dx, bk = 1 π π −π f(x) sin(kx) dx. While the complex expansion is more elegant, the real expansion has the advantage that one does not leave the real numbers. ck = 1 2π π −π f(x)e−ikx dx. f(x) = ∞ k=−∞ ckeikx . SUMMARY. a0 = 1 π π −π f(x)/ √ 2 dx ak = 1 π π −π f(x) cos(kx) dx bk = 1 π π −π f(x) sin(kx) dx f(x) = a0√ 2 + ∞ k=1 ak cos(kx) + ∞ k=1 bk sin(kx) EXAMPLE 1. Let f(x) = x on [−π, π]. This is an odd function (f(−x)+f(x) = 0) so that it has a sin series: with bk = 1 π π −π x sin(kx) dx = 1 π (x cos(kx)/k + sin(kx)/k2 |π −π) = 2(−1)k+1 /k we get x = ∞ k=1 2(−1)k+1 k sin(kx). For example, π/2 = 2(1 − 1/3 + 1/5 − 1/7...) a formula of Leibnitz. EXAMPLE 2. Let f(x) = cos(x)+1/7 cos(5x). This trigonometric polynomial is already the Fourier series. The nonzero coefficients are a1 = 1, a5 = 1/7. EXAMPLE 3. Let f(x) = |x| on [−π, π]. This is an even function f(−x) − f(x) = 0 so that it has a cos series: with a0 = 1/(2 √ 2), ak = 1 π π −π |x| cos(kx) dx = 2 π π 0 x cos(kx) dx = 2 π (+x sin(kx)/k + cos(kx)/k2 |π 0 ) = 2 π ((−1)k − 1)/k2 for k > 0. WHERE ARE FOURIER SERIES USEFUL? Examples: • Partial differential equations. PDE’s like ¨u = c2 u become ODE’s: ¨uk = c2 k2 uk can be solved as uk(t) = ak sin(ckt)uk(0) and lead to solutions u(t, x) = k ak sin(ckt) sin(kx) of the PDE. • Sound Coefficients ak form the frequency spec- trum of a sound f. Filters suppress frequencies, equalizers transform the Fourier space, compres- sors (i.e.MP3) select frequencies relevant to the ear. • Analysis: k ak sin(kx) = f(x) give explicit expres- sions for sums which would be hard to evaluate other- wise. The Leibnitz sum π/4 = 1−1/3+1/5−1/7+... (Example 1 above) is an example. • Number theory: Example: if α is irrational, then the fact that nα (mod1) are uniformly distributed in [0, 1] can be understood with Fourier theory. • Chaos theory: Quite many notions in Chaos the- ory can be defined or analyzed using Fourier theory. Examples are mixing properties or ergodicity. • Quantum dynamics: Transport properties of mate- rials are related to spectral questions for their Hamil- tonians. The relation is given by Fourier theory. • Crystallography: X ray Diffraction patterns of a crystal, analyzed using Fourier theory reveal the structure of the crystal. • Probability theory: The Fourier transform χX = E[eiX ] of a random variable is called characteristic function. Independent case: χx+y = χxχy. • Image formats: like JPG compress by cutting irrel- evant parts in Fourier space. WHY DOES IT WORK? One has a dot product on functions with f · g = 1 2π π −π f(x)g(x) dx. The functions ek = eikx serve as basis vectors. As in Euclidean space, where v · ek = vk is the k’th coordinate of v, the Fourier coefficient ck = ek · f = 1 2π π −π e−ikx f(x) dx is the k-th coordinate of f. In the same way as we wrote v = k vkek in Euclidean space, we have f(x) = k ckek(x). The vectors ek are orthonormal: 1 2π π −π ek(x)em(x) dx = 1 2π π −π e(m−k)ix dx which is zero for m = k and 1 for m = k. Also the functions 1/ √ 2, sin(nx), cos(nx) form an orthonormal basis with respect to the dot product (f, g) = 1 π π −π f(x)g(x) dx. The Fourier coefficients a0, an, bn are the coordinates of f in that basis. Fourier basis is not the only one. An other basis which has many applications are Wavelet expansions. The principle is the same.
LENGTH, DISTANCE AND ANGLE. With a dot product, one has a length ||f||2 = f · f = 1 2π π −π |f(x)|2 dx and can measure distances between two functions as ||f − g|| and angles between functions f, g by cos(α) = f·g/(||f||||g||). Functions are called orthogonal if (f, g) = f·g = 0. For example, an even function f and an odd function g are orthogonal. The functions einx form an orthonormal family. The vectors √ 2 cos(kx), √ 2 sin(kx) form an orthonormal family, cos(kx) is in the linear space of even functions and sin(kx) in the linear space of odd functions. If we work with sin or cos Fourier series, we use the dot product f · g = 1 π π −π f(x)g(x) dx for which the functions 1/ √ 2, cos(kx), sin(kx) are orthonormal. REWRITING THE DOT PRODUCT. If f(x) = k ckeikx and g(x) = k dkeikx , then 1 2π π −π f(x)g(x) dx = 1 2π k,m cke−ikx dmeimx = k ckdk. The dot product is the sum of the product of the coordinates as in finite dimensions. If f and g are even, then f = f0/ √ 2 + ∞ k=1 fk cos(kx), g = g0/ √ 2 + k gk cos(kx) and 1 π π π f(x)g(x) dx = ∞ k=0 fkgk. If f and g are odd, and f = k fk sin(kx), g = ∞ k=1 gk sin(kx) then (f, g) = f · g = ∞ k=1 fkgk. Especially the Perseval equality 1 π π −π |f(x)|2 = ∞ k=1 f2 k holds. It can be useful also to find closed formulas for the sum of some series. EXAMPLE. f(x) = x = 2(sin(x) − sin(2x)/2 + sin(3x)/3 − sin(4x)/4 + ... has coefficients fk = 2(−1)k+1 /k and so 4(1 + 1/4 + 1/9 + ...) = 1 π π −π x2 dx = 2π2 /3 or 1 + 1/4 + 1/9 + 1/16 + 1/25 + ....) = π2 /6 . APPROXIMATIONS. -3 -2 -1 1 2 3 -0.2 0.2 0.4 0.6 0.8 1 1.2 If f(x) = k bk cos(kx), then fn(x) = n k=1 bk cos(kx) is an approximation to f. Be- cause ||f − fk||2 = ∞ k=n+1 b2 k goes to zero, the graphs of the functions fn come for large n close to the graph of the function f. The picture to the left shows an approximation of a piecewise continuous even function, the right hand side the values of the coefficients ak = 1 π π −π f(x) cos(kx) dx. 2 4 6 8 10 12 -0.1 0.1 0.2 0.3 0.4 SOME HISTORY. The Greeks approximation of planetary motion through epicycles was an early use of Fourier theory: z(t) = eit is a circle (Aristarchus system), z(t) = eit + eint is an epicycle (Ptolemaeus system), 18’th century Mathematicians like Euler, Lagrange, Bernoulli knew experimentally that Fourier series worked. Fourier’s (picture left) claim of the convergence of the series was con- firmed in the 19’th century by Cauchy and Dirichlet. For continuous functions the sum does not need to converge everywhere. However, as the 19 year old Fej´er (picture right) demonstrated in his theses in 1900, the coefficients still determine the function n−1 k=−(n−1) n−|k| n fkeikx → f(x) for n → ∞ if f is continuous and f(−π) = f(π). Partial differ- ential equations, i.e. the theory of heat had sparked early research in Fourier theory. OTHER FOURIER TRANSFORMS. On a finite interval one obtains a series, on the line an integral, on finite sets, finite sums. The discrete Fourier transformation (DFT) is important for applications. It can be determined efficiently by the (FFT=Fast Fourier transform) found in 1965, reducing the n2 steps to n log(n). Domain Name Synthesis Coefficients T = [−π, π) Fourier series f(x) = k ˆfkeikx ˆfk = 1 2π π −π f(x)e−ikx dx. R = (−∞, ∞)) Fourier transforms f(x) = ∞ ∞ ˆf(k)eikx dx ˆf(k) = 1 2π ∞ ∞ ˆf(k)e−ikx dx Zn = {1, .., n} DFT fm = n k=1 ˆfkeimk2π/n ˆfk = 1 n n m=1 fme−ikm2π/n All these transformations can be defined in dimension d. Then k = (k1, ..., kd) etc. are vectors. 2D DFT is for example useful in image manipulation. COMPUTER ALGEBRA. Packages like Mathematica have the discrete Fourier transform built in Fourier[{0.3, 0.4, 0.5}] for example, gives the DFT of a three vector. You can perform a simple Fourier analysis yourself by listening to a sound like Play[Sin[2000 ∗ x ∗ Floor[7 ∗ x]/12], {x, 0, 20}] ...
HEAT AND WAVE EQUATION Math 21b, O. Knill NOTATION. We write Tx(x, t) and Tt(x, t) for the partial derivatives with respect to x or t. THE HEAT EQUATION. The temperature distribution T(x, t) in a metal bar [0, T] satisfies the heat equation Tt(x, t) = µTxx(x, t). At every point, the rate of change of the temperature is proportional to the second derivative of x . . . T(x, t) at x. The function T(t, x) is zero at both ends of the bar and f(x) = T(0, x) is a given initial temperature distribution. SOLVING IT. This partial differential equation (PDE) is solved by writing T(x, t) = u(x)v(t) which leads to ˙v(t)u(x) = v(t)u (x) or ˙v(t)/(µv(t)) = u (x)/u(x). Because the LHS does not depend on x and the RHS not on t, this must be a constant λ. The ODE u (x) = λu(x) is an eigenvalue problem D2 u = λu which has only solutions when λ = −k2 for integers k. The eigenfunctions of D2 are uk(x) = sin(kx). The second equation ˙v(t) = k2 µv is solved by v(t) = e− k2 µt so that u(x, t) = sin(kx)e−k2 t is a solution of the heat equation. Because linear combinations of solutions are solutions too, a general solution is of the form T(x, t) = ∞ k=1 ak sin(kx)e−k2 t . For fixed t this is a Fourier series. The coefficients ak are determined from f(x) = T(x, 0) = ∞ k=1 ak sin(kx), where ak = 2 π π 0 f(x) sin(kx). (The factor 2 comes from the fact that f should be thought to be extended to an odd function on [−π, π] and the integral from [−π, 0] is the same as the from [0, π].) FOURIER: (Solving the heat equation was the birth of Fourier theory) The heat equation Tt(t, x) = µTxx(t, x) with smooth T(x, 0) = f(x), T(0, 0) = T(π, 0) = 0 has the solution ∞ k=1 ak sin(kx)e−k2 µt with ak = 2 π π 0 f(x) sin(kx) dx. Remark. If T(0, 0) = a, T(π, 0) = b instead, then the solution is T(x, t) = T0(x, t)+a+bx/π, where T0(x, t) is the solution in the box using f0(x) = f(x)−(a+bx/π) which satisfies f(0) = f(π) = 0. (Because T(t, x) = a+bx/π is a solution of the heat equation, we can add it to any solution and get new solutions. This allows to tune the boundary conditions). EXAMPLE. f(x) = x on [−π/2, π/2 π-periodically continued has the Fourier coefficients ak = 2 π π/2 −π/2 x sin(kx) dx = 4/(k2 π)(−1)(k−1)/2 for odd k and 0 else. The solution is T(x, t) = ∞ m=0 4(−1)m (2m + 1)2π e−µ(2m+1)2 t sin((2m + 1)x) . The exponential term containing the time makes the function T(x, t) converge to 0: ”The bar cools.” The higher frequency terms are damped faster because ”Smaller disturbances are smoothed out faster.” VISUALIZATION. We can just plot the graph of the function T(x, t) or plot the temperature distribution for different times ti. 0 0.2 0.4 0.6 0.8 1 0 2 4 6 0 0.5 1 1.5 0 0.2 0.4 0.6 0.8 1
2000 4000 6000 8000 10000 -50 -25 25 50 75 100 DERIVATION OF THE HEAT EQUATION. The temperature T measures the kinetic energy distribution of atoms, and electrons in the bar. Each particle makes a random walk. We can model that as follows: assume we have n adjacent cells containing particles and in each time step, a fraction of the particles moves randomly either to the right or to the left. If Ti(t) is the energy of particles in cell i at time t, then the energy of particles at time t+1 is ρ 2 (Ti−1(t)−2Ti(t)+Ti+1)(t)). This is a discrete version of the second derivative T (x, t) ∼ (T(x + dx, t) − 2T(x, t) + T(x + dx, t))/(dx2 ). FOURIER TRANSFORM=DIAGONALISATION OF D. Fourier theory actually diagonalizes the linear map D: write f = n ˆfneinx = S−1 f and Sf = ˆf = (. . . , c−1, c0, c1, . . .). SDS−1 ˆf = SD n ˆfneinx = S n n ˆfneinx == (. . . , i(−1)c−1, 0c0, ic1, . . .). This means that differentiation changes the Fourier coefficients as cn → incn. From this we see that the diagonalisation of D2 is multiplying with −n2 . Now, if ˙u = D2 u, then ˙ˆun = −µn2 ˆun so that ˆun(t) = e−n2 t ˆu)n(0). The partial differential equation is transformed into a sequence of ordinary differential equations. Differentiation of the function f S ← ˆf D ↓ ↓ M Df S−1 → M ˆf becomes multiplica- tion with the diago- nal matrix M =           · · · · · · · · 2i 0 0 0 0 · · 0 i 0 0 0 · · 0 0 0 0 0 · · 0 0 0 i 0 · · 0 0 0 0 2i · · · · · · · ·           WAVE EQUATION. The wave equation Ttt(x, t) = c2 Txx(x, t) is solved in a similar way as the heat equation: writing T(x, t) = u(x)v(t) gives ¨vu = c2 vu or ¨v/(c2 v) = u /u. Because the left hand is independent of x and the right hand side is independent of t, we have ¨v/(c2 v) = u /u = −k2 = const. The right equation has solutions uk(x) = sin(kx). Now, vk(t) = ak cos(kct) + bk sin(kct) solves ¨v = −c2 k2 v so that Tk(x, t) = uk(x)vk(t) = sin(kx)(ak cos(kct) + bk sin(kct)) is a solution of the wave equation. General solutions can be obtained by taking superpositions of these waves T(x, t) = ∞ k=1 ak(sin(kx) cos(kct) + bk sin(kx) sin(kct). The coefficients ak, bk are obtained from T(x, 0) = ∞ k=1 ak sin(kx) and ˙T(x, 0) = ∞ k=1 kcbk sin(kx). Unlike in the heat equation both T(x, 0) and ˙T(x, 0) have to be specified. WAVE EQUATION: The wave equation Ttt = c2 Txx with smooth T(x, 0) = f(x), ˙T(x, 0) = g(x), T(0, 0) = T(π, 0) = 0 has the solution ∞ k=1 ak sin(kx) cos(kct) + bk sin(kx) sin(kct) with ak = 2 π π 0 f(x) sin(kx) dx and kcbk = 2 π π 0 g(x) sin(kx) dx. QUANTUM MECHANICS = MA- TRIX MECHANICS A particle in a box is described by a linear map L = −D2 ¯h2 /(2m) acting on func- tions on [0, π]. In the Fourier pic- ture, this linear map is M = ¯h2 2m           · · · · · · · · −4 0 0 0 0 · · 0 −1 0 0 0 · · 0 0 0 0 0 · · 0 0 0 −1 0 · · 0 0 0 0 −4 · · · · · · · ·           The diagonal entries are the eigenvalues of L as usual after diagonalisa- tion. Quantum mech. observables are symmetric linear maps: for example: position, X : f → xf, momentum P : f → i¯h f, kinetic energy f → P 2 /2m = −¯h2 ∆/(2m), Coulomb potential energy f → (4π 0)−1 e2 /rf. HIGHER DIMENSIONS. The heat or wave equations can be solved also in higher dimensions using Fourier theory. For example, on a matellic plate [−π, π] × [−π, π] the temperature distribution T(x, y, t) satisfies Tt = µ(Txx + Tyy). The Fourier series of a function f(x, y) in two variables is n,m cn,meinx+my , where cn,m = (2π)−2 π −π π −π f(x, y)e−inx e−imy dx dy. The Gaussian blur filter applied to each color coordinate of a picture acts very similar than the heat flow.
LAPLACE EQUATION (informal) Math 21b, O. Knill LAPLACE EQUATION. The linear map T → ∆T = Txx + Tyy for smooth functions in the plane is called the Laplacian. The PDE ∆T = 0 is called the Laplace equation. A solution T is in the kernel of ∆ and called harmonic. For example T(x, y) = x2 − y2 is harmonic. One can show that T(x, y) = Re((x + iy)n or T(x, y) = Im((x + iy)n ) are all harmonic. DIRICHLET PROBLEM: find a function T(x, y) on a region Ω which satisfies ∆T(x, y) = 0 inside Ω and which is prescribed function f(x, y) at the boundary. This boundary value problem can be solved explicitly in many cases. IN A DISC. {x2 + y2 ≤ 1}. If f(t) = T(cos(t), sin(t)) is prescribed on the boundary and T satisfies ∆T = 0 inside the disc, then T can be found via Fourier theory: If f(t) = a0 + ∞ n=1 an cos(nt) + bn sin(nt), then T(x, y) = a0 + n anRe((x + iy)n ) + bnIm((x+iy)n ) satisfies the Laplace equation and T(cos(t), sin(t)) = f(t) on the boundary of the disc. PROOF. If f = a0 is constant on the boundary then T(x, y) = a. If f(t) = sin(nt) on the boundary z = eit = x+iy then T(x, y) = Im(zn ) and if f(t) = cos(nt) on the boundary then T(x, y) = Re(zn ) is a solution. EXAMPLE. Re(z2 ) = x2 − y2 + i(2xy) gives the two harmonic functions f(x, y) = x2 − y2 and g(x, y) = 2xy: they satisfy (∂2 x + ∂2 y)f = 0. ESPECIALLY: the mean value property for harmonic functions T(0, 0) = 1 2 2π 0 f(t) dt holds. POISSON FORMULA: With z = x + iy = reiθ and T(x, y) = f(z), the general solution is given also by the Poisson integral formula: f(reiθ ) = 1 2π 2π 0 1 − r2 r2 − 2r cos(θ − φ) + 1 f(eiφ ) dφ . PHYSICAL RELEVANCE. If the temperature or charge distribution on the boundary of a region Ω, then T(x, y) is the stable temperature or charge distribution inside that region. In the later case, the electric field inside Ω is then U. An other application is that the velocity v of an ideal incompressible fluid satisfies v = U, where U satisfies the Laplace differential equation ∆U = 0 in the region. SQUARE, SPECIAL CASE. We solve first the case, when T vanishes on three sides x = 0, x = π, y = 0 and T(x, π) = f(x). Separation of variables gives then T(x, y) = n an sin(nx) sinh(ny), where the coefficients an are obtained from T(x, π) = n bn sin(nx) = n an sin(nx) sinh(nπ), where bn = 2 π π 0 f(x) sin(nx) dx are the Fourier coefficients of f. The solution is therefore Tf (x, y) = ∞ n=1 bn sin(nx) sinh(ny)/ sinh(nπ) SQUARE GENERAL CASE Solutions to general boundary conditions can be obtained by adding the solutions in the following cases: Tf (x, π − y) solves the case, when T vanishes on the sides x = 0, x = π, y = π and is f on y = 0. The function Tf (y, x) solves the case, when T vanishes on the sides y = 0, y = π, x = 0 and Tf (π − y, x) solves the case, when T vanishes on y = 0, y = π and x = π. The general solution is T(x, y) = Tf (x, y) + Tg(x, π − y) +Th(y, x) + Tk(π − y, x)
A GENERAL REGION. For a general region, one uses numerical methods. One pos- sibility is by conformal transportation: to map the region into the the disc using a complex map F, which maps the region Ω into the disc. Solve then the problem in the disc with boundary value T(F −1 (x, y)) on the disc. If S(x, y) is the solution there, then T(x, y) = S(F −1 (x, y)) is the solution in the region Ω. The picture shows the example of the Joukowski map z → (z + 1/z)/2 which has been used in fluid dynamics (N.J. Joukowski (1847-1921) was a Russian aerodynamics researcher.) A tourists view on related PDE topics POISSON EQUATION. The Poisson equation ∆f = g in a square [0, π]2 can be solved via Fourier theory: If f(x, y) = k,m an,m cos(nx) cos(my) and g(x, y) = k,m bn,m cos(nx) cos(my), then fxx + fyy = n,m −(n2 + m2 )an,m cos(nx) cos(my) = k,m bn,m cos(nx) cos(my), then an,m = −bn,m/(n2 − m2 ). The poisson equation is important in electrostatics, for example to determine the electromagnetic field when the charge and current distribution is known: ∆U(x) = −(1/ 0)ρ, then E = U is the electric field to the charge distribution ρ(x). EIGENVALUES. The functions sin(nx) sin(ny) are eigenfunctions for the Laplace operator fxx + fyy = n2 f on the disc. For a general bounded region Ω, we can look at all smooth functions which are zero on the boundary of Ω. The possi- ble eigenvalues of ∆f = λf are the possible ener- gies of a particle in Ω. COMPLEX DYNAMICS. Also in complex dy- namics, harmonic functions appear. Finding properties of complicated sets like the Mandel- brot set is done by mapping the exterior to the outside of the unit circle. If the Mandelbrot set is charged then the contour lines of equal poten- tial can be obtained as the corresponding contour lines in the disc case (where the lines are circles). SCHR¨ODINGER EQUATION. If H is the energy operator, then i¯h ˙f = Hf is called the Schr¨odinger equa- tion. If H = −¯h2 ∆/(2m), this looks very much like the heat equation, if there were not the i. If f is an eigenvalue of H, then i¯h ˙f = λf and f(t) = ei¯ht f(0). In the heat equation, we would get f(t) = e−µ(t) f(0). The evolution of the Schr¨odinger equation is very smilar to the wave equation. QUANTUM CHAOS studies the eigenvalues and eigenvectors of the Laplacian in a bounded region. If the billiard in the region is chaotic, the study of the eigensystem is called quantum chaos. The eigenvalue problem ∆f = λf and the billiard problem in the region are related. A famous open problem is whether two smooth convex regions in the plane for which the eigenvalues λj are the same are equivalent up to rotation and translation. In a square of size L we know all the eigenvalues π2 L2 (n2 + m2 ). The eigenvectors are sin(pi L nx) sin(pi L mx). We have explicit formulas for the eigenfunctions and eigenvalues. On the classical level, for the billiard, we have also a complete picture, how a billiard path will look like. The square is an example, where we have no quantum chaos.

More Related Content

PDF
Towards a stable definition of Algorithmic Randomness
Hector Zenil
 
PDF
directed-research-report
Ryen Krusinga
 
PDF
Fractal dimension versus Computational Complexity
Hector Zenil
 
PDF
Fractal Dimension of Space-time Diagrams and the Runtime Complexity of Small ...
Hector Zenil
 
PDF
Graph Spectra through Network Complexity Measures: Information Content of Eig...
Hector Zenil
 
PDF
A Numerical Method for the Evaluation of Kolmogorov Complexity, An alternativ...
Hector Zenil
 
PDF
Machine learning (11)
NYversity
 
PDF
Thesis Presentation_Extreme Learning Machine_Nimai_SC14M045
Nimai Chand Das Adhikari
 
Towards a stable definition of Algorithmic Randomness
Hector Zenil
 
directed-research-report
Ryen Krusinga
 
Fractal dimension versus Computational Complexity
Hector Zenil
 
Fractal Dimension of Space-time Diagrams and the Runtime Complexity of Small ...
Hector Zenil
 
Graph Spectra through Network Complexity Measures: Information Content of Eig...
Hector Zenil
 
A Numerical Method for the Evaluation of Kolmogorov Complexity, An alternativ...
Hector Zenil
 
Machine learning (11)
NYversity
 
Thesis Presentation_Extreme Learning Machine_Nimai_SC14M045
Nimai Chand Das Adhikari
 

What's hot (20)

PDF
Common fixed point theorems for contractive maps of
Alexander Decker
 
PPT
. An introduction to machine learning and probabilistic ...
butest
 
PDF
Lecture 8
Krishna Karri
 
DOC
Lesson 8
Vinnu Vinay
 
PDF
Information Content of Complex Networks
Hector Zenil
 
PDF
Matrix Completion Presentation
Michael Hankin
 
PDF
Ordinal Regression and Machine Learning: Applications, Methods, Metrics
Francesco Casalegno
 
PDF
overviewPCA
Edwin Heredia
 
PDF
Exact Matrix Completion via Convex Optimization Slide (PPT)
Joonyoung Yi
 
PDF
2. polynomial interpolation
EasyStudy3
 
PPTX
Statistical Physics Assignment Help
Statistics Assignment Help
 
PPTX
Application of interpolation and finite difference
Manthan Chavda
 
PDF
A likelihood-free version of the stochastic approximation EM algorithm (SAEM)...
Umberto Picchini
 
PDF
On observer design methods for a
csandit
 
PDF
Regularized Compression of A Noisy Blurred Image
ijcsa
 
PDF
A NEW PARALLEL ALGORITHM FOR COMPUTING MINIMUM SPANNING TREE
ijscmc
 
PDF
A new implementation of k-MLE for mixture modelling of Wishart distributions
Frank Nielsen
 
PDF
Random Matrix Theory in Array Signal Processing: Application Examples
Förderverein Technische Fakultät
 
PDF
010_20160216_Variational Gaussian Process
Ha Phuong
 
PPT
Numerical Methods
ESUG
 
Common fixed point theorems for contractive maps of
Alexander Decker
 
. An introduction to machine learning and probabilistic ...
butest
 
Lecture 8
Krishna Karri
 
Lesson 8
Vinnu Vinay
 
Information Content of Complex Networks
Hector Zenil
 
Matrix Completion Presentation
Michael Hankin
 
Ordinal Regression and Machine Learning: Applications, Methods, Metrics
Francesco Casalegno
 
overviewPCA
Edwin Heredia
 
Exact Matrix Completion via Convex Optimization Slide (PPT)
Joonyoung Yi
 
2. polynomial interpolation
EasyStudy3
 
Statistical Physics Assignment Help
Statistics Assignment Help
 
Application of interpolation and finite difference
Manthan Chavda
 
A likelihood-free version of the stochastic approximation EM algorithm (SAEM)...
Umberto Picchini
 
On observer design methods for a
csandit
 
Regularized Compression of A Noisy Blurred Image
ijcsa
 
A NEW PARALLEL ALGORITHM FOR COMPUTING MINIMUM SPANNING TREE
ijscmc
 
A new implementation of k-MLE for mixture modelling of Wishart distributions
Frank Nielsen
 
Random Matrix Theory in Array Signal Processing: Application Examples
Förderverein Technische Fakultät
 
010_20160216_Variational Gaussian Process
Ha Phuong
 
Numerical Methods
ESUG
 
Ad

Similar to Linear algebra havard university (20)

DOCX
M Jamee Raza (BSE-23S-056)LA project.docx
chomukhan112
 
PPTX
Applications of linear algebra
Prerak Trivedi
 
PPTX
Application of matrices in Daily life
shubham mishra
 
PDF
Lecture50
Muhammad Kamran
 
PPTX
Applications of linear algebra in computer science
Arnob Khan
 
PPT
Linear algebra notes 1
Ghulam Murtaza
 
PPT
Linear algebra notes 2
Ghulam Murtaza
 
PPT
Linear algebra notes
Ghulam Mehdi Sahito
 
PDF
Ebook mst209 block3_e1i1_n9780749252830_l1 (2)
Fred Stanworth
 
PDF
Matrix Theory And Linear Algebra First Peter Selinger
kyneroqya
 
PDF
Matrix Theory And Linear Algebra First Peter Selinger
gyimahveimar82
 
PDF
Taller2 parcial2 grupo_4_
MISHELLESTEFANIACUAS
 
PDF
Download full ebook of Matrix Algebra James E Gentle instant download pdf
mekdismateo
 
PDF
Harvard_University_-_Linear_Al
Ramil Jay Ureta
 
PDF
Harvard_University_-_Linear_Al
ramiljayureta
 
DOCX
Applications of Matrices
santhosh kumar
 
PPTX
Lecture 1- Introduction.pptx
ManjulGupta10
 
PDF
A first course in linear algebra robert a. beezer university of puget sound v...
chen john
 
PPT
Intro Class.ppt
PrakashDuraisamyCIT
 
DOCX
Applicationofmatrices21155
GnaneshKamisetti
 
M Jamee Raza (BSE-23S-056)LA project.docx
chomukhan112
 
Applications of linear algebra
Prerak Trivedi
 
Application of matrices in Daily life
shubham mishra
 
Lecture50
Muhammad Kamran
 
Applications of linear algebra in computer science
Arnob Khan
 
Linear algebra notes 1
Ghulam Murtaza
 
Linear algebra notes 2
Ghulam Murtaza
 
Linear algebra notes
Ghulam Mehdi Sahito
 
Ebook mst209 block3_e1i1_n9780749252830_l1 (2)
Fred Stanworth
 
Matrix Theory And Linear Algebra First Peter Selinger
kyneroqya
 
Matrix Theory And Linear Algebra First Peter Selinger
gyimahveimar82
 
Taller2 parcial2 grupo_4_
MISHELLESTEFANIACUAS
 
Download full ebook of Matrix Algebra James E Gentle instant download pdf
mekdismateo
 
Harvard_University_-_Linear_Al
Ramil Jay Ureta
 
Harvard_University_-_Linear_Al
ramiljayureta
 
Applications of Matrices
santhosh kumar
 
Lecture 1- Introduction.pptx
ManjulGupta10
 
A first course in linear algebra robert a. beezer university of puget sound v...
chen john
 
Intro Class.ppt
PrakashDuraisamyCIT
 
Applicationofmatrices21155
GnaneshKamisetti
 
Ad

Linear algebra havard university

  • 1. USE OF LINEAR ALGEBRA I Math 21b, O. Knill This is not a list of topics covered in the course. It is rather a lose selection of subjects, in which linear algebra is useful or relevant. The aim is to convince you that it is worth learning this subject. Most likely, some of this handout does not make much sense yet to you. Look at this page at the end of the course again, some of the content will become more interesting then. 1 2 3 4 GRAPHS, NETWORKS. Linear al- gebra can be used to understand networks which is a collection of nodes connected by edges. Networks are also called graphs. The adja- cency matrix of a graph is defined by Aij = 1 if there is an edge from node i to node j in the graph. Oth- erwise the entry is zero. A prob- lem using such matrices appeared on a blackboard at MIT in the movie ”Good will hunting”. Application: An ij is the number of n- step walks in the graph which start at the vertex i and end at the vertex j. CHEMISTRY, MECHANICS Complicated objects like a bridge (the picture shows Storrow Drive connection bridge which is part of the ”big dig”), or a molecule (i.e. a protein) can be modeled by finitely many parts (bridge elements or atoms) coupled with attractive and repelling forces. The vibrations of the system are described by a differential equation ˙x = Ax, where x(t) is a vector which depends on time. Differential equations are an important part of this course. The solution x(t) = exp(At) of the differential equation ˙x = Ax can be understood and computed by find- ing the eigenvalues of the matrix A. Knowing these frequencies is impor- tant for the design of a mechani- cal object because the engineer can damp dangerous frequencies. In chemistry or medicine, the knowl- edge of the vibration resonances al- lows to determine the shape of a molecule. QUANTUM COMPUTING A quantum computer is a quantum mechanical system which is used to perform computations. The state x of a machine is no more a sequence of bits like in a classical computer but a sequence of qubits, where each qubit is a vector. The memory of the computer can be represented as a vector. Each computation step is a multiplication x → Ax with a suitable matrix A. Theoretically, quantum computa- tions could speed up conventional computations significantly. They could be used for example for cryp- tological purposes. Freely available quantum computer language (QCL) interpreters can simulate quantum computers with an arbitrary num- ber of qubits. CHAOS THEORY. Dynamical systems theory deals with the iteration of maps or the analysis of solutions of differential equations. At each time t, one has a map T(t) on the vector space. The linear approximation DT(t) is called Jacobean is a matrix. If the largest eigenvalue of DT(t) grows exponentially in t, then the system shows ”sensitive dependence on initial conditions” which is also called ”chaos”. Examples of dynamical systems are our solar system or the stars in a galaxy, electrons in a plasma or par- ticles in a fluid. The theoretical study is intrinsically linked to linear algebra because stability properties often depends on linear approxima- tions.
  • 2. USE OF LINEAR ALGEBRA II Math 21b, O. Knill CODING, ERROR CORRECTION Coding theory is used for encryp- tion or error correction. In the first case, the data x are maped by a map T into code y=Tx. For a good code, T is a ”trapdoor function” in the sense that it is hard to get x back when y is known. In the sec- ond case, a code is a linear subspace X of a vector space and T is a map describing the transmission with er- rors. The projection of Tx onto the subspace X corrects the error. Linear algebra enters in different ways, often directly because the ob- jects are vectors but also indirectly like for example in algorithms which aim at cracking encryption schemes. DATA COMPRESSION Image- (i.e. JPG), video- (MPG4) and sound compression algorithms (i.e. MP3) make use of linear transformations like the Fourier transform. In all cases, the com- pression makes use of the fact that in the Fourier space, information can be cut away without disturbing the main information. Typically, a picture, a sound or a movie is cut into smaller junks. These parts are represented by vec- tors. If U denotes the Fourier trans- form and P is a cutoff function, then y = PUx is transferred or stored on a CD or DVD. The receiver obtains back UT y which is close to x in the sense that the human eye or ear does not notice a big difference. SOLVING SYSTEMS OR EQUA- TIONS When extremizing a func- tion f on data which satisfy a con- straint g(x) = 0, the method of Lagrange multipliers asks to solve a nonlinear system of equations f(x) = λ g(x), g(x) = 0 for the (n + 1) unknowns (x, l), where f is the gradient of f. Solving systems of nonlinear equa- tions can be tricky. Already for sys- tems of polynomial equations, one has to work with linear spaces of polynomials. Even if the Lagrange system is a linear system, the task of solving it can be done more ef- ficiently using a solid foundation of linear algebra. GAMES Moving around in a world described in a computer game re- quires rotations and translations to be implemented efficiently. Hard- ware acceleration can help to handle this. Rotations are represented by or- thogonal matrices. For example, if an object located at (0, 0, 0), turning around the y-axes by an angle φ, every point in the ob- ject gets transformed by the matrix  cos(φ) 0 sin(φ) 0 1 0 − sin(φ) 0 cos(φ)  
  • 3. USE OF LINEAR ALGEBRA (III) Math 21b, Oliver Knill STATISTICS When analyzing data statistically, one often is interested in the correlation matrix Aij = E[YiYj] of a random vector X = (X1, . . . , Xn) with Yi = Xi − E[Xi]. This matrix is derived from the data and determines often the random variables when the type of the dis- tribution is fixed. For example, if the random variables have a Gaussian (=Bell shaped) dis- tribution, the correlation matrix to- gether with the expectation E[Xi] determines the random variables. GAME THEORY Abstract Games are often represented by pay-off ma- trices. These matrices tell the out- come when the decisions of each player are known. A famous example is the prisoner dilemma. Each player has the choice to corporate or to cheat.. The game is described by a 2x2 matrix like for example 3 0 5 1 . If a player cooperates and his partner also, both get 3 points. If his partner cheats and he cooperates, he gets 5 points. If both cheat, both get 1 point. More generally, in a game with two players where each player can chose from n strategies, the pay- off matrix is a n times n matrix A. A Nash equilibrium is a vector p ∈ S = { i pi = 1, pi ≥ 0 } for which qAp ≤ pAp for all q ∈ S. NEURAL NETWORK In part of neural network theory, for exam- ple Hopfield networks, the state space is a 2n-dimensional vector space. Every state of the network is given by a vector x, where each component takes the values −1 or 1. If W is a symmetric nxn matrix, one can define a ”learning map” T : x → signWx, where the sign is taken component wise. The en- ergy of the state is the dot prod- uct −(x, Wx)/2. One is interested in fixed points of the map. For example, if Wij = xiyj, then x is a fixed point of the learning map.
  • 4. USE OF LINEAR ALGEBRA (IV) Math 21b, Oliver Knill MARKOV. Suppose we have three bags with 10 balls each. Every time we throw a dice and a 5 shows up, we move a ball from bag 1 to bag 2, if the dice shows 1 or 2, we move a ball from bag 2 to bag 3, if 3 or 4 turns up, we move a ball from bag 3 to bag 1 and a ball from bag 3 to bag 2. What distribution of balls will we see in average? The problem defines a Markov chain described by a matrix  5/6 1/6 0 0 2/3 1/3 1/6 1/6 2/3  . From this matrix, the equilibrium distribution can be read off as an eigenvector of a matrix. SPLINES In computer aided de- sign (CAD) used for example to construct cars, one wants to inter- polate points with smooth curves. One example: assume you want to find a curve connecting two points P and Q and the direction is given at each point. Find a cubic function f(x, y) = ax3 + bx2 y + cxy2 + dy3 which interpolates. If we write down the conditions, we will have to solve a system of 4 equa- tions for four unknowns. Graphic artists (i.e. at the company ”Pixar”) need to have linear algebra skills also at many other topics in computer graphics. a b c SYMBOLIC DYNAMICS Assume that a system can be in three dif- ferent states a, b, c and that tran- sitions a → b, b → a, b → c, c → c, c → a are allowed. A possible evolution of the system is then a, b, a, b, a, c, c, c, a, b, c, a... One calls this a description of the system with symbolic dynamics. This language is used in information the- ory or in dynamical systems theory. The dynamics of the system is coded with a symbolic dynamical system. The transition matrix is  0 1 0 1 0 1 1 0 1  . Information theoretical quantities like the ”entropy” can be read off from this matrix. o p q r a b c d INVERSE PROBLEMS The recon- struction of a density function from projections along lines reduces to the solution of the Radon trans- form. Studied first in 1917, it is to- day a basic tool in applications like medical diagnosis, tokamak moni- toring, in plasma physics or for as- trophysical applications. The re- construction is also called tomogra- phy. Mathematical tools developed for the solution of this problem lead to the construction of sophisticated scanners. It is important that the inversion h = R(f) → f is fast, ac- curate, robust and requires as few datas as possible. Toy problem: We have 4 containers with density a, b, c, d arranged in a square. We are able and measure the light absorption by by sending light through it. Like this, we get o = a + b,p = c + d,q = a + c and r = b + d. The problem is to re- cover a, b, c, d. The system of equa- tions is equivalent to Ax = b, with x = (a, b, c, d) and b = (o, p, q, r) and A =     1 1 0 0 0 0 1 1 1 0 1 0 0 1 0 1    .
  • 5. ICE: A BLACKBOARD PROBLEM Math 21b, O. Knill In the movie ”Good Will Hunting”, the main character Will Hunting (Matt Damon) solves a black- board challenge problem, which is given as a challenge to a linear algebra class. THE ”WILL HUNTING” PROBLEM. G is the graph 1 2 3 4 Find. 1) the adjacency matrix A. 2) the matrix giving the number of 3 step walks. 3) the generating function for walks from point i → j. 4) the generating function for walks from points 1 → 3. This problem belongs to linear algebra and calculus evenso the problem origins from graph the- ory or combinatorics. For a calculus student who has never seen the connection between graph theory, calculus and linear algebra, the assignment is actually hard – probably too hard - as the movie correctly indicates. The problem was posed in the last part of a linear algebra course. An explanation of some terms: THE ADJACENCY MATRIX. The structure of the graph can be encoded with a 4 × 4 array which encodes how many paths of length 1, one can take in the graph from one node to an other: no one no one one none two one no two no no one one no no which can more conve- niently be written as an array of numbers called a matrix: L =      0 1 0 1 1 0 2 1 0 2 0 0 1 1 0 0      Problem 2 asks to find the matrix which encodes all possible paths of length 3. GENERATING FUNCTION. To any graph one can assign for every pair of nodes i, j a series f(z) = ∞ n=0 a(ij) n zn , where a(ij) n is the number of possible walks from node i to node j with n steps. Problem 3) asks for an explicit expression of f(z) and problem 4) asks for an explicit expression in the special case i = 1, j = 3. Linear algebra has many relations to other fields in mathematics. It is not true that linear algebra is just about solving systems of linear equations.
  • 6. SOLUTION TO 2). [L2 ]ij is by definition of the matrix product the sum Li1L1j + Li2L2j + ... + Li4Lhj. Each term Li1L1j is 1 if and only if there is a path of length 2 going from i to j passing through k. Therefore [L2 ]ij is the number of paths of length 2 going from node i to j. Similarly, [Ln ]ij is the number of paths of length n going from i to j. The answer is L3 =      2 7 2 3 7 2 12 7 2 12 0 2 3 7 2 2      SOLUTION TO 3). The geometric series formula n xn = (1 − x)−1 holds also for matrices: f(z) = ∞ n=0 [Ln ]ijzn = [ ∞ n=0 Ln zn )]ij = (1 − Lz)−1 Cramers formula for the inverse of a matrix which involves the determinant A−1 = det(Aij)/det(A) This leads to an explicit formula det(1 − zLij)/det(1 − zL) which can also be written as det(Lij − z)/det(L − z) . SOLUTION TO 4). This is the special case, when i = 1 and j = 3. det(L − z) = det(    1 −z 1 0 2 0 1 1 −z   ) = −2 − z + 3z2 − z3 . det(L − z) =      −z 1 0 1 1 −z 2 1 0 2 −z 0 1 1 0 −z      = 4 − 2z − 7z2 + z4 The final result is (z3 + 3z2 − z − 2)/(z4 − 7z2 − 2z + 4) .
  • 7. LINEAR EQUATIONS Math 21b, O. Knill SYSTEM OF LINEAR EQUATIONS. A collection of linear equations is called a system of linear equations. An example is 3x − y − z = 0 −x + 2y − z = 0 −x − y + 3z = 9 . It consists of three equations for three unknowns x, y, z. Linear means that no nonlinear terms like x2 , x3 , xy, yz3 , sin(x) etc. appear. A formal definition of linearity will be given later. LINEAR EQUATION. More precicely, ax + by = c is the general linear equation in two variables. ax + by + cz = d is the general linear equation in three variables. The general linear equation in n variables is a1x1 + a2x2 + ... + anxn = 0 Finitely many such equations form a system of linear equations. SOLVING BY ELIMINATION. Eliminate variables. In the example the first equation gives z = 3x − y. Putting this into the second and third equation gives −x + 2y − (3x − y) = 0 −x − y + 3(3x − y) = 9 or −4x + 3y = 0 8x − 4y = 9 . The first equation gives y = 4/3x and plugging this into the other equation gives 8x − 16/3x = 9 or 8x = 27 which means x = 27/8. The other values y = 9/2, z = 45/8 can now be obtained. SOLVE BY SUITABLE SUBTRACTION. Addition of equations. If we subtract the third equation from the second, we get 3y − 4z = −9 and add three times the second equation to the first, we get 5y − 4z = 0. Subtracting this equation to the previous one gives −2y = −9 or y = 2/9. SOLVE BY COMPUTER. Use the computer. In Mathematica: Solve[{3x − y − z == 0, −x + 2y − z == 0, −x − y + 3z == 9}, {x, y, z}] . But what did Mathematica do to solve this equation? We will look in this course at some efficient algorithms. GEOMETRIC SOLUTION. Each of the three equations represents a plane in three-dimensional space. Points on the first plane satisfy the first equation. The second plane is the solution set to the second equation. To satisfy the first two equations means to be on the intersection of these two planes which is here a line. To satisfy all three equations, we have to intersect the line with the plane representing the third equation which is a point. LINES, PLANES, HYPERPLANES. The set of points in the plane satisfying ax + by = c form a line. The set of points in space satisfying ax + by + cd = d form a plane. The set of points in n-dimensional space satisfying a1x1 + ... + anxn = a0 define a set called a hyperplane.
  • 8. RIDDLES: ”15 kids have bicycles or tricycles. Together they count 37 wheels. How many have bicycles?” Solution. With x bicycles and y tricycles, then x + y = 15, 2x + 3y = 37. The solution is x = 8, y = 7. ”Tom, the brother of Carry has twice as many sisters as brothers while Carry has equal number of systers and brothers. How many kids is there in total in this family?” Solution If there are x brothers and y systers, then Tom has y sisters and x−1 brothers while Carry has x brothers and y − 1 sisters. We know y = 2(x − 1), x = y − 1 so that x + 1 = 2(x − 1) and so x = 3, y = 4. INTERPOLATION. Find the equation of the parabola which passes through the points P = (0, −1), Q = (1, 4) and R = (2, 13). Solution. Assume the parabola is ax2 +bx+c = 0. So, c = −1, a+b+c = 4, 4a+2b+c = 13. Elimination of c gives a + b = 5, 4a + 2b = 14 so that 2b = 6 and b = 3, a = 2. The parabola has the equation 2x2 + 3x − 1 = 0 TOMOGRAPHY Here is a toy example of a problem one has to solve for magnetic resonance imaging (MRI), which makes use of the absorption and emission of energy in the radio frequency range of the electromag- netic spectrum. Assume we have 4 hydrogen atoms whose nuclei are excited with intensity a, b, c, d. We measure the spin echo intensity in 4 different directions. 3 = a + b,7 = c + d,5 = a + c and 5 = b + d. What is a, b, c, d? Solution: a = 2, b = 1, c = 3, d = 4. However, also a = 0, b = 3, c = 5, d = 2 solves the problem. This system has not a unique solution even so there are 4 equations and 4 unknowns. A good introduction into MRI can be found online at (http://www.cis.rit.edu/htbooks/mri/inside.htm). o p q r a b c d INCONSISTENT. x − y = 4, y + z = 5, x + z = 6 is a system with no solutions. It is called inconsistent. x11 x12 x21 x22 a11 a12 a13 a14 a21 a24 a31 a34 a41 a42 a43 a44 EQUILIBRIUM. As an example of a system with many variables, consider a drum modeled by a fine net. The heights at each interior node needs the average the heights of the 4 neighboring nodes. The height at the boundary is fixed. With n2 nodes in the interior, we have to solve a system of n2 equations. For exam- ple, for n = 2 (see left), the n2 = 4 equations are x11 = a21 + a12 + x21 + x12, x12 = x11 + x13 + x22 + x22, x21 = x31+x11+x22+a43, x22 = x12+x21+a43+a34. To the right, we see the solution to a problem with n = 300, where the computer had to solve a system with 90 000 variables. LINEAR OR NONLINEAR? a) The ideal gas law PV = nKT for the P, V, T, the pressure, volume and temperature of the gas. b) The Hook law F = k(x − x0) relates the force F pulling a string at position x which is relaxed at x0.
  • 9. MATRICES AND GAUSS-JORDAN Math 21b, O. Knill MATRIX FORMULATION. Consider the sys- tem of linear equations 3x − y − z = 0 −x + 2y − z = 0 −x − y + 3z = 9 The system can be written as Ax = b, where A is a matrix (called coefficient matrix) and and x and b are vectors. A =   3 −1 −1 −1 2 −1 −1 −1 3   , x =   x y z   , b =   0 0 9   . ((Ax)i is the dot product of the i’th row with x). We also look at the augmented matrix where one puts separators for clarity reasons. B =   3 −1 −1 | 0 −1 2 −1 | 0 −1 −1 3 | 9   . MATRIX ”JARGON”. A rectangular array of numbers is called a matrix. If the matrix has m rows and n columns, it is called a m × n matrix. A matrix with one column only is called a column vector, a matrix with one row a row vector. The entries of a matrix are denoted by aij, where i is the row and j is the column. In the case of the linear equation above, the matrix A is a square matrix and the augmented matrix B above is a 3 × 4 matrix. m n GAUSS-JORDAN ELIMINATION. Gauss-Jordan Elimination is a process, where successive subtraction of multiples of other rows or scaling brings the matrix into reduced row echelon form. The elimination process consists of three possible steps which are called elementary row operations: • Swap two rows. • Divide a row by a scalar • Subtract a multiple of a row from an other row. The process transfers a given matrix A into a new matrix rref(A) REDUCED ECHELON FORM. A matrix is called in reduced row echelon form 1) if a row has nonzero entries, then the first nonzero entry is 1. (leading one) 2) if a column contains a leading 1, then the other column entries are 0. 3) if a row has a leading 1, then every row above has leading 1’s to the left. Pro memoriam: Leaders like to be number one, are lonely and want other leaders above to their left. RANK. The number of leading 1 in rref(A) is called the rank of A. SOLUTIONS OF LINEAR EQUATIONS. If Ax = b is a linear system of equations with m equations and n unknowns, then A is a m × n matrix. We have the following three possibilities: • Exactly one solution. There is a leading 1 in each row but not in the last row. • Zero solutions. There is a leading 1 in the last row. • Infinitely many solutions. There are rows without leading 1 and no leading 1 is in the last row. JIUZHANG SUANSHU. The technique of successively eliminating variables from systems of linear equations is called Gauss elimination or Gauss Jordan elimination and appeared already in the Chinese manuscript ”Jiuzhang Suanshu” (’Nine Chapters on the Mathematical art’). The manuscript appeared around 200 BC in the Han dynasty and was probably used as a textbook. For more history of Chinese Mathematics, see http://aleph0.clarku.edu/ ˜ djoyce/mathhist/china.html.
  • 10. EXAMPLES. The reduced echelon form of the augmented matrix B determines on how many solutions the linear system Ax = b has. THE GOOD (1 solution) THE BAD (0 solution) THE UGLY (∞ solutions) 0 1 2 | 2 1 −1 1 | 5 2 1 −1 | −2 1 −1 1 | 5 0 1 2 | 2 2 1 −1 | −2 1 −1 1 | 5 0 1 2 | 2 0 3 −3 | −12 1 −1 1 | 5 0 1 2 | 2 0 1 −1 | −4 1 0 3 | 7 0 1 2 | 2 0 0 −3 | −6 1 0 3 | 7 0 1 2 | 2 0 0 1 | 2   1 0 0 | 1 0 1 0 | −2 0 0 1 | 2   Rank(A) = 3, Rank(B) = 3.   0 1 2 | 2 1 −1 1 | 5 1 0 3 | −2     1 −1 1 | 5 0 1 2 | 2 1 0 3 | −2     1 −1 1 | 5 0 1 2 | 2 0 1 2 | −7     1 −1 1 | 5 0 1 2 | 2 0 0 0 | −9     1 0 3 | 7 0 1 2 | 2 0 0 0 | −9     1 0 3 | 7 0 1 2 | 2 0 0 0 | 1   Rank(A) = 2, Rank(B) = 3.   0 1 2 | 2 1 −1 1 | 5 1 0 3 | 7     1 −1 1 | 5 0 1 2 | 2 1 0 3 | 7     1 −1 1 | 5 0 1 2 | 2 0 1 2 | 2     1 −1 1 | 5 0 1 2 | 2 0 0 0 | 0     1 0 3 | 7 0 1 2 | 2 0 0 0 | 0   Rank(A) = 2, Rank(B) = 2. JORDAN. The German geodesist Wilhelm Jordan (1842-1899) applied the Gauss-Jordan method to finding squared errors to work on surveying. (An other ”Jordan”, the French Mathematician Camille Jordan (1838- 1922) worked on linear algebra topics also (Jordan form) and is often mistakenly credited with the Gauss-Jordan process.) GAUSS. Gauss developed Gaussian elimination around 1800 and used it to solve least squares problems in celestial mechanics and later in geodesic compu- tations. In 1809, Gauss published the book ”Theory of Motion of the Heavenly Bodies” in which he used the method for solving astronomical problems. One of Gauss successes was the prediction of an asteroid orbit using linear algebra. CERES. On 1. January of 1801, the Italian astronomer Giuseppe Piazzi (1746- 1826) discovered Ceres, the first and until 2001 the largest known asteroid in the solar system. (A new found object called 2001 KX76 is estimated to have a 1200 km diameter, half the size of Pluto) Ceres is a rock of 914 km diameter. (The pictures Ceres in infrared light). Gauss was able to predict the orbit of Ceres from a few observations. By parameterizing the orbit with parameters and solving a linear system of equations (similar to one of the homework problems, where you will fit a cubic curve from 4 observations), he was able to derive the orbit parameters.
  • 11. ON SOLUTIONS OF LINEAR EQUATIONS Math 21b, O. Knill MATRIX. A rectangular array of numbers is called a matrix. A =     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn     = w w w w v v v 1 2 3 4 1 2 3 A matrix with m rows and n columns is called a m × n matrix. A matrix with one column is a column vector. The entries of a matrix are denoted aij, where i is the row number and j is the column number. ROW AND COLUMN PICTURE. Two interpretations Ax =     −w1− −w2− . . . −wm−       | x |   =     w1 · x w2 · x . . . wm · x     Ax =   | | · · · | v1 v2 · · · vn | | · · · |       x1 x2 · · · xm     = x1v1 +x2v2 +· · ·+xmvm = b . Row picture: each bi is the dot product of a row vector wi with x. Column picture: b is a sum of scaled column vectors vj. ”Row and Column at Harvard” EXAMPLE. The system of linear equations 3x − 4y − 5z = 0 −x + 2y − z = 0 −x − y + 3z = 9 is equivalent to Ax = b, where A is a coefficient matrix and x and b are vectors. A =   3 −4 −5 −1 2 −1 −1 −1 3   , x =   x y z   , b =   0 0 9   . The augmented matrix (separators for clarity) B =   3 −4 −5 | 0 −1 2 −1 | 0 −1 −1 3 | 9   . In this case, the row vectors of A are w1 = 3 −4 −5 w2 = −1 2 −1 w3 = −1 −1 3 The column vectors are v1 =   3 −1 −1   , v2 =   −4 −2 −1   , v3 =   −5 −1 3   Row picture: 0 = b1 = 3 −4 −5 ·   x y z   Column picture:   0 0 9   = x1   3 −1 −1   + x2   3 −1 −1   + x3   3 −1 −1   SOLUTIONS OF LINEAR EQUATIONS. A system Ax = b with m equations and n unknowns is defined by the m × n matrix A and the vector b. The row reduced matrix rref(B) of B determines the number of solutions of the system Ax = b. There are three possibilities: • Consistent: Exactly one solution. There is a leading 1 in each row but none in the last row of B. • Inconsistent: No solutions. There is a leading 1 in the last row of B. • Infinitely many solutions. There are rows of B without leading 1. If m < n (less equations then unknowns), then there are either zero or infinitely many solutions. The rank rank(A) of a matrix A is the number of leading ones in rref(A). 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
  • 12. (exactly one solution) (no solution) (infinitely many solutions) MURPHYS LAW. ”If anything can go wrong, it will go wrong”. ”If you are feeling good, don’t worry, you will get over it!” ”Whenever you do Gauss-Jordan elimination, you screw up during the first couple of steps.” MURPHYS LAW IS TRUE. Two equations could contradict each other. Geometrically this means that the two planes do not intersect. This is possible if they are parallel. Even without two planes be- ing parallel, it is possible that there is no intersection between all three of them. Also possible is that we don’t have enough equations (for example because two equations are the same) and that there are many solutions. Furthermore, we can have too many equations and the four planes would not intersect. RELEVANCE OF EXCEPTIONAL CASES. There are important applications, where ”unusual” situations happen: For example in medical tomography, systems of equations appear which are ”ill posed”. In this case one has to be careful with the method. The linear equations are then obtained from a method called the Radon transform. The task for finding a good method had led to a Nobel prize in Medicis 1979 for Allan Cormack. Cormack had sabbaticals at Harvard and probably has done part of his work on tomography here. Tomography helps today for example for cancer treatment. MATRIX ALGEBRA. Matrices can be added, subtracted if they have the same size: A+B =     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn    +     b11 b12 · · · b1n b21 b22 · · · b2n · · · · · · · · · · · · bm1 bm2 · · · bmn     =     a11 + b11 a12 + b12 · · · a1n + b1n a21 + b21 a22 + b22 · · · a2n + b2n · · · · · · · · · · · · am1 + bm2 am2 + bm2 · · · amn + bmn     They can also be scaled by a scalar λ: λA = λ     a11 a12 · · · a1n a21 a22 · · · a2n · · · · · · · · · · · · am1 am2 · · · amn     =     λa11 λa12 · · · λa1n λa21 λa22 · · · λa2n · · · · · · · · · · · · λam1 λam2 · · · λamn    
  • 13. LINEAR TRANSFORMATIONS Math 21b, O. Knill TRANSFORMATIONS. A transformation T from a set X to a set Y is a rule, which assigns to every element in X an element y = T(x) in Y . One calls X the domain and Y the codomain. A transformation is also called a map. LINEAR TRANSFORMATION. A map T from Rn to Rm is called a linear transformation if there is a m × n matrix A such that T(x) = Ax . EXAMPLES. • To the linear transformation T(x, y) = (3x+4y, x+5y) belongs the matrix 3 4 1 5 . This transformation maps the plane onto itself. • T(x) = −3x is a linear transformation from the real line onto itself. The matrix is A = [−3]. • To T(x) = y ·x from R3 to R belongs the matrix A = y = y1 y2 y3 . This 1×3 matrix is also called a row vector. If the codomain is the real axes, one calls the map also a function. function defined on space. • T(x) = xy from R to R3 . A = y =   y1 y2 y3   is a 3 × 1 matrix which is also called a column vector. The map defines a line in space. • T(x, y, z) = (x, y) from R3 to R2 , A is the 2 × 3 matrix A =   1 0 0 1 0 0  . The map projects space onto a plane. • To the map T(x, y) = (x + y, x − y, 2x − 3y) belongs the 3 × 2 matrix A = 1 1 2 1 −1 −3 . The image of the map is a plane in three dimensional space. • If T(x) = x, then T is called the identity transformation. PROPERTIES OF LINEAR TRANSFORMATIONS. T(0) = 0 T(x + y) = T(x) + T(y) T(λx) = λT(x) In words: Linear transformations are compatible with addition and scalar multiplication. It does not matter, whether we add two vectors before the transformation or add the transformed vectors. ON LINEAR TRANSFORMATIONS. Linear transformations generalize the scaling transformation x → ax in one dimensions. They are important in • geometry (i.e. rotations, dilations, projections or reflections) • art (i.e. perspective, coordinate transformations), • CAD applications (i.e. projections), • physics (i.e. Lorentz transformations), • dynamics (linearizations of general maps are linear maps), • compression (i.e. using Fourier transform or Wavelet trans- form), • coding (many codes are linear codes), • probability (i.e. Markov processes). • linear equations (inversion is solving the equation)
  • 14. LINEAR TRANSFORMATION OR NOT? (The square to the right is the image of the square to the left): COLUMN VECTORS. A linear transformation T(x) = Ax with A =   | | · · · | v1 v2 · · · vn | | · · · |   has the property that the column vector v1, vi, vn are the images of the standard vectors e1 =       1 · · · 0       . ei =       0 · 1 · 0       . en =       0 · · · 1       . In order to find the matrix of a linear transformation, look at the image of the standard vectors and use those to build the columns of the matrix. QUIZ. a) Find the matrix belonging to the linear transformation, which rotates a cube around the diagonal (1, 1, 1) by 120 degrees (2π/3). b) Find the linear transformation, which reflects a vector at the line containing the vector (1, 1, 1). INVERSE OF A TRANSFORMATION. If S is a second transformation such that S(Tx) = x, for every x, then S is called the inverse of T. We will discuss this more later. SOLVING A LINEAR SYSTEM OF EQUATIONS. Ax = b means to invert the linear transformation x → Ax. If the linear system has exactly one solution, then an inverse exists. We will write x = A−1 b and see that the inverse of a linear transformation is again a linear transformation. THE BRETSCHER CODE. Otto Bretschers book contains as a motivation a ”code”, where the encryption happens with the linear map T(x, y) = (x+3y, 2x+ 5y). The map has the inverse T −1 (x, y) = (−5x + 3y, 2x − y). Cryptologists use often the following approach to crack a encryption. If one knows the input and output of some data, one often can decode the key. Assume we know, the enemy uses a Bretscher code and we know that T(1, 1) = (3, 5) and T(2, 1) = (7, 5). How do we get the code? The problem is to find the matrix A = a b c d . 2x2 MATRIX. It is useful to decode the Bretscher code in general If ax + by = X and cx + dy = Y , then x = (dX − bY )/(ad − bc), y = (cX − aY )/(ad − bc). This is a linear transformation with matrix A = a b c d and the corresponding matrix is A−1 = d −b −c a /(ad − bc). ”Switch diagonally, negate the wings and scale with a cross”.
  • 15. LINEAR TRAFOS IN GEOMETRY Math 21b, O. Knill LINEAR TRANSFORMATIONS DE- FORMING A BODY A CHARACTERIZATION OF LINEAR TRANSFORMATIONS: a transformation T from Rn to Rm which satisfies T(0) = 0, T(x + y) = T(x) + T(y) and T(λx) = λT(x) is a linear transformation. Proof. Call vi = T(ei) and define S(x) = Ax. Then S(ei) = T(ei). With x = x1e1 + ... + xnen, we have T(x) = T(x1e1 + ... + xnen) = x1v1 + ... + xnvn as well as S(x) = A(x1e1 + ... + xnen) = x1v1 + ... + xnvn proving T(x) = S(x) = Ax. SHEAR: A = 1 0 1 1 A = 1 1 0 1 In general, shears are transformation in the plane with the property that there is a vector w such that T(w) = w and T(x) − x is a multiple of w for all x. If u is orthogonal to w, then T(x) = x + (u · x)w. SCALING: A = 2 0 0 2 A = 1/2 0 0 1/2 One can also look at transformations which scale x differently then y and where A is a diagonal matrix. REFLECTION: A = cos(2α) sin(2α) sin(2α) − cos(2α) A = 1 0 0 −1 Any reflection at a line has the form of the matrix to the left. A reflection at a line containing a unit vector u is T(x) = 2(x · u)u − x with matrix A = 2u2 1 − 1 2u1u2 2u1u2 2u2 2 − 1 PROJECTION: A = 1 0 0 0 A = 0 0 0 1 A projection onto a line containing unit vector u is T(x) = (x · u)u with matrix A = u1u1 u2u1 u1u2 u2u2
  • 16. ROTATION: A = −1 0 0 −1 A = cos(α) − sin(α) sin(α) cos(α) Any rotation has the form of the matrix to the right. ROTATION-DILATION: A = 2 −3 3 2 A = a −b b a A rotation dilation is a composition of a rotation by angle arctan(y/x) and a dilation by a factor x2 + y2. If z = x + iy and w = a + ib and T(x, y) = (X, Y ), then X + iY = zw. So a rotation dilation is tied to the process of the multiplication with a complex number. BOOST: A = cosh(α) sinh(α) sinh(α) cosh(α) The boost is a basic Lorentz transformation in special relativity. It acts on vectors (x, ct), where t is time, c is the speed of light and x is space. Unlike in Galileo transformation (x, t) → (x + vt, t) (which is a shear), time t also changes during the transformation. The transformation has the effect that it changes length (Lorentz contraction). The angle α is related to v by tanh(α) = v/c. One can write also A(x, ct) = ((x + vt)/γ, t + (v/c2 )/γx), with γ = 1 − v2/c2. ROTATION IN SPACE. Rotations in space are defined by an axes of rotation and an angle. A rotation by 120◦ around a line containing (0, 0, 0) and (1, 1, 1) blongs to A =   0 0 1 1 0 0 0 1 0   which permutes e1 → e2 → e3. REFLECTION AT PLANE. To a reflection at the xy-plane belongs the matrix A =   1 0 0 0 1 0 0 0 −1   as can be seen by looking at the images of ei. The picture to the right shows the textbook and reflections of it at two different mirrors. PROJECTION ONTO SPACE. To project a 4d-object into xyz-space, use for example the matrix A =     1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0    . The picture shows the pro- jection of the four dimensional cube (tesseract, hypercube) with 16 edges (±1, ±1, ±1, ±1). The tesseract is the theme of the horror movie ”hypercube”.
  • 17. THE INVERSE Math 21b, O. Knill INVERTIBLE TRANSFORMATIONS. A map T from X to Y is invertible if there is for every y ∈ Y a unique point x ∈ X such that T(x) = y. ⇒ EXAMPLES. 1) T(x) = x3 is invertible from X = R to X = Y . 2) T(x) = x2 is not invertible from X = R to X = Y . 3) T(x, y) = (x2 + 3x − y, x) is invertible from X = R2 to Y = R2 . 4) T(x) = Ax linear and rref(A) has an empty row, then T is not invertible. 5) If T(x) = Ax is linear and rref(A) = 1n, then T is invertible. INVERSE OF LINEAR TRANSFORMATION. If A is a n × n matrix and T : x → Ax has an inverse S, then S is linear. The matrix A−1 belonging to S = T−1 is called the inverse matrix of A. First proof: check that S is linear using the characterization S(a + b) = S(a) + S(b), S(λa) = λS(a) of linearity. Second proof: construct the inverse using Gauss-Jordan elimination. FINDING THE INVERSE. Let 1n be the n × n identity matrix. Start with [A|1n] and perform Gauss-Jordan elimination. Then rref([A|1n]) = 1n|A−1 Proof. The elimination process actually solves Ax = ei simultaneously. This leads to solutions vi which are the columns of the inverse matrix A−1 because A−1 ei = vi. EXAMPLE. Find the inverse of A = 2 6 1 4 . 2 6 | 1 0 1 4 | 0 1 A | 12 1 3 | 1/2 0 1 4 | 0 1 .... | ... 1 3 | 1/2 0 0 1 | −1/2 1 .... | ... 1 0 | 2 −3 0 1 | −1/2 1 12 | A−1 The inverse is A−1 = 2 −3 −1/2 1 . THE INVERSE OF LINEAR MAPS R2 → R2 : If ad−bc = 0, the inverse of a linear transformation x → Ax with A = a b c d is given by the matrix A−1 = d −b −d a /(ad − bc). SHEAR: A = 1 0 −1 1 A−1 = 1 0 1 1
  • 18. DIAGONAL: A = 2 0 0 3 A−1 = 1/2 0 0 1/3 REFLECTION: A = cos(2α) sin(2α) sin(2α) − cos(2α) A−1 = A = cos(2α) sin(2α) sin(2α) − cos(2α) ROTATION: A = cos(α) sin(α) − sin(α) cos(−α) A−1 = cos(α) − sin(α) sin(α) cos(α) ROTATION-DILATION: A = a −b b a A−1 = a/r2 b/r2 −b/r2 a/r2 , r2 = a2 + b2 BOOST: A = cosh(α) sinh(α) sinh(α) cosh(α) A−1 = cosh(α) − sinh(α) − sinh(α) cosh(α) NONINVERTIBLE EXAMPLE. The projection x → Ax = 1 0 0 0 is a non-invertible transformation. MORE ON SHEARS. The shears T(x, y) = (x + ay, y) or T(x, y) = (x, y + ax) in R2 can be generalized. A shear is a linear transformation which fixes some line L through the origin and which has the property that T(x) − x is parallel to L for all x. PROBLEM. T(x, y) = (3x/2 + y/2, y/2 − x/2) is a shear along a line L. Find L. SOLUTION. Solve the system T(x, y) = (x, y). You find that the vector (1, −1) is preserved. MORE ON PROJECTIONS. A linear map T with the property that T(T(x)) = T(x) is a projection. Examples: T(x) = (y · x)y is a projection onto a line spanned by a unit vector y. WHERE DO PROJECTIONS APPEAR? CAD: describe 3D objects using projections. A photo of an image is a projection. Compression algorithms like JPG or MPG or MP3 use projections where the high frequencies are cut away. MORE ON ROTATIONS. A linear map T which preserves the angle between two vectors and the length of each vector is called a rotation. Rotations form an important class of transformations and will be treated later in more detail. In two dimensions, every rotation is of the form x → A(x) with A = cos(φ) − sin(φ) sin(φ) cos(φ) . An example of a rotations in three dimensions are x → Ax, with A =   cos(φ) − sin(φ) 0 sin(φ) cos(φ) 0 0 0 1  . it is a rotation around the z axis. MORE ON REFLECTIONS. Reflections are linear transformations different from the identity which are equal to their own inverse. Examples: 2D reflections at the origin: A = −1 0 0 1 , 2D reflections at a line A = cos(2φ) sin(2φ) sin(2φ) − cos(2φ) . 3D reflections at origin: A =   −1 0 0 0 −1 0 0 0 −1  . 3D reflections at a line A =   −1 0 0 0 −1 0 0 0 1  . By the way: in any dimensions, to a reflection at the line containing the unit vector u belongs the matrix [A]ij = 2(uiuj) − [1n]ij, because [B]ij = uiuj is the matrix belonging to the projection onto the line. The reflection at a line containng the unit vector u = [u1, u2, u3] is A =   u2 1 − 1 u1u2 u1u3 u2u1 u2 2 − 1 u2u3 u3u1 u3u2 u2 3 − 1  . 3D reflection at a plane A =   1 0 0 0 1 0 0 0 −1  . Reflections are important symmetries in physics: T (time reflection), P (reflection at a mirror), C (change of charge) are reflections. It seems today that the composition of TCP is a fundamental symmetry in nature.
  • 19. MATRIX PRODUCT Math 21b, O. Knill MATRIX PRODUCT. If B is a m × n matrix and A is a n × p matrix, then BA is defined as the m × p matrix with entries (BA)ij = n k=1 BikAkj. EXAMPLE. If B is a 3 × 4 matrix, and A is a 4 × 2 matrix then BA is a 3 × 2 matrix. B =   1 3 5 7 3 1 8 1 1 0 9 2  , A =     1 3 3 1 1 0 0 1    , BA =   1 3 5 7 3 1 8 1 1 0 9 2       1 3 3 1 1 0 0 1     =   15 13 14 11 10 5  . COMPOSING LINEAR TRANSFORMATIONS. If S : Rm → Rn , x → Ax and T : Rn → Rp , y → By are linear transformations, then their composition T ◦ S : x → B(A(x)) is a linear transformation from Rm to Rp . The corresponding matrix is the matrix product BA. EXAMPLE. Find the matrix which is a composition of a rotation around the x-axes by an agle π/2 followed by a rotation around the z-axes by an angle π/2. SOLUTION. The first transformation has the property that e1 → e1, e2 → e3, e3 → −e2, the second e1 → e2, e2 → −e1, e3 → e3. If A is the matrix belonging to the first transformation and B the second, then BA is the matrix to the composition. The composition maps e1 → −e2 → e3 → e1 is a rotation around a long diagonal. B =   0 −1 0 1 0 0 0 0 1   A =   1 0 0 0 0 −1 0 1 0  , BA =   0 0 1 1 0 0 0 1 0  . EXAMPLE. A rotation dilation is the composition of a rotation by α = arctan(b/a) and a dilation (=scale) by r = √ a2 + b2. REMARK. Matrix multiplication is a generalization of usual multiplication of numbers or the dot product. MATRIX ALGEBRA. Note that AB = BA in general! Otherwise, the same rules apply as for numbers: A(BC) = (AB)C, AA−1 = A−1 A = 1n, (AB)−1 = B−1 A−1 , A(B + C) = AB + AC, (B + C)A = BA + CA etc. PARTITIONED MATRICES. The entries of matrices can themselves be matrices. If B is a m × n matrix and A is a n × p matrix, and assume the entries are k × k matrices, then BA is a m × p matrix where each entry (BA)ij = n k=1 BikAkj is a k × k matrix. Partitioning matrices can be useful to improve the speed of matrix multiplication (i.e. Strassen algorithm). EXAMPLE. If A = A11 A12 0 A22 , where Aij are k × k matrices with the property that A11 and A22 are invertible, then B = A−1 11 −A−1 11 A12A−1 22 0 A−1 22 is the inverse of A. APPLICATIONS. (The material which follows is for motivation puposes only, more applications appear in the homework). 1 4 2 3 NETWORKS. Let us associate to the computer network (shown at the left) a matrix     0 1 1 1 1 0 1 0 1 1 0 1 1 0 1 0     To a worm in the first computer we associate a vector     1 0 0 0    . The vector Ax has a 1 at the places, where the worm could be in the next step. The vector (AA)(x) tells, in how many ways the worm can go from the first computer to other hosts in 2 steps. In our case, it can go in three different ways back to the computer itself. Matrices help to solve combinatorial problems (see movie ”Good will hunting”). For example, what does [A1000 ]22 tell about the worm infection of the network? What does it mean if A100 has no zero entries?
  • 20. FRACTALS. Closely related to linear maps are affine maps x → Ax + b. They are compositions of a linear map with a translation. It is not a linear map if B(0) = 0. Affine maps can be disguised as linear maps in the following way: let y = x 1 and defne the (n+1)∗(n+1) matrix B = A b 0 1 . Then By = Ax + b 1 . Fractals can be constructed by taking for example 3 affine maps R, S, T which contract area. For a given object Y0 define Y1 = R(Y0) ∪ S(Y0) ∪ T(Y0) and recursively Yk = R(Yk−1) ∪ S(Yk−1) ∪ T(Yk−1). The above picture shows Yk after some iterations. In the limit, for example if R(Y0), S(Y0) and T(Y0) are disjoint, the sets Yk converge to a fractal, an object with dimension strictly between 1 and 2. CHAOS. Consider a map in the plane like T : x y → 2x + 2 sin(x) − y x We apply this map again and again and follow the points (x1, y1) = T(x, y), (x2, y2) = T(T(x, y)), etc. One writes T n for the n-th iteration of the map and (xn, yn) for the image of (x, y) under the map T n . The linear approximation of the map at a point (x, y) is the matrix DT(x, y) = 2 + 2 cos(x) − 1 1 . (If T x y = f(x, y) g(x, y) , then the row vectors of DT(x, y) are just the gradients of f and g). T is called chaotic at (x, y), if the entries of D(T n )(x, y) grow exponentially fast with n. By the chain rule, D(T n ) is the product of matrices DT(xi, yi). For example, T is chaotic at (0, 0). If there is a positive probability to hit a chaotic point, then T is called chaotic. FALSE COLORS. Any color can be represented as a vector (r, g, b), where r ∈ [0, 1] is the red g ∈ [0, 1] is the green and b ∈ [0, 1] is the blue component. Changing colors in a picture means applying a transformation on the cube. Let T : (r, g, b) → (g, b, r) and S : (r, g, b) → (r, g, 0). What is the composition of these two linear maps? OPTICS. Matrices help to calculate the motion of light rays through lenses. A light ray y(s) = x + ms in the plane is described by a vector (x, m). Following the light ray over a distance of length L corresponds to the map (x, m) → (x + mL, m). In the lens, the ray is bent depending on the height x. The transformation in the lens is (x, m) → (x, m − kx), where k is the strength of the lense. x m → AL x m = 1 L 0 1 x m , x m → Bk x m = 1 0 −k 1 x m . Examples: 1) Eye looking far: ARBk. 2) Eye looking at distance L: ARBkAL. 3) Telescope: Bk2 ALBk1 . (More about it in problem 80 in section 2.4).
  • 21. IMAGE AND KERNEL Math 21b, O. Knill IMAGE. If T : Rn → Rm is a linear transformation, then {T(x) | x ∈ Rn } is called the image of T. If T(x) = Ax, then the image of T is also called the image of A. We write im(A) or im(T). EXAMPLES. 1) If T(x, y, z) = (x, y, 0), then T(x) = A   x y z   =   1 0 0 0 1 0 0 0 0     x y z  . The image of T is the x − y plane. 2) If T(x, y)(cos(φ)x − sin(φ)y, sin(φ)x + cos(φ)y) is a rotation in the plane, then the image of T is the whole plane. 3) If T(x, y, z) = x + y + z, then the image of T is R. SPAN. The span of vectors v1, . . . , vk in Rn is the set of all combinations c1v1 + . . . ckvk, where ci are real numbers. PROPERTIES. The image of a linear transformation x → Ax is the span of the column vectors of A. The image of a linear transformation contains 0 and is closed under addition and scalar multiplication. KERNEL. If T : Rn → Rm is a linear transformation, then the set {x | T(x) = 0 } is called the kernel of T. If T(x) = Ax, then the kernel of T is also called the kernel of A. We write ker(A) or ker(T). EXAMPLES. (The same examples as above) 1) The kernel is the z-axes. Every vector (0, 0, z) is mapped to 0. 2) The kernel consists only of the point (0, 0, 0). 3) The kernel consists of all vector (x, y, z) for which x + y + z = 0. The kernel is a plane. PROPERTIES. The kernel of a linear transformation contains 0 and is closed under addition and scalar multiplication. IMAGE AND KERNEL OF INVERTIBLE MAPS. A linear map x → Ax, Rn → Rn is invertible if and only if ker(A) = {0} if and only if im(A) = Rn . HOW DO WE COMPUTE THE IMAGE? The rank of rref(A) is the dimension of the image. The column vectors of A span the image. (Dimension will be discussed later in detail). EXAMPLES. (The same examples as above) 1)   1 0 0   and   0 1 0   span the image. 2) cos(φ) − sin(φ) and sin(φ) cos(φ) span the image. 3) The 1D vector 1 spans the image. HOW DO WE COMPUTE THE KERNEL? Just solve Ax = 0. Form rref(A). For every column without leading 1 we can introduce a free variable si. If x is the solution to Axi = 0, where all sj are zero except si = 1, then x = j sjxj is a general vector in the kernel. EXAMPLE. Find the kernel of the linear map R3 → R4 , x → Ax with A =     1 3 0 2 6 5 3 9 1 −2 −6 0    . Gauss-Jordan elimination gives: B = rref(A) =     1 3 0 0 0 1 0 0 0 0 0 0    . We see one column without leading 1 (the second one). The equation Bx = 0 is equivalent to the system x + 3y = 0, z = 0. After fixing z = 0, can chose y = t freely and obtain from the first equation x = −3t. Therefore, the kernel consists of vectors t   −3 1 0  . In the book, you have a detailed calculation, in a case, where the kernel is 2 dimensional.
  • 22. domain codomain kernel image WHY DO WE LOOK AT THE KERNEL? • It is useful to understand linear maps. To which degree are they non-invertible? • Helpful to understand quantitatively how many solutions a linear equation Ax = b has. If x is a solution and y is in the kernel of A, then also A(x + y) = b, so that x + y solves the system also. WHY DO WE LOOK AT THE IMAGE? • A solution Ax = b can be solved if and only if b is in the image of A. • Knowing about the kernel and the image is use- ful in the similar way that it is useful to know about the domain and range of a general map and to understand the graph of the map. In general, the abstraction helps to understand topics like error correcing codes (Problem 53/54 in Bretschers book), where two matrices H, M with the property that ker(H) = im(M) appear. The encoding x → Mx is robust in the sense that adding an error e to the result Mx → Mx + e can be corrected: H(Mx + e) = He allows to find e and so Mx. This allows to recover x = PMx with a projection P. PROBLEM. Find ker(A) and im(A) for the 1 × 3 matrix A = [5, 1, 4], a row vector. ANSWER. A · x = Ax = 5x + y + 4z = 0 shows that the kernel is a plane with normal vector [5, 1, 4] through the origin. The image is the codomain, which is R. PROBLEM. Find ker(A) and im(A) of the linear map x → v × x, (the cross product with v. ANSWER. The kernel consists of the line spanned by v, the image is the plane orthogonal to v. PROBLEM. Fix a vector w in space. Find ker(A) and image im(A) of the linear map from R6 to R3 given by x, y → [x, v, y] = (x × y) · w. ANSWER. The kernel consist of all (x, y) such that their cross product orthogonal to w. This means that the plane spanned by x, y contains w. PROBLEM Find ker(T) and im(T) if T is a composition of a rotation R by 90 degrees around the z-axes with with a projection onto the x-z plane. ANSWER. The kernel of the projection is the y axes. The x axes is rotated into the y axes and therefore the kernel of T. The image is the x-z plane. PROBLEM. Can the kernel of a square matrix A be trivial if A2 = 0, where 0 is the matrix containing only 0? ANSWER. No: if the kernel were trivial, then A were invertible and A2 were invertible and be different from 0. PROBLEM. Is it possible that a 3 × 3 matrix A satisfies ker(A) = R3 without A = 0? ANSWER. No, if A = 0, then A contains a nonzero entry and therefore, a column vector which is nonzero. PROBLEM. What is the kernel and image of a projection onto the plane Σ : x − y + 2z = 0? ANSWER. The kernel consists of all vectors orthogonal to Σ, the image is the plane Σ. PROBLEM. Given two square matrices A, B and assume AB = BA. You know ker(A) and ker(B). What can you say about ker(AB)? ANSWER. ker(A) is contained in ker(BA). Similar ker(B) is contained in ker(AB). Because AB = BA, the kernel of AB contains both ker(A) and ker(B). (It can be bigger: A = B = 0 1 0 0 .) PROBLEM. What is the kernel of the partitioned matrix A 0 0 B if ker(A) and ker(B) are known? ANSWER. The kernel consists of all vectors (x, y), where x in ker(A) and y ∈ ker(B).
  • 23. BASIS Math 21b, O. Knill LINEAR SUBSPACE. A subset X of Rn which is closed under addition and scalar multiplication is called a linear subspace of Rn . WHICH OF THE FOLLOWING SETS ARE LINEAR SPACES? a) The kernel of a linear map. b) The image of a linear map. c) The upper half plane. d) the line x + y = 0. e) The plane x + y + z = 1. f) The unit circle. BASIS. A set of vectors v1, . . . , vm is a basis of a linear subspace X of Rn if they are linear independent and if they span the space X. Linear independent means that there are no nontrivial linear relations aiv1 + . . . + amvm = 0. Spanning the space means that very vector v can be written as a linear combination v = a1v1 +. . .+amvm of basis vectors. A linear subspace is a set containing {0} which is closed under addition and scaling. EXAMPLE 1) The vectors v1 =   1 1 0   , v2 =   0 1 1   , v3 =   1 0 1   form a basis in the three dimensional space. If v =   4 3 5  , then v = v1 + 2v2 + 3v3 and this representation is unique. We can find the coefficients by solving Ax = v, where A has the vi as column vectors. In our case, A =   1 0 1 1 1 0 0 1 1     x y z   =   4 3 5   had the unique solution x = 1, y = 2, z = 3 leading to v = v1 + 2v2 + 3v3. EXAMPLE 2) Two nonzero vectors in the plane which are not parallel form a basis. EXAMPLE 3) Three vectors in R3 which are in a plane form not a basis. EXAMPLE 4) Two vectors in R3 do not form a basis. EXAMPLE 5) Three nonzero vectors in R3 which are not contained in a single plane form a basis in R3 . EXAMPLE 6) The columns of an invertible n × n matrix form a basis in Rn as we will see. FACT. If v1, ..., vn is a basis, then every vector v can be represented uniquely as a linear combination of the vi. v = a1v1 + . . . + anvn. REASON. There is at least one representation because the vectors vi span the space. If there were two different representations v = a1v1 +. . .+anvn and v = b1v1 +. . .+bnvn, then subtraction would lead to 0 = (a1 − b1)v1 + ... + (an − bn)vn. This nontrivial linear relation of the vi is forbidden by assumption. FACT. If n vectors v1, ..., vn span a space and w1, ..., wm are linear independent, then m ≤ n. REASON. This is intuitively clear in dimensions up to 3. You can not have more then 4 vectors in space which are linearly independent. We will give a precise reason later. A BASIS DEFINES AN INVERTIBLE MATRIX. The n × n matrix A =   | | | v1 v2 . . . vn | | |   is invertible if and only if v1, . . . , vn define a basis in Rn . EXAMPLE. In the example 1), the 3 × 3 matrix A is invertible. FINDING A BASIS FOR THE KERNEL. To solve Ax = 0, we bring the matrix A into the reduced row echelon form rref(A). For every non-leading entry in rref(A), we will get a free variable ti. Writing the system Ax = 0 with these free variables gives us an equation x = i tivi. The vectors vi form a basis of the kernel of A.
  • 24. REMARK. The problem to find a basis for all vectors wi which are orthogonal to a given set of vectors, is equivalent to the problem to find a basis for the kernel of the matrix which has the vectors wi in its rows. FINDING A BASIS FOR THE IMAGE. Bring the m × n matrix A into the form rref(A). Call a column a pivot column, if it contains a leading 1. The corresponding set of column vectors of the original matrix A form a basis for the image because they are linearly independent and are in the image. Assume there are k of them. They span the image because there are (k − n) non-leading entries in the matrix. REMARK. The problem to find a basis of the subspace generated by v1, . . . , vn, is the problem to find a basis for the image of the matrix A with column vectors v1, ..., vn. EXAMPLES. 1) Two vectors on a line are linear dependent. One is a multiple of the other. 2) Three vectors in the plane are linear dependent. One can find a relation av1 + bv2 = v3 by changing the size of the lengths of the vectors v1, v2 until v3 becomes the diagonal of the parallelogram spanned by v1, v2. 3) Four vectors in three dimensional space are linearly dependent. As in the plane one can change the length of the vectors to make v4 a diagonal of the parallelepiped spanned by v1, v2, v3. EXAMPLE. Let A be the matrix A = 0 0 1 1 1 0 1 1 1 . In reduced row echelon form is B = rref(A) = 1 1 0 0 0 1 0 0 0 . To determine a basis of the kernel we write Bx = 0 as a system of linear equations: x + y = 0, z = 0. The variable y is the free variable. With y = t, x = −t is fixed. The linear system rref(A)x = 0 is solved by x =   x y z   = t   −1 1 0  . So, v =   −1 1 0   is a basis of the kernel. EXAMPLE. Because the first and third vectors in rref(A) are columns with leading 1’s, the first and third columns v1 =   0 1 1   , v2 =   1 0 1   of A form a basis of the image of A. WHY DO WE INTRODUCE BASIS VECTORS? Wouldn’t it be just easier to look at the standard basis vectors e1, . . . , en only? The rea- son for more general basis vectors is that they allow a more flexible adaptation to the situation. A person in Paris prefers a different set of basis vectors than a person in Boston. We will also see that in many applications, problems can be solved easier with the right basis. For example, to describe the reflection of a ray at a plane or at a curve, it is preferable to use basis vec- tors which are tangent or orthogonal. When looking at a rotation, it is good to have one basis vector in the axis of rotation, the other two orthogonal to the axis. Choosing the right basis will be especially im- portant when studying differential equations. A PROBLEM. Let A =   1 2 3 1 1 1 0 1 2  . Find a basis for ker(A) and im(A). SOLUTION. From rref(A) = 1 0 −1 0 1 2 0 0 0 we see that = v =   1 −2 1   is in the kernel. The two column vectors   1 1 0   , 2, 1, 1 of A form a basis of the image because the first and third column are pivot columns.
  • 25. DIMENSION Math 21b, O. Knill REVIEW LINEAR SPACE. X linear space: 0 ∈ X closed under addition and scalar multiplication. Examples: Rn , X = ker(A), X = im(A) are linear spaces. REVIEW BASIS. B = {v1, . . . , vn} ⊂ X B linear independent: c1v1 +...+cnvn = 0 implies c1 = ... = cn = 0. B span X: v ∈ X then v = a1v1 + ... + anvn. B basis: both linear independent and span. BASIS: ENOUGH BUT NOT TOO MUCH. The spanning condition for a basis assures that there are enough vectors to represent any other vector, the linear independence condi- tion assures that there are not too many vectors. A basis is, where J.Lo meets A.Hi: Left: J.Lopez in ”Enough”, right ”The man who new too much” by A.Hitchcock AN UNUSUAL EXAMPLE. Let X be the space of polynomials up to degree 4. For example p(x) = 3x4 + 2x3 + x + 5 is an element in this space. It is straightforward to check that X is a linear space. The ”zero vector” is the function f(x) = 0 which is zero everywhere. We claim that e1(x) = 1, e2(x) = x, e3(x) = x2 , e4(x) = x3 and e5(x) = x4 form a basis in X. PROOF. The vectors span the space: every polynomial f(x) = c0 + c1x + c2x2 + c3x3 + c4x4 is a sum f = c0e1 + c1e2 + c2e3 + c3e4 + c4e5 of basis elements. The vectors are linearly independent: a nontrivial relation 0 = c0e1 + c1e2 + c2e3 + c3e4 + c4e5 would mean that c0 + c1x + c2x2 + c3x3 + c4x4 = 0 for all x which is not possible unless all cj are zero. DIMENSION. The number of elements in a basis of X is independent of the choice of the basis. It is called the dimension of X. UNIQUE REPRESENTATION. v1, . . . , vn ∈ X ba- sis ⇒ every v ∈ X can be written uniquely as a sum v = a1v1 + . . . + anvn. EXAMPLES. The dimension of {0} is zero. The dimension of any line 1. The dimension of a plane is 2, the dimension of three dimensional space is 3. The dimension is independent on where the space is embedded in. For example: a line in the plane and a line in space have dimension 1. IN THE UNUSUAL EXAMPLE. The set of polynomials of degree ≤ 4 form a linear space of dimension 5. REVIEW: KERNEL AND IMAGE. We can construct a basis of the kernel and image of a linear transformation T(x) = Ax by forming B = rrefA. The set of Pivot columns in A form a basis of the image of T, a basis for the kernel is obtained by solving Bx = 0 and introducing free variables for each non-pivot column. EXAMPLE. Let X the linear space from above. Define the linear transformation T(f)(x) = f (x). For example: T(x3 + 2x4 ) = 3x2 + 8x3 . Find a basis for the kernel and image of this transformation. SOLUTION. Because T(e1) = 0, T(e2) = e1, T(e3) = 2e2, T(e4) = 3e3, T(e5) = 4e4, the matrix of T is A =       0 1 0 0 0 0 0 2 0 0 0 0 0 3 0 0 0 0 0 4 0 0 0 0 0       which is almost in row reduced echelon form. You see that the last four columns are pivot columns. The kernel is spanned by e1 which corresponds to the con- stant function f(x) = 1. The image is the 4 dimensional space of polynomials of degree ≤ 3. Mathematicians call a fact a ”lemma” if it is used to prove a theorem and if does not deserve the be honored by the name ”theorem”:
  • 26. LEMMA. If n vectors v1, ..., vn span a space and w1, ..., wm are linearly independent, then m ≤ n. REASON. Assume m > n. Because vi span, each vector wi can be written as j aijvj = wi. After doing Gauss-Jordan elimination of the augmented (m × (n + 1))-matrix a11 . . . a1n | w1 . . . . . . . . . | . . . am1 . . . amn | wm which belong to these relations, we end up with a matrix, which has in the last line 0 ... 0 | b1w1 + ... + bmwm . But this means that b1w1 + ... + bmwm = 0 and this is a nontrivial relation between the wi. Contradiction. The assumption m > n was absurd. THEOREM. Given a basis A = {v1, ..., vn } and a basis B = {w1, ..., .wm } of X, then m = n. PROOF. Because A spans X and B is linearly independent, we know that n ≤ m. Because B spans X and A is linearly independent also m ≤ n holds. Together, n ≤ m and m ≤ n implies n = m. DIMENSION OF THE KERNEL. The number of columns in rref(A) without leading 1’s is the dimension of the kernel dim(ker(A)): we can introduce a parameter for each such col- umn when solving Ax = 0 using Gauss-Jordan elimination. DIMENSION OF THE IMAGE. The number of leading 1 in rref(A), the rank of A is the dimension of the image dim(im(A)) because every such leading 1 produces a differ- ent column vector (called pivot column vectors) and these column vectors are linearly independent. 1 1 1 1 * * * * DIMENSION FORMULA: (A : Rn → Rm ) dim(ker(A)) + dim(im(A)) = n EXAMPLE: A invertible is equivalant that the dimension of the image is n and that the dim(ker)(A) = 0. PROOF. There are n columns. dim(ker(A)) is the number of columns without leading 1, dim(im(A)) is the number of columns with leading 1. EXAMPLE. In the space X of polynomials of degree 4 define T(f)(x) = f (x). The kernel consists of linear polynomials spanned by e1, e2, the image consists of all polynomials of degree ≤ 2. It is spanned by e3, e4, e5. Indeed dim(ker(T)) + dim(im(T)) = 2 + 3 = 5 = n. FRACTAL DIMENSION. Mathematicians study objects with non-integer dimension since the early 20’th cen- tury. The topic became fashion in the 80’ies, when people started to generate fractals on computers. To define fractals, the notion of dimension is extended: define a s-volume of accuracy r of a bounded set X in Rn as the infimum of all hs,r(X) = Uj |Uj|s , where Uj are cubes of length ≤ r covering X and |Uj| is the length of Uj. The s-volume is then defined as the limit hs(X) of hs(X) = hs,r(X) when r → 0. The dimension is the limiting value s, where hs(X) jumps from 0 to ∞. Examples: 1) A smooth curve X of length 1 in the plane can be covered with n squares Uj of length |Uj| = 1/n and hs,1/n(X) = n j=1(1/n)s = n(1/n)s . If s < 1, this converges, if s > 1 it diverges for n → ∞. So dim(X) = 1. 2) A square X in space of area 1 can be covered with n2 cubes Uj of length |Uj| = 1/n and hs,1/n(X) = n2 j=1(1/n)s = n2 (1/n)s which converges to 0 for s < 2 and diverges for s > 2 so that dim(X) = 2. 3) The Shirpinski carpet is constructed recur- sively by dividing a square in 9 equal squares and cutting away the middle one, repeating this proce- dure with each of the squares etc. At the k’th step, we need 8k squares of length 1/3k to cover the car- pet. The s−volume hs,1/3k (X) of accuracy 1/3k is 8k (1/3k )s = 8k /3ks , which goes to 0 for k → ∞ if 3ks < 8k or s < d = log(8)/ log(3) and diverges if s > d. The dimension is d = log(8)/ log(3) = 1.893.. INFINITE DIMENSIONS. Linear spaces also can have infinite dimensions. An example is the set X of all continuous maps from the real R to R. It contains all polynomials and because Xn the space of polynomials of degree n with dimension n + 1 is contained in X, the space X is infinite dimensional. By the way, there are functions like g(x) = ∞ n=0 sin(2n x)/2n in X which have graphs of fractal dimension > 1 and which are not differentiable at any point x.
  • 27. COORDINATES Math 21b, O. Knill B-COORDINATES. Given a basis v1, . . . vn, define the matrix S =   | . . . | v1 . . . vn | . . . |  . It is invertible. If x = i civi, then ci are called the B-coordinates of v. We write [x]B =   c1 . . . cn  . If x =   x1 . . . xn  , we have x = S([x]B). B-coordinates of x are obtained by applying S−1 to the coordinates of the standard basis: [x]B = S−1 (x) . EXAMPLE. If v1 = 1 2 and v2 = 3 5 , then S = 1 3 2 5 . A vector v = 6 9 has the coordinates S−1 v = −5 3 2 −1 6 9 = −3 3 Indeed, as we can check, −3v1 + 3v2 = v. EXAMPLE. Let V be the plane x + y − z = 1. Find a basis, in which every vector in the plane has the form  a b 0  . SOLUTION. Find a basis, such that two vectors v1, v2 are in the plane and such that a third vector v3 is linearly independent to the first two. Since (1, 0, 1), (0, 1, 1) are points in the plane and (0, 0, 0) is in the plane, we can choose v1 =   1 0 1   v2 =   0 1 1   and v3 =   1 1 −1   which is perpendicular to the plane. EXAMPLE. Find the coordinates of v = 2 3 with respect to the basis B = {v1 = 1 0 , v2 = 1 1 }. We have S = 1 1 0 1 and S−1 = 1 −1 0 1 . Therefore [v]B = S−1 v = −1 3 . Indeed −1v1 + 3v2 = v. B-MATRIX. If B = {v1, . . . , vn} is a basis in Rn and T is a linear transformation on Rn , then the B-matrix of T is defined as B =   | . . . | [T(v1)]B . . . [T(vn)]B | . . . |   COORDINATES HISTORY. Cartesian geometry was introduced by Fermat and Descartes (1596-1650) around 1636. It had a large influence on mathematics. Algebraic methods were introduced into geometry. The beginning of the vector concept came only later at the beginning of the 19’th Century with the work of Bolzano (1781-1848). The full power of coordinates becomes possible if we allow to chose our coordinate system adapted to the situation. Descartes biography shows how far dedication to the teaching of mathematics can go ...: (...) In 1649 Queen Christina of Sweden persuaded Descartes to go to Stockholm. However the Queen wanted to draw tangents at 5 a.m. in the morning and Descartes broke the habit of his lifetime of getting up at 11 o’clock. After only a few months in the cold northern climate, walking to the palace at 5 o’clock every morning, he died of pneumonia. Fermat Descartes Christina Bolzano
  • 28. CREATIVITY THROUGH LAZINESS? Legend tells that Descartes (1596-1650) introduced coordinates while lying on the bed, watching a fly (around 1630), that Archimedes (285-212 BC) discovered a method to find the volume of bodies while relaxing in the bath and that Newton (1643-1727) discovered New- ton’s law while lying under an apple tree. Other examples are August Kekul´e’s analysis of the Benzene molecular structure in a dream (a snake biting in its tail revieled the ring structure) or Steven Hawkings discovery that black holes can radiate (while shaving). While unclear which of this is actually true, there is a pattern: According David Perkins (at Harvard school of education): ”The Eureka effect”, many creative breakthroughs have in common: a long search without apparent progress, a prevailing moment and break through, and finally, a transformation and realization. A breakthrough in a lazy moment is typical - but only after long struggle and hard work. EXAMPLE. Let T be the reflection at the plane x + 2y + 3z = 0. Find the transformation matrix B in the basis v1 =   2 −1 0   v2 =   1 2 3   v3 =   0 3 −2  . Because T(v1) = v1 = [e1]B, T(v2) = v2 = [e2]B, T(v3) = −v3 = −[e3]B, the solution is B =   1 0 0 0 −1 0 0 0 1  . SIMILARITY. The B matrix of A is B = S−1 AS, where S =   | . . . | v1 . . . vn | . . . |  . One says B is similar to A. EXAMPLE. If A is similar to B, then A2 +A+1 is similar to B2 +B+1. B = S−1 AS, B2 = S−1 B2 S, S−1 S = 1, S−1 (A2 + A + 1)S = B2 + B + 1. PROPERTIES OF SIMILARITY. A, B similar and B, C similar, then A, C are similar. If A is similar to B, then B is similar to A. QUIZZ: If A is a 2 × 2 matrix and let S = 0 1 1 0 , What is S−1 AS? MAIN IDEA OF CONJUGATION S. The transformation S−1 maps the coordinates from the standard basis into the coordinates of the new basis. In order to see what a transformation A does in the new coordinates, we map it back to the old coordinates, apply A and then map it back again to the new coordinates: B = S−1 AS. The transformation in standard coordinates. v S ← w = [v]B A ↓ ↓ B Av S−1 → Bw The transformation in B-coordinates. QUESTION. Can the matrix A which belongs to a projection from R3 to a plane x + y + 6z = 0 be similar to a matrix which is a rotation by 20 degrees around the z axis? No: a non-invertible A can not be similar to an invertible B: if it were, the inverse A = SBS−1 would exist: A−1 = SB−1 S−1 . PROBLEM. Find a clever basis for the reflection of a light ray at the line x+2y = 0. v1 = 1 2 , v2 = −2 1 . SOLUTION. You can achieve B = 1 0 0 −1 with S = 1 −2 2 1 . PROBLEM. Are all shears A = 1 a 0 1 with a = 0 similar? Yes, use a basis v1 = ae1 and v2 = e2. PROBLEM. You know A = 3 0 −1 2 is similar to B = 1 0 0 −1 with S = 0 −1 1 1 . Find eA = 1+A+A2 +A3 /3!+.... SOLUTION. Because Bk = S−1 Ak S for every k we have eA = SeB S−1 = 1/e 0 e + 1/e e .
  • 29. ORTHOGONALITY Math 21b, O. Knill ORTHOGONALITY. v and w are called orthogonal if v · w = 0. Examples. 1) 1 2 and 6 −3 are orthogonal in R2 . 2) v and w are both orthogonal to v × w in R3 . v is called a unit vector if ||v|| = √ v · v = 1. B = {v1, . . . , vn} are called orthogonal if they are pairwise orthogonal. They are called orthonormal if they are also unit vectors. A basis is called an orthonormal basis if it is orthonormal. For an orthonormal basis, the matrix Aij = vi · vj is the unit matrix. FACT. Orthogonal vectors are linearly independent and n orthogonal vectors in Rn form a basis. Proof. The dot product of a linear relation a1v1 + . . . + anvn = 0 with vk gives akvk · vk = ak||vk||2 = 0 so that ak = 0. If we have n linear independent vectors in Rn then they automatically span the space. ORTHOGONAL COMPLEMENT. A vector w ∈ Rn is called orthogonal to a linear space V if w is orthogonal to every vector in v ∈ V . The orthogonal complement of a linear space V is the set W of all vectors which are orthogonal to V . It forms a linear space because v · w1 = 0, v · w2 = 0 implies v · (w1 + w2) = 0. ORTHOGONAL PROJECTION. The orthogonal projection onto a linear space V with orthnormal basis v1, . . . , vn is the linear map T(x) = projV (x) = (v1 · x)v1 + . . . + (vn · x)vn The vector x − projV (x) is in the orthogonal complement of V . (Note that vi in the projection formula are unit vectors, they have also to be orthogonal.) SPECIAL CASE. For an orthonormal basis vi, one can write x = (v1 · x)v1 + ... + (vn · x)vn . PYTHAGORAS: If x and y are orthogonal, then ||x + y||2 = ||x||2 + ||y||2 . Proof. Expand (x + y) · (x + y). PROJECTIONS DO NOT INCREASE LENGTH: ||projV(x)|| ≤ ||x||. Proof. Use Pythagoras: on x = projV(x) + (x − projV(x))). If ||projV(x)|| = ||x||, then x is in V . CAUCHY-SCHWARTZ INEQUALITY: |x · y| ≤ ||x|| ||y|| . Proof: x · y = ||x||||y|| cos(α). If |x · y| = ||x||||y||, then x and y are parallel. TRIANGLE INEQUALITY: ||x + y|| ≤ ||x|| + ||y||. Proof: (x + y) · (x + y) = ||x||2 + ||y||2 + 2x · y ≤ ||x||2 + ||y||2 + 2||x||||y|| = (||x|| + ||y||)2 . ANGLE. The angle between two vectors x, y is α = arccos x · y ||x||||y|| . CORRELATION. cos(α) = x·y ||x||||y|| is called the correlation between x and y. It is a number in [−1, 1]. EXAMPLE. The angle between two orthogonal vectors is 90 degrees or 270 degrees. If x and y represent data showing the deviation from the mean, then x·y ||x||||y|| is called the statistical correlation of the data. QUESTION. Express the fact that x is in the kernel of a matrix A using orthogonality. ANSWER: Ax = 0 means that wk · x = 0 for every row vector wk of Rn . REMARK. We will call later the matrix AT , obtained by switching rows and columns of A the transpose of A. You see already that the image of AT is orthogonal to the kernel of A. QUESTION. Find a basis for the orthogonal complement of the linear space V spanned by     1 2 3 4     ,     4 5 6 7    . ANSWER: The orthogonality of     x y z u     to the two vectors means solving the linear system of equations x + 2y + 3z + 4w = 0, 4x + 5y + 6z + 7w = 0. An other way to solve it: the kernel of A = 1 2 3 4 4 5 6 7 is the orthogonal complement of V . This reduces the problem to an older problem.
  • 30. ON THE RELEVANCE OF ORTHOGONALITY. 1) During the pyramid age in Egypt (from -2800 til -2300 BC), the Egyptians used ropes divided into length ratios 3 : 4 : 5 to build triangles. This allowed them to triangulate areas quite pre- cisely: for example to build irrigation needed because the Nile was reshaping the land constantly or to build the pyramids: for the great pyramid at Giza with a base length of 230 meters, the average error on each side is less then 20cm, an error of less then 1/1000. A key to achieve this was orthogonality. 2) During one of Thales (-624 til -548 BC) journeys to Egypt, he used a geometrical trick to measure the height of the great pyramid. He measured the size of the shadow of the pyramid. Using a stick, he found the relation between the length of the stick and the length of its shadow. The same length ratio applies to the pyramid (orthogonal triangles). Thales found also that triangles inscribed into a circle and having as the base as the diameter must have a right angle. 3) The Pythagoreans (-572 until -507) were interested in the dis- covery that the squares of a lengths of a triangle with two or- thogonal sides would add up as a2 + b2 = c2 . They were puzzled in assigning a length to the diagonal of the unit square, which is √ 2. This number is irrational because √ 2 = p/q would imply that q2 = 2p2 . While the prime factorization of q2 contains an even power of 2, the prime factorization of 2p2 contains an odd power of 2. 4) Eratosthenes (-274 until 194) realized that while the sun rays were orthogonal to the ground in the town of Scene, this did no more do so at the town of Alexandria, where they would hit the ground at 7.2 degrees). Because the distance was about 500 miles and 7.2 is 1/50 of 360 degrees, he measured the circumference of the earth as 25’000 miles - pretty close to the actual value 24’874 miles. 5) Closely related to orthogonality is parallelism. For a long time mathematicians tried to prove Euclid’s parallel axiom using other postulates of Euclid (-325 until -265). These attempts had to fail because there are geometries in which parallel lines always meet (like on the sphere) or geometries, where parallel lines never meet (the Poincar´e half plane). Also these geometries can be studied using linear algebra. The geometry on the sphere with rotations, the geometry on the half plane uses M¨obius transformations, 2 × 2 matrices with determinant one. 6) The question whether the angles of a right triangle are in reality always add up to 180 degrees became an issue when geometries where discovered, in which the measure- ment depends on the position in space. Riemannian geometry, founded 150 years ago, is the foundation of general relativity a theory which describes gravity geometrically: the presence of mass bends space-time, where the dot product can depend on space. Orthogonality becomes relative too. 7) In probability theory the notion of independence or decorrelation is used. For example, when throwing a dice, the number shown by the first dice is independent and decorrelated from the number shown by the second dice. Decorrelation is identical to orthogonality, when vectors are associated to the random variables. The correlation coefficient between two vectors v, w is defined as v · w/(|v|w|). It is the cosine of the angle between these vectors. 8) In quantum mechanics, states of atoms are described by functions in a linear space of functions. The states with energy −EB/n2 (where EB = 13.6eV is the Bohr energy) in a hydrogen atom. States in an atom are orthogonal. Two states of two different atoms which don’t interact are orthogonal. One of the challenges in quantum computing, where the computation deals with qubits (=vectors) is that orthogonality is not preserved during the computation. Different states can interact. This coupling is called decoherence.
  • 31. GRAM SCHMIDT AND QR Math 21b, O. Knill GRAM-SCHMIDT PROCESS. Let v1, ..., vn be a basis in V . Let u1 = v1 and w1 = u1/||u1||. The Gram-Schmidt process recursively constructs from the already constructed orthonormal set w1, ..., wi−1 which spans a linear space Vi−1 the new vector ui = (vi − projVi−1 (vi)) which is orthogonal to Vi−1, and then normalizing ui to to get wi = ui/||ui||. Each vector wi is orthonormal to the linear space Vi−1. The vectors {w1, .., wn} form an orthonormal basis in V . EXAMPLE. Find an orthonormal basis for v1 =   2 0 0  , v2 =   1 3 0   and v3 =   1 2 5  . SOLUTION. 1. w1 = v1/||v1|| =   1 0 0  . 2. u2 = (v2 − projV1 (v2)) = v2 − (w1 · v2)w1 =   0 3 0  . w2 = u2/||u2|| =   0 1 0  . 3. u3 = (v3 − projV2 (v3)) = v3 − (w1 · v3)w1 − (w2 · v3)w2 =   0 0 5  , w3 = u3/||u3|| =   0 0 1  . QR FACTORIZATION. The formulas can be written as v1 = ||v1||w1 = r11w1 · · · vi = (w1 · vi)w1 + · · · + (wi−1 · vi)wi−1 + ||ui||wi = ri1w1 + · · · + riiwi · · · vn = (w1 · vn)w1 + · · · + (wn−1 · vn)wn−1 + ||un||wn = rn1w1 + · · · + rnnwn which means in matrix form A =   | | · | v1 · · · · vm | | · |   =   | | · | w1 · · · · wm | | · |     r11 r12 · r1m 0 r22 · r2m 0 0 · rmm   = QR , where A and Q are n × m matrices and R is a m × m matrix. Any matrix A with linearly independent columns can be decom- posed as A = QR, where Q has orthonormal column vectors and where R is an upper triangular square matrix.
  • 32. BACK TO THE EXAMPLE. The matrix with the vectors v1, v2, v3 is A =   2 1 1 0 3 2 0 0 5  . v1 = ||v1||w1 v2 = (w1 · v2)w1 + ||u2||w2 v3 = (w1 · v3)w1 + (w2 · v3)w2 + ||u3||w3, so that Q =   1 0 0 0 1 0 0 0 1   and R =   2 1 1 0 3 2 0 0 5  . PRO MEMORIA. While building the matrix R we keep track of the vectors ui during the Gram-Schmidt procedure. At the end you have vectors ui, vi, wi and the matrix R has ||ui|| in the diagonal as well as the dot products wi · vj in the upper right triangle. PROBLEM. Make the QR decomposition of A = 0 −1 1 1 . w1 = 0 −1 . u2 = −1 1 − 0 1 = −1 0 . w2 = u2. A = 0 −1 1 0 1 1 0 1 = QR. WHY do we care to have an orthonormal basis? • An orthonormal basis looks like the standard basis v1 = (1, 0, ..., 0), . . ., vn = (0, 0, ..., 1). Actually, we will see that an orthonormal basis into a standard basis or a mirror of the standard basis. • The Gram-Schmidt process is tied to the factorization A = QR. The later helps to solve linear equations. In physical problems like in astrophysics, the numerical methods to simulate the problems one needs to invert huge matrices in every time step of the evolution. The reason why this is necessary sometimes is to assure the numerical method is stable implicit methods. Inverting A−1 = R−1 Q−1 is easy because R and Q are easy to invert. • For many physical problems like in quantum mechanics or dynamical systems, matrices are symmetric A∗ = A, where A∗ ij = Aji. For such matrices, there will a natural orthonormal basis. • The formula for the projection onto a linear subspace V simplifies with an orthonormal basis vj in V : projV (x) = (v1 · x)w1 + · · · + (wn · x)wn. • An orthonormal basis simplifies compuations due to the presence of many zeros wj · wi = 0. This is especially the case for problems with symmetry. • The Gram Schmidt process can be used to define and construct classes of classical polynomials, which are important in physics. Examples are Chebyshev polynomials, Laguerre polynomials or Hermite polynomi- als. • QR factorization allows fast computation of the determinant, least square solutions R−1 Q−1 b of overde- termined systems Ax = b or finding eigenvalues - all topics which will appear later. SOME HISTORY. The recursive formulae of the process were stated by Erhard Schmidt (1876-1959) in 1907. The essence of the formulae were already in a 1883 paper of J.P.Gram in 1883 which Schmidt mentions in a footnote. The process seems already have been used by Laplace (1749-1827) and was also used by Cauchy (1789-1857) in 1836. Gram Schmidt Laplace Cauchy
  • 33. ORTHOGONAL MATRICES 10/27/2002 Math 21b, O. Knill TRANSPOSE The transpose of a matrix A is the matrix (AT )ij = Aji. If A is a n×m matrix, then AT is a m × n matrix. For square matrices, the transposed matrix is obtained by reflecting the matrix at the diagonal. EXAMPLES The transpose of a vector A =   1 2 3   is the row vector AT = 1 2 3 . The transpose of the matrix 1 2 3 4 is the matrix 1 3 2 4 . A PROPERTY OF THE TRANSPOSE. a) x · Ay = AT x · y. b) (AB)T = BT AT . c) (AT )T = A. PROOFS. a) Because x · Ay = j i xiAijyj and AT x · y = j i Ajixiyj the two expressions are the same by renaming i and j. b) (AB)kl = i AkiBil. (AB)T kl = i AliBik = AT BT . c) ((AT )T )ij = (AT )ji = Aij. ORTHOGONAL MATRIX. A n × n matrix A is called orthogonal if AT A = 1. The corresponding linear transformation is called orthogonal. INVERSE. It is easy to invert an orthogonal matrix: A−1 = AT . EXAMPLES. The rotation matrix A = cos(φ) sin(φ) − sin(φ) cos(φ) is orthogonal because its column vectors have length 1 and are orthogonal to each other. Indeed: AT A = cos(φ) sin(φ) − sin(φ) cos(φ) · cos(φ) − sin(φ) sin(φ) cos(φ) = 1 0 0 1 . A reflection at a line is an orthogonal transformation because the columns of the matrix A have length 1 and are orthogonal. Indeed: AT A = cos(2φ) sin(2φ) sin(2φ) − cos(2φ) · cos(2φ) sin(2φ) sin(2φ) − cos(2φ) = 1 0 0 1 . FACTS. An orthogonal transformation preserves the dot product: Ax · Ay = x · y Proof: this is a homework assignment: Hint: just look at the properties of the transpose. Orthogonal transformations preserve the length of vectors as well as the angles between them. Proof. We have ||Ax||2 = Ax · Ax = x · x ||x||2 . Let α be the angle between x and y and let β denote the angle between Ax and Ay and α the angle between x and y. Using Ax·Ay = x·y we get ||Ax||||Ay|| cos(β) = Ax·Ay = x · y = ||x||||y|| cos(α). Because ||Ax|| = ||x||, ||Ay|| = ||y||, this means cos(α) = cos(β). Because this property holds for all vectors we can rotate x in plane V spanned by x and y by an angle φ to get cos(α+φ) = cos(β +φ) for all φ. Differentiation with respect to φ at φ = 0 shows also sin(α) = sin(β) so that α = β. ORTHOGONAL MATRICES AND BASIS. A linear transformation A is orthogonal if and only if the column vectors of A form an orthonormal basis. (That is what AT A = 1n means.) COMPOSITION OF ORTHOGONAL TRANSFORMATIONS. The composition of two orthogonal transformations is orthogonal. The inverse of an orthogonal transformation is orthogonal. Proof. The properties of the transpose give (AB)T AB = BT AT AB = BT B = 1 and (A−1 )T A−1 = (AT )−1 A−1 = (AAT )−1 = 1. EXAMPLES. The composition of two reflections at a line is a rotation. The composition of two rotations is a rotation. The composition of a reflections at a plane with a reflection at an other plane is a rotation (the axis of rotation is the intersection of the planes).
  • 34. ORTHOGONAL PROJECTIONS. The orthogonal projection P onto a linear space with orthonormal basis v1, . . . , vn is the matrix AAT , where A is the matrix with column vectors vi. To see this just translate the formula Px = (v1 · x)v1 + . . . + (vn · x)vn into the language of matrices: AT x is a vector with components bi = (vi · x) and Ab is the sum of the bivi, where vi are the column vectors of A. Orthogonal projections are no orthogonal transformations in general! EXAMPLE. Find the orthogonal projection P from R3 to the linear space spanned by v1 =   0 3 4   1 5 and v2 =   1 0 0  . Solution: AAT =   0 1 3/5 0 4/5 0   0 3/5 4/5 1 0 0 =   1 0 0 0 9/25 12/25 0 12/25 16/25  . WHY ARE ORTHOGONAL TRANSFORMATIONS USEFUL? • In Physics, Galileo transformations are compositions of translations with orthogonal transformations. The laws of classical mechanics are invariant under such transformations. This is a symmetry. • Many coordinate transformations are orthogonal transformations. We will see examples when dealing with differential equations. • In the QR decomposition of a matrix A, the matrix Q is orthogonal. Because Q−1 = Qt , this allows to invert A easier. • Fourier transformations are orthogonal transformations. We will see this transformation later in the course. In application, it is useful in computer graphics (i.e. JPG) and sound compression (i.e. MP3). • Quantum mechanical evolutions (when written as real matrices) are orthogonal transformations. WHICH OF THE FOLLOWING MAPS ARE ORTHOGONAL TRANSFORMATIONS?: Yes No Shear in the plane. Yes No Projection in three dimensions onto a plane. Yes No Reflection in two dimensions at the origin. Yes No Reflection in three dimensions at a plane. Yes No Dilation with factor 2. Yes No The Lorenz boost x → Ax in the plane with A = cosh(α) sinh(α) sinh(α) cosh(α) Yes No A translation. CHANGING COORDINATES ON THE EARTH. Problem: what is the matrix which rotates a point on earth with (latitude,longitude)=(a1, b1) to a point with (latitude,longitude)=(a2, b2)? Solution: The ma- trix which rotate the point (0, 0) to (a, b) a composition of two rotations. The first rotation brings the point into the right latitude, the second brings the point into the right longitude. Ra,b =  cos(b) − sin(b) 0 sin(b) cos(b) 0 0 0 1     cos(a) 0 − sin(a) 0 1 0 sin(a) 0 cos(a)  . To bring a point (a1, b1) to a point (a2, b2), we form A = Ra2,b2 R−1 a1,b1 . EXAMPLE: With Cambridge (USA): (a1, b1) = (42.366944, 288.893889)π/180 and Z¨urich (Switzerland): (a2, b2) = (47.377778, 8.551111)π/180, we get the matrix A =   0.178313 −0.980176 −0.0863732 0.983567 0.180074 −0.0129873 0.028284 −0.082638 0.996178  .
  • 35. LEAST SQUARES AND DATA Math 21b, O. Knill GOAL. The best possible ”solution” of an inconsistent linear systems Ax = b will be called the least square solution. It is the orthogonal projection of b onto the image im(A) of A. The theory of the kernel and the image of linear transformations helps to understand this situation and leads to an explicit formula for the least square fit. Why do we care about non-consistent systems? Often we have to solve linear systems of equations with more constraints than variables. An example is when we try to find the best polynomial which passes through a set of points. This problem is called data fitting. If we wanted to accommodate all data, the degree of the polynomial would become too large, the fit would look too wiggly. Taking a smaller degree polynomial will not only be more convenient but also give a better picture. Especially important is regression, the fitting of data with lines. The above pictures show 30 data points which are fitted best with polynomials of degree 1, 6, 11 and 16. The first linear fit maybe tells most about the trend of the data. THE ORTHOGONAL COMPLEMENT OF im(A). Because a vector is in the kernel of AT if and only if it is orthogonal to the rows of AT and so to the columns of A, the kernel of AT is the orthogonal complement of im(A): (im(A))⊥ = ker(AT ) EXAMPLES. 1) A =   a b c  . The kernel V of AT = a b c consists of all vectors satisfying ax + by + cz = 0. V is a plane. The orthogonal complement is the image of A which is spanned by the normal vector   a b c   to the plane. 2) A = 1 1 0 0 . The image of A is spanned by 1 0 the kernel of AT is spanned by 0 1 . ORTHOGONAL PROJECTION. If b is a vector and V is a linear subspace, then projV (b) is the vector closest to b on V : given any other vector v on V , one can form the triangle b, v, projV (b) which has a right angle at projV (b) and invoke Pythagoras. THE KERNEL OF AT A. For any m × n matrix ker(A) = ker(AT A) Proof. ⊂ is clear. On the other hand AT Av = 0 means that Av is in the kernel of AT . But since the image of A is orthogonal to the kernel of AT , we have Av = 0, which means v is in the kernel of A. LEAST SQUARE SOLUTION. The least square so- lution of Ax = b is the vector x∗ such that Ax∗ is closest to b from all other vectors Ax. In other words, Ax∗ = projV (b), where V = im(V ). Because b − Ax∗ is in V ⊥ = im(A)⊥ = ker(AT ), we have AT (b − Ax∗ ) = 0. The last equation means that x∗ is a solution of AT Ax = AT b, the normal equation of Ax = b . If the kernel of A is triv- ial, then the kernel of AT A is trivial and AT A can be inverted. Therefore x∗ = (AT A)−1 AT b is the least square solution. V=im(A) T ker(A )= im(A) T b Ax* A (b-Ax)=0 T Ax=A(A A) A bT T-1* WHY LEAST SQUARES? If x∗ is the least square solution of Ax = b then ||Ax∗ − b|| ≤ ||Ax − b|| for all x. Proof. AT (Ax∗ − b) = 0 means that Ax∗ − b is in the kernel of AT which is orthogonal to V = im(A). That is projV (b) = Ax∗ which is the closest point to b on V .
  • 36. ORTHOGONAL PROJECTION If v1, . . . , vn is a basis in V which is not necessarily orthonormal, then the orthogonal projection is x → A(AT A)−1 AT (x) where A = [v1, . . . , vn]. Proof. x = (AT A)−1 AT b is the least square solution of Ax = b. Therefore Ax = A(AT A)−1 AT b is the vector in im(A) closest to b. Special case: If w1, . . . , wn is an orthonormal basis in V , we had seen earlier that AAT with A = [w1, . . . , wn] is the orthogonal projection onto V (this was just rewriting Ax = (w1 · x)w1 + · · · + (wn · x)wn in matrix form.) This follows from the above formula because AT A = I in that case. EXAMPLE Let A =   1 0 2 0 0 1  . The orthogonal projection onto V = im(A) is b → A(AT A)−1 AT b. We have AT A = 5 0 2 1 and A(AT A)−1 AT =   1/5 2/5 0 2/5 4/5 0 0 0 1  . For example, the projection of b =   0 1 0   is x∗ =   2/5 4/5 0   and the distance to b is 1/ √ 5. The point x∗ is the point on V which is closest to b. Remember the formula for the distance of b to a plane V with normal vector n? It was d = |n · b|/||n||. In our case, we can take n = [−2, 1, 0] this formula gives the distance 1/ √ 5. Let’s check: the distance of x∗ and b is ||(2/5, −1/5, 0)|| = 1/ √ 5. EXAMPLE. Let A =     1 2 0 1    . Problem: find the matrix of the orthogonal projection onto the image of A. The image of A is a one-dimensional line spanned by the vector v = (1, 2, 0, 1). We calculate AT A = 6. Then A(AT A)−1 AT =     1 2 0 1     1 2 0 1 /6 =     1 2 0 1 2 4 0 2 0 0 0 0 1 2 0 1     /6 DATA FIT. Find a quadratic polynomial p(t) = at2 + bt + c which best fits the four data points (−1, 8), (0, 8), (1, 4), (2, 16). A =     1 −1 1 0 0 1 1 1 1 4 2 1     b =     8 8 4 16     T . AT A =   18 8 6 8 6 2 6 2 4   and x∗ = (AT A)−1 AT b =   3 −1 5  . Software packages like Mathematica have already built in the facility to fit numerical data: The series expansion of f showed that indeed, f(t) = 5 − t + 3t2 is indeed best quadratic fit. Actually, Mathematica does the same to find the fit then what we do: ”Solving” an inconsistent system of linear equations as best as possible. PROBLEM: Prove im(A) = im(AAT ). SOLUTION. The image of AAT is contained in the image of A because we can write v = AAT x as v = Ay with y = AT x. On the other hand, if v is in the image of A, then v = Ax. If x = y + z, where y in the kernel of A and z orthogonal to the kernel of A, then Ax = Az. Because z is orthogonal to the kernel of A, it is in the image of AT . Therefore, z = AT u and v = Az = AAT u is in the image of AAT .
  • 37. THE DIMENSION OF THE MASS COAST LINE Math 21b, O.Knill To measure the dimension of an object, one can count the number f(n) of boxes of length 1/n needed to cover the object and see how f(n) grows. If f(n) grows like n2 , then the dimension is 2, if n(k) grows like n, the dimension is 1. For fractal objects, like coast lines, the number of boxes grows like ns for a number s between 1 and 2. The dimension is obtained by correlating yk = log2 f(k) with xk = log2(k). We measure: The Massachusetts coast line is a fractal of dimension 1.3. We measure the data: f(1) = 5, f(2) = 12, f(4) = 32 and f(8) = 72. A plot of the data (xk, yk) = (log2(k), log2 f(k)) together with a least square fit can be seen to the right. The slope of the line is the dimension of the coast. It is about 1.295. In order to measure the di- mension better, one would need better maps. xk yk 0 log2(5) 1 log2(12) 2 log2(32) 3 log2(72)
  • 38. Finding the best linear fit y = ax + b is equivalent to find the least square solution of the system 0a + b = log2(5), 1a + b = log2(12), 2a + b = log2(32), 3a + b = log2(72) which is Ax = b with x = a b , b =     2.3 3.5 5 6     and A =     0 1 1 1 2 1 3 1    . We have AT A = 14 6 6 4 , (AT A)−1 = 2 −3 −3 7 /10 and B = (AT A)−1 AT = −3 −1 1 3 7 4 1 −2 /10. We get = a b = Bb = 1.29 2.32 . COMPARISON: THE DIMENSION OF THE CIRCLE Let us compare this with a smooth curve. To cover a circle with squares of size 2π/2n , we need about f(n) = 2n to cover the cir- cle. We measure that the circle is not a fractal. It has dimension 1. xk yk 3 log2(8) = 3 4 log2(16) = 4 5 log2(32) = 5 6 log2(64) = 6 COMPARISON: THE DIMENSION OF THE DISK For an other comparison, we take a disk. To cover the disk with squares of size 2π/2n , we need about f(n) = 22n squares. We measure that the disk is not a frac- tal. It has dimension 2. xk yk 3 log2(5) = 2.3 4 log2(13) = 3.7 5 log2(49) = 5.61 6 log2(213) = 7.73 REMARKS Calculating the dimension of coast lines is a classical illustration of ”fractal theory”. The coast of Brittain has been measured to have a dimension of 1.3 too. For natural objects, it is typical that measuring the length on a smaller scale gives larger results: one can measure empirically the dimension of mountains, clouds, plants, snowflakes, the lung etc. The dimension is an indicator how rough a curve or surface is.
  • 39. DETERMINANTS I Math 21b, O. Knill PERMUTATIONS. A permutation of n elements {1, 2, . . . , n} is a rearrangement of {1, 2, . . . , n}. There are n! = n · (n − 1)... · 1 different permutations of {1, 2, . . . , n}: fixing the position of first element leaves (n − 1)! possibilities to permute the rest. EXAMPLE. There are 6 permutations of {1, 2, 3}: (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1). PATTERNS AND SIGN. The matrix A with zeros everywhere except Ai,π(i) = 1 is called a permutation matrix or the pattern of π. An inversion is a pair k < l such that σ(k) < σ(l). The sign of a permutation π, denoted by (−1)σ(π) is (−1) if there are an odd number of inversions in the pattern. Otherwise, the sign is defined to be +1. EXAMPLES. σ(1, 2) = 0, (−1)σ = 1, σ(2, 1) = 1, (−1)σ = −1. The permutations(1, 2, 3), (3, 2, 1), (2, 3, 1) have sign 1, the permutations (1, 3, 2), (3, 2, 1), (2, 1, 3) have sign −1. DETERMINANT The determinant of a n × n matrix A is the sum π(−1)σ(π) A1π(1)A2π(2) · · · Anπ(n), where π runs over all permutations of {1, 2, . . . , n} and σ(π) is the sign of the permutation. 2 × 2 CASE. The determinant of A = a b c d is ad − bc. There are two permutations of (1, 2). The identity permutation (1, 2) gives A11A12, the permutation (2, 1) gives A21A22. If you have seen some multi-variable calculus, you know that det(A) is the area of the parallelogram spanned by the column vectors of A. The two vectors form a basis if and only if det(A) = 0. 3 × 3 CASE. The determinant of A = a b c d e f g h i is aei + bfg + cdh − ceg − fha − bdi corresponding to the 6 permutations of (1, 2, 3). Geometrically, det(A) is the volume of the parallelepiped spanned by the column vectors of A. The three vectors form a basis if and only if det(A) = 0. EXAMPLE DIAGONAL AND TRIANGULAR MATRICES. The determinant of a diagonal or triangular matrix is the product of the diagonal elements. EXAMPLE PERMUTATION MATRICES. The determinant of a matrix which has everywhere zeros except Aiπ(j) = 1 is just the sign (−1)σ(π) of the permutation. HOW FAST CAN WE COMPUTE THE DETERMINANT?. The cost to find the determinant is the same as for the Gauss- Jordan elimination as we will see below The graph to the left shows some measurements of the time needed for a CAS to calculate the determinant in dependence on the size of the n × n matrix. The matrix size ranges from n=1 to n=300. We also see a best cubic fit of these data using the least square method from the last lesson. It is the cubic p(x) = a + bx + cx2 + dx3 which fits best through the 300 data points. WHY DO WE CARE ABOUT DETERMINANTS? • check invertibility of matrices • have geometric interpretation as volume • explicit algebraic expressions for inverting a ma- trix • as a natural functional on matrices it appears in formulas in particle or statistical physics • allow to define orientation in any dimensions • appear in change of variable formulas in higher dimensional integration. • proposed alternative concepts are unnatural, hard to teach and and harder to understand • determinants are fun
  • 40. TRIANGULAR AND DIAGONAL MATRICES. The determinant of a diagonal or triangular ma- trix is the product of its diagonal elements. Example: det(     1 0 0 0 4 5 0 0 2 3 4 0 1 1 2 1    ) = 20. PARTITIONED MATRICES. The determinant of a partitioned matrix A 0 0 B is the product det(A)det(B). Example det(     3 4 0 0 1 2 0 0 0 0 4 −2 0 0 2 2    ) = 2 · 12 = 24. LINEARITY OF THE DETERMINANT. If the columns of A and B are the same except for the i’th column, det([v1, ..., v, ...vn]] + det([v1, ..., w, ...vn]] = det([v1, ..., v + w, ...vn]] . In general, one has det([v1, ..., kv, ...vn]] = k det([v1, ..., v, ...vn]] . The same holds for rows. These identities follow directly from the original definition of the determimant. PROPERTIES OF DETERMINANTS. det(AB) = det(A)det(B) . det(A−1 ) = det(A)−1 det(SAS−1 ) = det(A) . det(AT ) = det(A) de det(− If B is obtained from A by switching two rows, then det(B) = −det(A). If B is obtained by adding an other row to a given row, then this does not change the value of the determinant. PROOF OF det(AB) = det(A)det(B), one brings the n × n matrix [A|AB] into row reduced echelon form. Similar than the augmented matrix [A|b] was brought into the form [1|A−1 b], we end up with [1|A−1 AB] = [1|B]. By looking at the n × n matrix to the left during Gauss-Jordan elimination, the determinant has changed by a factor det(A). We end up with a matrix B which has determinant det(B). Therefore, det(AB) = det(A)det(B). PROOF OF det(AT ) = det(A). The transpose of a pattern is a pattern with the same signature. PROBLEM. Find the determi- nant of A =     0 0 0 2 1 2 4 5 0 7 2 9 0 0 6 4    . SOLUTION. Three row transpositions give B =     1 2 4 5 0 7 2 9 0 0 6 4 0 0 0 2     a ma- trix which has determinant 84. Therefore det(A) = (−1)3 det(B) = −84. PROBLEM. Determine det(A100 ), where A is the matrix 1 2 3 16 . SOLUTION. det(A) = 10, det(A100 ) = (det(A))100 = 10100 = 1 · gogool. This name as well as the gogoolplex = 1010100 are official. They are huge numbers: the mass of the universe for example is 1052 kg and 1/101051 is the chance to find yourself on Mars by quantum fluctuations. (R.E. Crandall, Scient. Amer., Feb. 1997). ROW REDUCED ECHELON FORM. Determining rref(A) also determines det(A). If A is a matrix and αi are the factors which are used to scale different rows and s is the number of times, two rows are switched, then det(A) = (−1)s α1 · · · αn det(rref(A)) . INVERTIBILITY. Because of the last formula: A n × n matrix A is invertible if and only if det(A) = 0. THE LAPLACE EXPANSION. In order to calculate by hand the determinant of n × n matrices A = aij for n > 3, the following expansion is useful. Choose a column i. For each entry aji in that column, take the (n − 1) × (n − 1) matrix Aij called minor which does not contain the i’th column and j’th row. Then det(A) = (−1)i+1 ai1det(Ai1) + · · · + (−1)i+n aindet(Ain) = n j=1(−1)i+j aijdet(Aij) ORTHOGONAL MATRICES. Because QT Q = 1, we have det(Q)2 = 1 and so |det(Q)| = 1 Rotations have determinant 1, reflections have determinant −1. QR DECOMPOSITION. If A = QR, then det(A) = det(Q)det(R). The determinant of Q is ±1, the determinant of R is the product of the diagonal elements of R.
  • 41. DETERMINANTS II, Math 21b, O.Knill DETERMINANT AND VOLUME. If A is a n × n matrix, then |det(A)| is the volume of the n-dimensional parallelepiped En spanned by the n column vectors vj of A. Proof. Use the QR decomposition A = QR, where Q is orthogonal and R is upper triangular. From QQT = 1, we get 1 = det(Q)det(QT ) = det(Q)2 see that |det(Q)| = 1. Therefore, det(A) = ±det(R). The determinant of R is the product of the ||ui|| = ||vi − projVj−1 vi|| which was the distance from vi to Vj−1. The volume vol(Ej) of a j-dimensional parallelepiped Ej with base Ej−1 in Vj−1 and height ||ui|| is vol(Ej−1)||uj||. Inductively vol(Ej) = ||uj||vol(Ej−1) and therefore vol(En) = n j=1 ||uj|| = det(R). The volume of a k dimensional parallelepiped defined by the vectors v1, . . . , vk is det(AT A). Proof. QT Q = In gives AT A = (QR)T (QR) = RT QT QR = RT R. So, det(RT R) = det(R)2 = ( k j=1 ||uj||)2 . (Note that A is a n × k matrix and that AT A = RT R and R are k × k matrices.) ORIENTATION. Determinants allow to define the orientation of n vectors in n-dimensional space. This is ”handy” because there is no ”right hand rule” in hyperspace... To do so, define the matrix A with column vectors vj and define the orientation as the sign of det(A). In three dimensions, this agrees with the right hand rule: if v1 is the thumb, v2 is the pointing finger and v3 is the middle finger, then their orientation is positive. b i det x det(A) =i CRAMER’S RULE. This is an explicit formula for the solution of Ax = b. If Ai denotes the matrix, where the column vi of A is replaced by b, then xi = det(Ai)/det(A) Proof. det(Ai) = det([v1, . . . , b, . . . , vn] = det([v1, . . . , (Ax), . . . , vn] = det([v1, . . . , i xivi, . . . , vn] = xidet([v1, . . . , vi, . . . , vn]) = xidet(A) EXAMPLE. Solve the system 5x+3y = 8, 8x+5y = 2 using Cramers rule. This linear system with A = 5 3 8 5 and b = 8 2 . We get x = det 8 3 2 5 = 34y = det 5 8 8 2 = −54. GABRIEL CRAMER. (1704-1752), born in Geneva, Switzerland, he worked on geometry and analysis. Cramer used the rule named after him in a book ”Introduction `a l’analyse des lignes courbes alg´ebraique”, where he solved like this a system of equations with 5 unknowns. According to a short biography of Cramer by J.J O’Connor and E F Robertson, the rule had however been used already before by other mathematicians. Solving systems with Cramers formulas is slower than by Gaussian elimination. The rule is still important. For example, if A or b depends on a parameter t, and we want to see how x depends on the parameter t one can find explicit formulas for (d/dt)xi(t). THE INVERSE OF A MATRIX. Because the columns of A−1 are solutions of Ax = ei, where ej are basis vectors, Cramers rule together with the Laplace expansion gives the formula: [A−1 ]ij = (−1)i+j det(Aji)/det(A) Bij = (−1)i+j det(Aji) is called the classical adjoint of A. Note the change ij → ji. Don’t confuse the classical adjoint with the transpose AT which is sometimes also called the adjoint.
  • 42. EXAMPLE. A =   2 3 1 5 2 4 6 0 7   has det(A) = −17 and we get A−1 =   14 −21 10 −11 8 −3 −12 18 −11   /(−17): B11 = (−1)1+1 det 2 4 0 7 = 14. B12 = (−1)1+2 det 3 1 0 7 = −21. B13 = (−1)1+3 det 3 1 2 4 = 10. B21 = (−1)2+1 det 5 4 6 7 = −11. B22 = (−1)2+2 det 2 1 6 7 = 8. B23 = (−1)2+3 det 2 1 5 4 = −3. B31 = (−1)3+1 det 5 2 6 0 = −12. B32 = (−1)3+2 det 2 3 6 0 = 18. B33 = (−1)3+3 det 2 3 5 2 = −11. THE ART OF CALCULATING DETERMINANTS. When confronted with a matrix, it is good to go through a checklist of methods to crack the determinant. Often, there are different possibilities to solve the problem, in many cases the solution is particularly simple using one method. • Is it a 2 × 2 or 3 × 3 matrix? • Is it a upper or lower triangular matrix? • Is it a partitioned matrix? • Is it a trick: like A1000 ? • Does geometry imply noninvertibility? • Do you see duplicated columns or rows? • Can you row reduce to a triangular case? • Are there only a few nonzero patters? • Laplace expansion with some row or column? • Later: Can you see the eigenvalues of A − λIn? EXAMPLES. 1) A =       1 2 3 4 5 2 4 6 8 10 5 5 5 5 4 1 3 2 7 4 3 2 8 4 9       Try row reduction. 2) A =       2 1 0 0 0 1 1 1 0 0 0 0 2 1 0 0 0 0 3 1 0 0 0 0 4       Laplace expansion. 3) A =       1 1 0 0 0 1 2 2 0 0 0 1 1 0 0 0 0 0 1 3 0 0 0 4 2       Partitioned matrix. 4) A =       1 6 10 1 15 2 8 17 1 29 0 0 3 8 12 0 0 0 4 9 0 0 0 0 5       Make it tridiagonal. APPLICATION HOFSTADTER BUTTERFLY. In solid state physics, one is interested in the function f(E) = det(L − EIn), where L =        λ cos(α) 1 0 · 0 1 1 λ cos(2α) 1 · · 0 0 1 · · · · · · · · 1 0 0 · · 1 λ cos((n − 1)α) 1 1 0 · 0 1 λ cos(nα)        describes an electron in a periodic crystal, E is the energy and α = 2π/n. The electron can move as a Bloch wave whenever the determinant is negative. These intervals form the spectrum of the quantum mechanical system. A physicist is interested in the rate of change of f(E) or its dependence on λ when E is fixed. . -4 -2 2 4 1 2 3 4 5 6 The graph to the left shows the function E → log(|det(L − EIn)|) in the case λ = 2 and n = 5. In the energy intervals, where this function is zero, the electron can move, otherwise the crystal is an in- sulator. The picture to the right shows the spectrum of the crystal depending on α. It is called the ”Hofs- tadter butterfly” made popular in the book ”G¨odel, Escher Bach” by Douglas Hofstadter.
  • 43. COMPUTING EIGENVALUES Math 21b, O.Knill THE TRACE. The trace of a matrix A is the sum of its diagonal elements. EXAMPLES. The trace of A =   1 2 3 3 4 5 6 7 8   is 1 + 4 + 8 = 13. The trace of a skew symmetric matrix A is zero because there are zeros in the diagonal. The trace of In is n. CHARACTERISTIC POLYNOMIAL. The polynomial fA(λ) = det(λIn − A) is called the characteristic polynomial of A. EXAMPLE. The characteristic polynomial of A above is x3 − 13x2 + 15x. The eigenvalues of A are the roots of the characteristic polynomial fA(λ). Proof. If λ is an eigenvalue of A with eigenfunction v, then A − λ has v in the kernel and A − λ is not invertible so that fA(λ) = det(λ − A) = 0. The polynomial has the form fA(λ) = λn − tr(A)λn−1 + · · · + (−1)n det(A) THE 2x2 CASE. The characteristic polynomial of A = a b c d is fA(λ) = λ2 − (a + d)/2λ + (ad − bc). The eigenvalues are λ± = T/2 ± (T/2)2 − D, where T is the trace and D is the determinant. In order that this is real, we must have (T/2)2 ≥ D. Away from that parabola, there are two different eigenvalues. The map A contracts volume for |D| < 1. EXAMPLE. The characteristic polynomial of A = 1 2 0 2 is λ2 − 3λ + 2 which has the roots 1, 2: pA(λ) = (λ − 1)(λ − 2). THE FIBONNACCI RABBITS. The Fibonacci’s recursion un+1 = un + un−1 defines the growth of the rabbit population. We have seen that it can be rewritten as un+1 un = A un un−1 with A = 1 1 1 0 . The roots of the characteristic polynomial pA(x) = λ2 − λ − 1. are ( √ 5 + 1)/2, ( √ 5 − 1)/2. ALGEBRAIC MULTIPLICITY. If fA(λ) = (λ − λ0)k g(λ), where g(λ0) = 0 then λ is said to be an eigenvalue of algebraic multiplicity k. EXAMPLE:   1 1 1 0 1 1 0 0 2   has the eigenvalue λ = 1 with algebraic multiplicity 2 and the eigenvalue λ = 2 with algebraic multiplicity 1. HOW TO COMPUTE EIGENVECTORS? Because (λ − A)v = 0, the vector v is in the kernel of λ − A. We know how to compute the kernel. EXAMPLE FIBONNACCI. The kernel of λI2 − A = λ± − 1 −1 −1 λ± is spanned by v+ = (1 + √ 5)/2, 1 and v− = (1 − √ 5)/2, 1 . They form a basis B. SOLUTION OF FIBONNACCI. To obtain a formula for An v with v = 1 0 , we form [v]B = 1 −1 / √ 5. Now, un+1 un = An v = An (v+/ √ 5 − v−/ √ 5) = An v+/ √ 5− An (v−/ √ 5 = λn +v+/ √ 5+ λn −v+/ √ 5. We see that un = [(1+ √ 5 2 )n + (1− √ 5 2 )n ]/ √ 5. ROOTS OF POLYNOMIALS. For polynomials of degree 3 and 4 there exist explicit formulas in terms of radicals. As Galois (1811-1832) and Abel (1802-1829) have shown, it is not possible for equations of degree 5 or higher. Still, one can compute the roots numerically.
  • 44. REAL SOLUTIONS. A (2n + 1) × (2n + 1) matrix A always has a real eigenvalue because the characteristic polynomial p(x) = x5 + ... + det(A) has the property that p(x) goes to ±∞ for x → ±∞. Because there exist values a, b for which p(a) < 0 and p(b) > 0, by the intermediate value theorem, there exists a real x with p(x) = 0. Application: A rotation in 11 dimensional space has all eigenvalues |λ| = 1. The real eigenvalue must have an eigenvalue 1 or −1. EIGENVALUES OF TRANSPOSE. We know (see homework) that the characteristic polynomials of A and AT agree. Therefore A and AT have the same eigenvalues. APPLICATION: MARKOV MATRICES. A matrix A for which each column sums up to 1 is called a Markov matrix. The transpose of a Markov matrix has the eigenvector     1 1 . . . 1     with eigenvalue 1. Therefore: A Markov matrix has an eigenvector v to the eigenvalue 1. This vector v defines an equilibrium point of the Markov process. EXAMPLE. If A = 1/3 1/2 2/3 1/2 . Then [3/7, 4/7] is the equilibrium eigenvector to the eigenvalue 1. BRETSCHERS HOMETOWN. Problem 28 in the book deals with a Markov problem in Andelfingen the hometown of Bretscher, where people shop in two shops. (Andelfingen is a beautiful village at the Thur river in the middle of a ”wine country”). Initially all shop in shop W. After a new shop opens, every week 20 percent switch to the other shop M. Missing something at the new place, every week, 10 percent switch back. This leads to a Markov matrix A = 8/10 1/10 2/10 9/10 . After some time, things will settle down and we will have certain percentage shopping in W and other percentage shopping in M. This is the equilibrium. . MARKOV PROCESS IN PROBABILITY. Assume we have a graph like a network and at each node i, the probability to go from i to j in the next step is [A]ij, where Aij is a Markov matrix. We know from the above result that there is an eigenvector p which satisfies Ap = p. It can be normalized that i pi = 1. The interpretation is that pi is the probability that the walker is on the node p. For example, on a triangle, we can have the probabilities: P(A → B) = 1/2, P(A → C) = 1/4, P(A → A) = 1/4, P(B → A) = 1/3, P(B → B) = 1/6, P(B → C) = 1/2, P(C → A) = 1/2, P(C → B) = 1/3, P(C → C) = 1/6. The corresponding matrix is A =   1/4 1/3 1/2 1/2 1/6 1/3 1/4 1/2 1/6   . In this case, the eigenvector to the eigenvalue 1 is p = [38/107, 36/107, 33/107]T . A B C
  • 45. CALCULATING EIGENVECTORS Math 21b, O.Knill NOTATION. We often just write 1 instead of the identity matrix 1n. COMPUTING EIGENVALUES. Recall: because λ−A has v in the kernel if λ is an eigenvalue the characteristic polynomial fA(λ) = det(λ − A) = 0 has eigenvalues as roots. 2×2 CASE. Recall: The characteristic polynomial of A = a b c d is fA(λ) = λ2 −(a+d)/2λ+(ad−bc). The eigenvalues are λ± = T/2 ± (T/2)2 − D, where T = a + d is the trace and D = ad − bc is the determinant of A. If (T/2)2 ≥ D, then the eigenvalues are real. Away from that parabola in the (T, D) space, there are two different eigenvalues. The map A contracts volume for |D| < 1. NUMBER OF ROOTS. Recall: There are examples with no real eigenvalue (i.e. rotations). By inspecting the graphs of the polynomials, one can deduce that n × n matrices with odd n always have a real eigenvalue. Also n × n matrixes with even n and a negative determinant always have a real eigenvalue. IF ALL ROOTS ARE REAL. fA(λ) = λn − tr(A)λn−1 + ... + (−1)n det(A) = (λ − λ1)...(λ − λn), we see that i λi = trace(A) and i λi = det(A). HOW TO COMPUTE EIGENVECTORS? Because (λ − A)v = 0, the vector v is in the kernel of λ − A. EIGENVECTORS of a b c d are v± with eigen- value λ±. If c = 0 and d = 0, then 1 0 and 0 1 are eigen- vectors. If c = 0, then the eigenvectors to λ± are v± = λ± − d c . If c = 0 and a = d, then the eigenvectors to a and d are 1 0 and b/(d − a) 1 . ALGEBRAIC MULTIPLICITY. If fA(λ) = (λ−λ0)k g(λ), where g(λ0) = 0, then f has algebraic multiplicity k. The algebraic multiplicity counts the number of times, an eigenvector occurs. EXAMPLE:   1 1 1 0 1 1 0 0 2   has the eigenvalue λ = 1 with algebraic multiplicity 2. GEOMETRIC MULTIPLICITY. The dimension of the eigenspace Eλ of an eigenvalue λ is called the geometric multiplicity of λ. EXAMPLE: the matrix of a shear is 1 1 0 1 . It has the eigenvalue 1 with algebraic multiplicity 2. The kernel of A − 1 = 0 1 0 0 is spanned by 1 0 and the geometric multiplicity is 1. It is different from the algebraic multiplicity. EXAMPLE: The matrix   1 1 1 0 0 1 0 0 1   has eigenvalue 1 with algebraic multiplicity 2 and the eigenvalue 0 with multiplicity 1. Eigenvectors to the eigenvalue λ = 1 are in the kernel of A − 1 which is the kernel of  0 1 1 0 −1 1 0 0 0   and spanned by   1 0 0  . The geometric multiplicity is 1. RELATION BETWEEN ALGEBRAIC AND GEOMETRIC MULTIPLICIY. (Proof later in the course). The geometric multiplicity is smaller or equal than the algebraic multiplicity. PRO MEMORIA. Remember that the geometric mean √ ab of two numbers is smaller or equal to the alge- braic mean (a + b)/2? (This fact is totally∗ unrelated to the above fact and a mere coincidence of expressions, but it helps to remember it). Quite deeply buried there is a connection in terms of convexity. But this is rather philosophical. .
  • 46. EXAMPLE. What are the algebraic and geometric multiplicities of A =       2 1 0 0 0 0 2 1 0 0 0 0 2 1 0 0 0 0 2 1 0 0 0 0 2       ? SOLUTION. The algebraic multiplicity of the eigenvalue 2 is 5. To get the kernel of A−2, one solves the system of equations x4 = x3 = x2 = x1 = 0 so that the geometric multiplicity of the eigenvalues 2 is 4. CASE: ALL EIGENVALUES ARE DIFFERENT. If all eigenvalues are different, then all eigenvectors are linearly independent and all geometric and alge- braic multiplicities are 1. PROOF. Let λi be an eigenvalue different from 0 and assume the eigenvectors are linearly dependent. We have vi = j=i ajvj and λivi = Avi = A( j=i ajvj) = j=i ajλjvj so that vi = j=i bjvj with bj = ajλj/λi. If the eigenvalues are different, then aj = bj and by subtracting vi = j=i ajvj from vi = j=i bjvj, we get 0 = j=i(bj − aj)vj = 0. Now (n − 1) eigenvectors of the n eigenvectors are linearly dependent. Use induction. CONSEQUENCE. If all eigenvalues of a n × n matrix A are different, there is an eigenbasis, a basis consisting of eigenvectors. EXAMPLE. A = 1 1 0 3 has eigenvalues 1, 3 to the eigenvectors 1 0 −1 1 . EXAMPLE. (See homework problem 40 in the book). Photos of the Swiss lakes in the text. The pollution story is fiction fortunately. The vector An (x)b gives the pollution levels in the three lakes (Silvaplana, Sils, St Moritz) after n weeks, where A =   0.7 0 0 0.1 0.6 0 0 0.2 0.8   and b =   100 0 0   is the initial pollution. 4 6 8 10 20 40 60 80 100 There is an eigenvector e3 = v3 =   0 0 1   to the eigenvalue λ3 = 0.8. There is an eigenvector v2 =   0 1 −1   to the eigenvalue λ2 = 0.6. There is further an eigenvector v1 =   1 1 −2   to the eigenvalue λ1 = 0.7. We know An v1, An v2 and An v3 explicitly. How do we get the explicit solution An b? Because b = 100 · e1 = 100(v1 − v2 + 3v3), we have An (b) = 100An (v1 − v2 + 3v3) = 100(λn 1 v1 − λn 2 v2 + 3λn 3 v3) = 100  0.7n   1 1 −2   + 0.6n   0 1 −1   + 3 · 0.8n   0 0 1     =   100(0.7)n 100(0.7n + 0.6n ) 100(−2 · 0.7n − 0.6n + 3 · 0.8n )  
  • 47. DIAGONALIZATION Math 21b, O.Knill SUMMARY. A n × n matrix, Av = λv with eigenvalue λ and eigenvector v. The eigenvalues are the roots of the characteristic polynomial fA(λ) = det(λ − A) = λn − tr(A)λn−1 + ... + (−1)n det(A). The eigenvectors to the eigenvalue λ are in ker(λ − A). The number of times, an eigenvalue λ occurs in the full list of n roots of fA(λ) is called algebraic multiplicity. It is bigger or equal than the geometric multiplicity: dim(ker(λ − A). EXAMPLE. The eigenvalues of a b c d are λ± = T/2 + T2/4 − D, where T = a + d is the trace and D = ad − bc is the determinant of A. If c = 0, the eigenvectors are v± = λ± − d c if c = 0, then a, d are eigenvalues to the eigenvectors a 0 and −b a − d . If a = d, then the second eigenvector is parallel to the first and the geometric multiplicity of the eigenvalue a = d is 1. EIGENBASIS. If A has n different eigenvalues, then A has an eigenbasis, consisting of eigenvectors of A. DIAGONALIZATION. How does the matrix A look in an eigenbasis? If S is the matrix with the eigenvectors as columns, then we know B = S−1 AS . We have Sei = vi and ASei = λivi we know S−1 ASei = λiei. Therefore, B is diagonal with diagonal entries λi. EXAMPLE. A = 2 3 1 2 has the eigenvalues λ1 = 2 + √ 3 with eigenvector v1 = [ √ 3, 1] and the eigenvalues λ2 = 2 − √ 3 with eigenvector v2 = [− √ 3, 1]. Form S = √ 3 − √ 3 1 1 and check S−1 AS = D is diagonal. APPLICATION: FUNCTIONAL CALCULUS. Let A be the matrix in the above example. What is A100 + A37 −1? The trick is to diagonalize A: B = S−1 AS, then Bk = S−1 Ak S and We can compute A100 +A37 −1 = S(B100 + B37 − 1)S−1 . APPLICATION: SOLVING LINEAR SYSTEMS. x(t+1) = Ax(t) has the solution x(n) = An x(0). To compute An , we diagonalize A and get x(n) = SBn S−1 x(0). This is an explicit formula. SIMILAR MATRICES HAVE THE SAME EIGENVALUES. One can see this in two ways: 1) If B = S−1 AS and v is an eigenvector of B to the eigenvalue λ, then Sv is an eigenvector of A to the eigenvalue λ. 2) From det(S−1 AS) = det(A), we know that the characteristic polynomials fB(λ) = det(λ − B) = det(λ − S−1 AS) = det(S−1 (λ − AS) = det((λ − A) = fA(λ) are the same. CONSEQUENCES. 1) Because the characteristic polynomials of similar matrices agree, the trace tr(A) of similar matrices agrees. 2) The trace is the sum of the eigenvalues of A. (Compare the trace of A with the trace of the diagonalize matrix.) THE CAYLEY HAMILTON THEOREM. If A is diagonalizable, then fA(A) = 0. PROOF. The DIAGONALIZATION B = S−1 AS has the eigenvalues in the diagonal. So fA(B), which contains fA(λi) in the diagonal is zero. From fA(B) = 0 we get SfA(B)S−1 = fA(A) = 0. By the way: the theorem holds for all matrices: the coefficients of a general matrix can be changed a tiny bit so that all eigenvalues are different. For any such perturbations one has fA(A) = 0. Because the coefficients of fA(A) depend continuously on A, they are zero 0 in general. CRITERIA FOR SIMILARITY. • If A and B have the same characteristic polynomial and diagonalizable, then they are similar. • If A and B have a different determinant or trace, they are not similar. • If A has an eigenvalue which is not an eigenvalue of B, then they are not similar.
  • 48. WHY DO WE WANT TO DIAGONALIZE? 1) FUNCTIONAL CALCULUS. If p(x) = 1 + x + x2 + x3 /3! + x4 /4! be a polynomial and A is a matrix, then p(A) = 1 + A + A2 /2! + A3 /3! + A4 /4! is a matrix. If B = S−1 AS is diagonal with diagonal entries λi, then p(B) is diagonal with diagonal entries p(λi). And p(A) = S p(B)S−1 . This speeds up the calculation because matrix multiplication costs much. The matrix p(A) can be written down with three matrix multiplications, because p(B) is diagonal. 2) SOLVING LINEAR DIFFERENTIAL EQUATIONS. A differential equation ˙v = Av is solved by v(t) = eAt v(0), where eAt = 1 + At + A2 t2 /2! + A3 t3 /3!...)x(0). (Differentiate this sum with respect to t to get AeAt v(0) = Ax(t).) If we write this in an eigenbasis of A, then y(t) = eBt y(0) with the diagonal matrix B =   λ1 0 . . . 0 0 λ2 . . . 0 0 0 . . . λn  . In other words, we have then explicit solutions yj(t) = eλj t yj(0). Linear differential equations later in this course. It is important motivation. 3) STOCHASTIC DYNAMICS (i.e MARKOV PROCESSES). Complicated systems can be modeled by putting probabilities on each possible event and computing the probabilities that an event switches to any other event. This defines a transition matrix. Such a matrix always has an eigenvalue 1. The corresponding eigenvector is the stable probability distribution on the states. If we want to understand, how fast things settle to this equilibrium, we need to know the other eigenvalues and eigenvectors. MOLECULAR VIBRATIONS. While quantum mechanics describes the motion of atoms in molecules, the vibrations can be described classically, when treating the atoms as ”balls” connected with springs. Such ap- proximations are necessary when dealing with large atoms, where quantum mechanical computations would be too costly. Examples of simple molecules are white phosphorus P4, which has tetrahedronal shape or methan CH4 the simplest organic compound or freon, CF2Cl2 which is used in refrigerants. Caffeine or aspirin form more complicated molecules. Freon CF2Cl2 Caffeine C8H10N4O2 Aspirin C9H8O4 WHITE PHOSPHORUS VIBRATIONS. (Differential equations appear later, the context is motivation at this stage). Let x1, x2, x3, x4 be the positions of the four phosphorus atoms (each of them is a 3-vector). The inter- atomar forces bonding the atoms is modeled by springs. The first atom feels a force x2 − x1 + x3 − x1 + x4 − x1 and is accelerated in the same amount. Let’s just chose units so that the force is equal to the acceleration. Then ¨x1 = (x2 − x1) + (x3 − x1) + (x4 − x1) ¨x2 = (x3 − x2) + (x4 − x2) + (x1 − x2) ¨x3 = (x4 − x3) + (x1 − x3) + (x2 − x3) ¨x4 = (x1 − x4) + (x2 − x4) + (x3 − x4) which has the form ¨x = Ax, where the 4 × 4 ma- trix A =     −3 1 1 1 1 −3 1 1 1 1 −3 1 1 1 1 −3     v1 =     1 1 1 1     , v2 =     −1 0 0 1     , v3 =     −1 0 1 0     , v4 =     −1 1 0 0     are the eigenvectors to the eigenvalues λ1 = 0, λ2 = −4, λ3 = −4, λ3 = −4. With S = [v1v2v3v4], the matrix B = S−1 BS is diagonal with entries 0, −4, −4, −4. The coordinates yi = Sxi satisfy ¨y1 = 0, ¨y2 = −4y2, ¨y3 = −4y3, ¨y4 = −4y4 which we can solve y0 which is the center of mass satisfies y0 = a + bt (move molecule with constant speed). The motions yi = ai cos(2t) + bi sin(2t) of the other eigenvectors are oscillations, called normal modes. The general motion of the molecule is a superposition of these modes.
  • 49. COMPLEX EIGENVALUES Math 21b, O. Knill NOTATION. Complex numbers are written as z = x + iy = r exp(iφ) = r cos(φ) + ir sin(φ). The real number r = |z| is called the absolute value of z, the value φ is the argument and denoted by arg(z). Complex numbers contain the real numbers z = x+i0 as a subset. One writes Re(z) = x and Im(z) = y if z = x + iy. ARITHMETIC. Complex numbers are added like vectors: x + iy + u + iv = (x + u) + i(y + v) and multiplied as z ∗ w = (x + iy)(u + iv) = xu − yv + i(yu − xv). If z = 0, one can divide 1/z = 1/(x + iy) = (x − iy)/(x2 + y2 ). ABSOLUTE VALUE AND ARGUMENT. The absolute value |z| = x2 + y2 satisfies |zw| = |z| |w|. The argument satisfies arg(zw) = arg(z) + arg(w). These are direct consequences of the polar representation z = r exp(iφ), w = s exp(iψ), zw = rs exp(i(φ + ψ)). GEOMETRIC INTERPRETATION. If z = x + iy is written as a vector x y , then multiplication with an other complex number w is a dilation-rotation: a scaling by |w| and a rotation by arg(w). THE DE MOIVRE FORMULA. zn = exp(inφ) = cos(nφ) + i sin(nφ) = (cos(φ) + i sin(φ))n follows directly from z = exp(iφ) but it is magic: it leads for example to formulas like cos(3φ) = cos(φ)3 − 3 cos(φ) sin2 (φ) which would be more difficult to come by using geometrical or power series arguments. This formula is useful for example in integration problems like cos(x)3 dx, which can be solved by using the above deMoivre formula. THE UNIT CIRCLE. Complex numbers of length 1 have the form z = exp(iφ) and are located on the unit circle. The characteristic polynomial fA(λ) = λ5 − 1 of the matrix       0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0       has all roots on the unit circle. The roots exp(2πki/5), for k = 0, . . . , 4 lye on the unit circle. THE LOGARITHM. log(z) is defined for z = 0 as log |z| + iarg(z). For example, log(2i) = log(2) + iπ/2. Riddle: what is ii ? (ii = ei log(i) = eiiπ/2 = e−π/2 ). The logarithm is not defined at 0 and the imaginary part is define only up to 2π. For example, both iπ/2 and 5iπ/2 are equal to log(i). HISTORY. The struggle with √ −1 is historically quite interesting. Nagging questions appeared for example when trying to find closed solutions for roots of polynomials. Cardano (1501-1576) was one of the mathemati- cians who at least considered complex numbers but called them arithmetic subtleties which were ”as refined as useless”. With Bombelli (1526-1573), complex numbers found some practical use. Descartes (1596-1650) called roots of negative numbers ”imaginary”. Although the fundamental theorem of algebra (below) was still not proved in the 18th century, and complex numbers were not fully understood, the square root of minus one √ −1 was used more and more. Euler (1707- 1783) made the observation that exp(ix) = cos x + i sin x which has as a special case the magic formula eiπ + 1 = 0 which relate the constants 0, 1, π, e in one equation. For decades, many mathematicians still thought complex numbers were a waste of time. Others used complex numbers extensively in their work. In 1620, Girard suggested that an equation may have as many roots as its degree in 1620. Leibniz (1646-1716) spent quite a bit of time trying to apply the laws of algebra to complex numbers. He and Johann Bernoulli used imaginary numbers as integration aids. Lambert used complex numbers for map projections, d’Alembert used them in hydrodynamics, while Euler, D’Alembert and Lagrange used them in their incorrect proofs of the fundamental theorem of algebra. Euler write first the symbol i for √ −1.
  • 50. Gauss published the first correct proof of the fundamental theorem of algebra in his doctoral thesis, but still claimed in 1825 that the true metaphysics of the square root of −1 is elusive as late as 1825. By 1831 Gauss overcame his uncertainty about complex numbers and published his work on the geometric representation of complex numbers as points in the plane. In 1797, a Norwegian Caspar Wessel (1745-1818) and in 1806 a Swiss clerk named Jean Robert Argand (1768-1822) (who stated the theorem the first time for polynomials with complex coefficients) did similar work. But these efforts went unnoticed. William Rowan Hamilton (1805-1865) (who would also discover the quaternions while walking over a bridge) expressed in 1833 complex numbers as vectors. Complex numbers continued to develop to complex function theory or chaos theory, a branch of dynamical systems theory. Complex numbers are helpful in geometry in number theory or in quantum mechanics. Once believed fictitious they are now most ”natural numbers” and the ”natural numbers” themselves are in fact the most ”complex”. A philospher who asks ”does √ −1 really exist?” might be shown the representation of x + iy as x −y y x . When adding or multiplying such dilation-rotation matrices, they behave like complex numbers: for example 0 −1 1 0 plays the role of i. FUNDAMENTAL THEOREM OF ALGEBRA. (Gauss 1799) A polynomial of degree n has exactly n roots. CONSEQUENCE: A n × n MATRIX HAS n EIGENVALUES. The characteristic polynomial fA(λ) = λn + an−1λn−1 + . . . + a1λ + a0 satisfies fA(λ) = (λ − λ1) . . . (λ − λn), where λi are the roots of f. TRACE AND DETERMINANT. Comparing fA(λ) = (λ − λn) . . . (λ − λn) with λn − tr(A) + .. + (−1)n det(A) gives tr(A) = λ1 + · · · + λn, det(A) = λ1 · · · λn. COMPLEX FUNCTIONS. The characteristic polynomial is an example of a function f from C to C. The graph of this function would live in lC × lC which corresponds to a four dimensional real space. One can visualize the function however with the real-valued function z → |f(z)|. The figure to the left shows the contour lines of such a function z → |f(z)|, where f is a polynomial. ITERATION OF POLYNOMIALS. A topic which is off this course (it would be a course by itself) is the iteration of polynomials like fc(z) = z2 + c. The set of parameter values c for which the iterates fc(0), f2 c (0) = fc(fc(0)), . . . , fn c (0) stay bounded is called the Mandelbrot set. It is the fractal black region in the picture to the left. The now already dusty object appears everywhere, from photoshop plugins to decorations. In Mathematica, you can compute the set very quickly (see http://www.math.harvard.edu/computing/math/mandelbrot.m). COMPLEX NUMBERS IN MATHEMATICA OR MAPLE. In both computer algebra systems, the letter I is used for i = √ −1. In Maple, you can ask log(1 + I), in Mathematica, this would be Log[1 + I]. Eigenvalues or eigenvectors of a matrix will in general involve complex numbers. For example, in Mathematica, Eigenvalues[A] gives the eigenvalues of a matrix A and Eigensystem[A] gives the eigenvalues and the corresponding eigenvectors. EIGENVALUES AND EIGENVECTORS OF A ROTATION. The rotation matrix A = cos(φ) sin(φ) − sin(φ) cos(φ) has the characteristic polynomial λ2 − 2 cos(φ) + 1. The eigenvalues are cos(φ) ± cos2(φ) − 1 = cos(φ) ± i sin(φ) = exp(±iφ). The eigenvector to λ1 = exp(iφ) is v1 = −i 1 and the eigenvector to the eigenvector λ2 = exp(−iφ) is v2 = i 1 .
  • 51. STABILITY Math 21b, O. Knill LINEAR DYNAMICAL SYSTEM. A linear map x → Ax defines a dynamical system. Iterating the map produces an orbit x0, x1 = Ax, x2 = A2 = AAx, .... The vector xn = An x0 describes the situation of the system at time n. Where does xn go when time evolves? Can one describe what happens asymptotically when time n goes to infinity? In the case of the Fibonacci sequence xn which gives the number of rabbits in a rabbit population at time n, the population grows essentially exponentially. Such a behavior would be called unstable. On the other hand, if A is a rotation, then An v stays bounded which is a type of stability. If A is a dilation with a dilation factor < 1, then An v → 0 for all v, a thing which we will call asymptotic stability. The next pictures show experiments with some orbits An v with different matrices. 0.99 −1 1 0 stable (not asymptotic) 0.54 −1 0.95 0 asymptotic stable 0.99 −1 0.99 0 asymptotic stable 0.54 −1 1.01 0 unstable 2.5 −1 1 0 unstable 1 0.1 0 1 unstable ASYMPTOTIC STABILITY. The origin 0 is invariant under a linear map T(x) = Ax. It is called asymptot- ically stable if An (x) → 0 for all x ∈ IRn . EXAMPLE. Let A = p −q q p be a dilation rotation matrix. Because multiplication wich such a matrix is analogue to the multiplication with a complex number z = p+iq, the matrix An corresponds to a multiplication with (p+iq)n . Since |(p+iq)|n = |p+iq|n , the origin is asymptotically stable if and only if |p+iq| < 1. Because det(A) = |p+iq|2 = |z|2 , rotation-dilation matrices A have an asymptotic stable origin if and only if |det(A)| < 1. Dilation-rotation matrices p −q q p have eigenvalues p ± iq and can be diagonalized in the complex. EXAMPLE. If a matrix A has an eigenvalue |λ| ≥ 1 to an eigenvector v, then An v = λn v, whose length is |λn | times the length of v. So, we have no asymptotic stability if an eigenvalue satisfies |λ| ≥ 1. STABILITY. The book also writes ”stable” for ”asymptotically stable”. This is ok to abbreviate. Note however that the commonly used term ”stable” also includes linear maps like rotations, reflections or the identity. It is therefore preferable to leave the attribute ”asymptotic” in front of ”stable”. ROTATIONS. Rotations cos(φ) − sin(φ) sin(φ) cos(φ) have the eigenvalue exp(±iφ) = cos(φ) + i sin(φ) and are not asymptotically stable. DILATIONS. Dilations r 0 0 r have the eigenvalue r with algebraic and geometric multiplicity 2. Dilations are asymptotically stable if |r| < 1. CRITERION. A linear dynamical system x → Ax has an asymptotically stable origin if and only if all its eigenvalues have an absolute value < 1. PROOF. We have already seen in Example 3, that if one eigenvalue satisfies |λ| > 1, then the origin is not asymptotically stable. If |λi| < 1 for all i and all eigenvalues are different, there is an eigenbasis v1, . . . , vn. Every x can be written as x = n j=1 xjvj. Then, An x = An ( n j=1 xjvj) = n j=1 xjλn j vj and because |λj|n → 0, there is stability. The proof of the general (nondiagonalizable) case will be accessible later.
  • 52. THE 2-DIMENSIONAL CASE. The characteristic polynomial of a 2 × 2 matrix A = a b c d is fA(λ) = λ2 − tr(A)λ + det(A). If c = 0, the eigenvalues are λ± = tr(A)/2 ± (tr(A)/2)2 − det(A). If the discriminant (tr(A)/2)2 − det(A) is nonnegative, then the eigenvalues are real. This happens below the parabola, where the discriminant is zero. CRITERION. In two dimensions we have asymptotic stability if and only if (tr(A), det(A)) is contained in the stability triangle bounded by the lines det(A) = 1, det(A) = tr(A)−1 and det(A) = −tr(A)−1. PROOF. Write T = tr(A)/2, D = det(A). If |D| ≥ 1, there is no asymptotic stability. If λ = T + √ T2 − D = ±1, then T2 − D = (±1 − T)2 and D = 1 ± 2T. For D ≤ −1 + |2T| we have a real eigenvalue ≥ 1. The conditions for stability is therefore D > |2T| − 1. It implies automatically D > −1 so that the triangle can be described shortly as |tr(A)| − 1 < det(A) < 1 . EXAMPLES. 1) The matrix A = 1 1/2 −1/2 1 has determinant 5/4 and trace 2 and the origin is unstable. It is a dilation- rotation matrix which corresponds to the complex number 1 + i/2 which has an absolute value > 1. 2) A rotation A is never asymptotically stable: det(A)1 and tr(A) = 2 cos(φ). Rotations are the upper side of the stability triangle. 3) A dilation is asymptotically stable if and only if the scaling factor has norm < 1. 4) If det(A) = 1 and tr(A) < 2 then the eigenvalues are on the unit circle and there is no asymptotic stability. 5) If det(A) = −1 (like for example Fibonacci) there is no asymptotic stability. For tr(A) = 0, we are a corner of the stability triangle and the map is a reflection, which is not asymptotically stable neither. SOME PROBLEMS. 1) If A is a matrix with asymptotically stable origin, what is the stability of 0 with respect to AT ? 2) If A is a matrix which has an asymptotically stable origin, what is the stability with respect to to A−1 ? 3) If A is a matrix which has an asymptotically stable origin, what is the stability with respect to to A100 ? ON THE STABILITY QUESTION. For general dynamical systems, the question of stability can be very difficult. We deal here only with linear dynamical systems, where the eigenvalues determine everything. For nonlinear systems, the story is not so simple even for simple maps like the Henon map. The questions go deeper: it is for example not known, whether our solar system is stable. We don’t know whether in some future, one of the planets could get expelled from the solar system (this is a mathematical question because the escape time would be larger than the life time of the sun). For other dynamical systems like the atmosphere of the earth or the stock market, we would really like to know what happens in the near future ... A pioneer in stability theory was Alek- sandr Lyapunov (1857-1918). For nonlin- ear systems like xn+1 = gxn − x3 n − xn−1 the stability of the origin is nontrivial. As with Fibonacci, this can be written as (xn+1, xn) = (gxn − x2 n − xn−1, xn) = A(xn, xn−1) called cubic Henon map in the plane. To the right are orbits in the cases g = 1.5, g = 2.5. The first case is stable (but proving this requires a fancy theory called KAM theory), the second case is unstable (in this case actually the linearization at 0 determines the picture).
  • 53. SYMMETRIC MATRICES Math 21b, O. Knill SYMMETRIC MATRICES. A matrix A with real entries is symmetric, if AT = A. EXAMPLES. A = 1 2 2 3 is symmetric, A = 1 1 0 3 is not symmetric. EIGENVALUES OF SYMMETRIC MATRICES. Symmetric matrices A have real eigenvalues. PROOF. The dot product is extend to complex vectors as (v, w) = i viwi. For real vectors it satisfies (v, w) = v · w and has the property (Av, w) = (v, AT w) for real matrices A and (λv, w) = λ(v, w) as well as (v, λw) = λ(v, w). Now λ(v, v) = (λv, v) = (Av, v) = (v, AT v) = (v, Av) = (v, λv) = λ(v, v) shows that λ = λ because (v, v) = 0 for v = 0. EXAMPLE. A = p −q q p has eigenvalues p + iq which are real if and only if q = 0. EIGENVECTORS OF SYMMETRIC MATRICES. Symmetric matrices have an orthonormal eigenbasis PROOF. If Av = λv and Aw = µw. The relation λ(v, w) = (λv, w) = (Av, w) = (v, AT w) = (v, Aw) = (v, µw) = µ(v, w) is only possible if (v, w) = 0 if λ = µ. WHY ARE SYMMETRIC MATRICES IMPORTANT? In applications, matrices are often symmetric. For ex- ample in geometry as generalized dot products v·Av, or in statistics as correlation matrices Cov[Xk, Xl] or in quantum mechanics as observables or in neural networks as learning maps x → sign(Wx) or in graph theory as adjacency matrices etc. etc. Symmetric matrices play the same role as real numbers do among the complex numbers. Their eigenvalues often have physical or geometrical interpretations. One can also calculate with symmetric matrices like with numbers: for example, we can solve B2 = A for B if A is symmetric matrix and B is square root of A.) This is not possible in general: try to find a matrix B such that B2 = 0 1 0 0 ... RECALL. We have seen when an eigenbasis exists, a matrix A can be transformed to a diagonal matrix B = S−1 AS, where S = [v1, ..., vn]. The matrices A and B are similar. B is called the diagonalization of A. Similar matrices have the same characteristic polynomial det(B − λ) = det(S−1 (A − λ)S) = det(A − λ) and have therefore the same determinant, trace and eigenvalues. Physicists call the set of eigenvalues also the spectrum. They say that these matrices are isospectral. The spectrum is what you ”see” (etymologically the name origins from the fact that in quantum mechanics the spectrum of radiation can be associated with eigenvalues of matrices.) SPECTRAL THEOREM. Symmetric matrices A can be diagonalized B = S−1 AS with an orthogonal S. PROOF. If all eigenvalues are different, there is an eigenbasis and diagonalization is possible. The eigenvectors are all orthogonal and B = S−1 AS is diagonal containing the eigenvalues. In general, we can change the matrix A to A = A + (C − A)t where C is a matrix with pairwise different eigenvalues. Then the eigenvalues are different for all except finitely many t. The orthogonal matrices St converges for t → 0 to an orthogonal matrix S and S diagonalizes A. WAIT A SECOND ... Why could we not perturb a general matrix At to have disjoint eigenvalues and At could be diagonalized: S−1 t AtSt = Bt? The problem is that St might become singular for t → 0. See problem 5) first practice exam. EXAMPLE 1. The matrix A = a b b a has the eigenvalues a + b, a − b and the eigenvectors v1 = 1 1 / √ 2 and v2 = −1 1 / √ 2. They are orthogonal. The orthogonal matrix S = v1 v2 diagonalized A.
  • 54. EXAMPLE 2. The 3 × 3 matrix A =   1 1 1 1 1 1 1 1 1   has 2 eigenvalues 0 to the eigenvectors 1 −1 0 , 1 0 −1 and one eigenvalue 3 to the eigenvector 1 1 1 . All these vectors can be made orthogonal and a diagonalization is possible even so the eigenvalues have multiplicities. SQUARE ROOT OF A MATRIX. How do we find a square root of a given symmetric matrix? Because S−1 AS = B is diagonal and we know how to take a square root of the diagonal matrix B, we can form C = S √ BS−1 which satisfies C2 = S √ BS−1 S √ BS−1 = SBS−1 = A. RAYLEIGH FORMULA. We write also (v, w) = v ·w. If v(t) is an eigenvector of length 1 to the eigenvalue λ(t) of a symmetric matrix A(t) which depends on t, differentiation of (A(t) − λ(t))v(t) = 0 with respect to t gives (A −λ )v+(A−λ)v = 0. The symmetry of A−λ implies 0 = (v, (A −λ )v)+(v, (A−λ)v ) = (v, (A −λ )v). We see that the Rayleigh quotient λ = (A v, v) is a polynomial in t if A(t) only involves terms t, t2 , . . . , tm . The formula shows how λ(t) changes, when t varies. For example, A(t) = 1 t2 t2 1 has for t = 2 the eigenvector v = [1, 1]/ √ 2 to the eigenvalue λ = 5. The formula tells that λ (2) = (A (2)v, v) = ( 0 4 4 0 v, v) = 4. Indeed, λ(t) = 1 + t2 has at t = 2 the derivative 2t = 4. EXHIBITION. ”Where do symmetric matrices occur?” Some informal motivation: I) PHYSICS: In quantum mechanics a system is described with a vector v(t) which depends on time t. The evolution is given by the Schroedinger equation ˙v = i¯hLv, where L is a symmetric matrix and ¯h is a small number called the Planck constant. As for any linear differential equation, one has v(t) = ei¯hLt v(0). If v(0) is an eigenvector to the eigenvalue λ, then v(t) = eit¯hλ v(0). Physical observables are given by symmetric matrices too. The matrix L represents the energy. Given v(t), the value of the observable A(t) is v(t) · Av(t). For example, if v is an eigenvector to an eigenvalue λ of the energy matrix L, then the energy of v(t) is λ. This is called the Heisenberg picture. In order that v · A(t)v = v(t) · Av(t) = S(t)v · AS(t)v we have A(t) = S(T)∗ AS(t), where S∗ = ST is the correct generalization of the adjoint to complex matrices. S(t) satisfies S(t)∗ S(t) = 1 which is called unitary and the complex analogue of orthogonal. The matrix A(t) = S(t)∗ AS(t) has the same eigenvalues as A and is similar to A. II) CHEMISTRY. The adjacency matrix A of a graph with n vertices determines the graph: one has Aij = 1 if the two vertices i, j are connected and zero otherwise. The matrix A is symmetric. The eigenvalues λj are real and can be used to analyze the graph. One interesting question is to what extent the eigenvalues determine the graph. In chemistry, one is interested in such problems because it allows to make rough computations of the electron density distribution of molecules. In this so called H¨uckel theory, the molecule is represented as a graph. The eigenvalues λj of that graph approximate the energies an electron on the molecule. The eigenvectors describe the electron density distribution. The Freon molecule for example has 5 atoms. The adjacency matrix is       0 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0       . This matrix A has the eigenvalue 0 with multiplicity 3 (ker(A) is obtained im- mediately from the fact that 4 rows are the same) and the eigenvalues 2, −2. The eigenvector to the eigenvalue ±2 is ±2 1 1 1 1 T . III) STATISTICS. If we have a random vector X = [X1, · · · , Xn] and E[Xk] denotes the expected value of Xk, then [A]kl = E[(Xk − E[Xk])(Xl − E[Xl])] = E[XkXl] − E[Xk]E[Xl] is called the covariance matrix of the random vector X. It is a symmetric n × n matrix. Diagonalizing this matrix B = S−1 AS produces new random variables which are uncorrelated. For example, if X is is the sum of two dice and Y is the value of the second dice then E[X] = [(1 + 1) + (1 + 2) + ... + (6 + 6)]/36 = 7, you throw in average a sum of 7 and E[Y ] = (1 + 2 + ... + 6)/6 = 7/2. The matrix entry A11 = E[X2 ] − E[X]2 = [(1 + 1) + (1 + 2) + ... + (6 + 6)]/36 − 72 = 35/6 known as the variance of X, and A22 = E[Y 2 ] − E[Y ]2 = (12 + 22 + ... + 62 )/6 − (7/2)2 = 35/12 known as the variance of Y and A12 = E[XY ] − E[X]E[Y ] = 35/12. The covariance matrix is the symmetric matrix A = 35/6 35/12 35/12 35/12 .
  • 55. CONTINUOUS DYNAMICAL SYSTEMS I Math 21b, O. Knill CONTINUOUS DYNAMICAL SYSTMES. A differ- ential equation d dt x = f(x) defines a dynamical sys- tem. The solutions is a curve x(t) which has the velocity vector f(x(t)) for all t. We often write ˙x for d dt x. ONE DIMENSION. A system ˙x = g(x, t) (written in the form ˙x = g(x, t), ˙t = 1) and often has explicit solutions • If ˙x = g(t), then x(t) = t 0 g(t) dt. • If ˙x = h(x), then dx/h(x) = dt and so t = x 0 dx/h(x) = H(x) so that x(t) = H−1 (t). • If ˙x = g(t)/h(x), then H(x) = x 0 h(x) dx = t 0 g(t) dt = G(t) so that x(t) = H−1 (G(t)). In general, we have no closed form solutions in terms of known functions. The solution x(t) = t 0 e−t2 dt of ˙x = e−t2 for example can not be expressed in terms of functions exp, sin, log, √ · etc but it can be solved using Taylor series: because e−t2 = 1 − t2 + t4 /2! − t6 /3! + . . . taking coefficient wise the anti-derivatives gives: x(t) = t − t3 /3 + t4 /(32!) − t7 /(73!) + . . .. HIGHER DIMENSIONS. In higher dimensions, chaos can set in and the dynamical system can become unpredictable. The nonlinear Lorentz system to the right ˙x(t) = 10(y(t) − x(t)), ˙y(t) = −x(t)z(t) + 28x(t) − y(t), ˙z(t) = x(t) ∗ y(t) − 8z(t)/3 shows a ”strange attractor”. Evenso completely deterministic: (from x(0) all the path x(t) is determined), there are observables which can be used as a ran- dom number generator. The Duffing sys- tem ¨x + ˙x.10 − x + x3 − 12 cos(t) = 0 to the left can be written in the form ˙v = f(v) with a vector v = (x, ˙x, t). 1D LINEAR DIFFERENTIAL EQUATIONS. A linear differential equation in one dimension ˙x = λx has the solution is x(t) = eλt x(0) . This differential equation appears • as population models with λ > 0: birth rate of the popolation is proportional to its size. • as a model for radioactive decay with λ < 0: the rate of decay is proportional to the number of atoms. LINEAR DIFFERENTIAL EQUATIONS IN HIGHER DIMENSIONS. Linear dynamical systems have the form ˙x = Ax, where A is a matrix. Note that the origin 0 is an equilibrium point: if x(0) = 0, then x(t) = 0 for all t. The general solution is x(t) = eAt = 1 + At + A2 t2 /2! + . . . because ˙x(t) = A + 2A2 t/2! + . . . = A(1 + At + A2 t2 /2! + . . .) = AeAt = Ax(t). If B = S−1 AS is diagonal with the eigenvalues λj = aj + ibj in the diagonal, then y = S−1 x satisfies y(t) = eBt and therefore yj(t) = eλj t yj(0) = eaj t eibj t yj(0). The solutions in the original coordinates are x(t) = Sy(t). PHASE PORTRAITS. For differential equations ˙x = f(x) in 2D one can draw the vector field x → f(x). The solution x(t) is tangent to the vector f(x(t)) everywhere. The phase portraits together with some solution
  • 56. -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 -1 -0.5 0.5 1 UNDERSTANDING A DIFFERENTIAL EQUATION. The closed form solution like x(t) = eAt x(0) for ˙x = Ax is actually quite useless. One wants to understand the solution quantitatively. Questions one wants to answer are: what happens in the long term? Is the origin stable, are there periodic solutions. Can one decompose the system into simpler subsystems? We will see that diagonalisation allows to understand the system: by decomposing it into one-dimensional linear systems, which can be analyzed seperately. In general, ”understanding” can mean different things: Plotting phase portraits. Computing solutions numerically and esti- mate the error. Finding special solutions. Predicting the shape of some orbits. Finding regions which are invariant. Finding special closed form solutions x(t). Finding a power series x(t) = n antn in t. Finding quantities which are unchanged along the flow (called ”Integrals”). Finding quantities which increase along the flow (called ”Lyapunov functions”). LINEAR STABILITY. A linear dynamical system ˙x = Ax with diagonalizable A is linearly stable if and only if aj = Re(λj) < 0 for all eigenvalues λj of A. PROOF. We see that from the explicit solutions yj(t) = eaj t eibj t yj(0) in the basis consisting of eigenvectors. Now, y(t) → 0 if and only if aj < 0 for all j and x(t) = Sy(t) → 0 if and only if y(t) → 0. RELATION WITH DISCRETE TIME SYSTEMS. From ˙x = Ax, we obtain x(t + 1) = Bx(t), with the matrix B = eA . The eigenvalues of B are µj = eλj . Now |µj| < 1 if and only if Reλj < 0. The criterium for linear stability of discrete dynamical systems is compatible with the criterium for linear stability of ˙x = Ax. EXAMPLE 1. The system ˙x = y, ˙y = −x can in vector form v = (x, y) be written as ˙v = Av, with A = 0 −1 1 0 . The matrix A has the eigenvalues i, −i. After a coordi- nate transformation w = S−1 v we get with w = (a, b) the differential equations ˙a = ia, ˙b = −i = b which has the solutions a(t) = eit a(0), b(t) = e−it b(0). The original coordates satisfy x(t) = cos(t)x(0) − sin(t)y(0), y(t) = sin(t)x(0) + cos(t)y(0). Indeed eAt is a rotation in the plane. -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1 EXAMPLE 2. A harmonic oscillator ¨x = −x can be written with y = ˙x as ˙x = y, ˙y = −x (see Example 1). The general solution is therefore x(t) = cos(t)x(0) − sin(t) ˙x(0). EXAMPLE 3. We take two harmonic oscillators and couple them: ¨x1 = −x1 − (x2 − x1), ¨x2 = −x2 + (x2 − x1). For small xi one can simulate this with two coupled penduli. The system can be written as ¨v = Av, with A = −1 + − − −1 + . The matrix A has an eigenvalue λ1 = −1 to the eigenvector 1 1 and an eigenvalue λ2 = −1 + 2 ∗ to the eigenvector 1 −1 . The coordinate change S is S = 1 1 1 −1 . It has the inverse S−1 = 1 1 −1 1 /2. In the coordinates w = S−1 v = [y1, y2], we have oscillations ¨y1 = −y1 corresponding to the case x1 − x2 = 0 (the pendula swing synchroneous) and ¨y2 = −(1 − 2 )y2 corresponding to x1 + x2 = 0 (the pendula swing against each other).
  • 57. CONTINUOUS DYNAMICAL SYSTEMS II Math 21b, O. Knill COMPLEX LINEAR 1D CASE. ˙x = λx for λ = a + ib has solution x(t) = eat eibt x(0) and norm ||x(t)|| = eat ||x(0)||. OSCILLATOR: The system ¨x = −λx has the solution x(t) = cos( √ λt)x(0) + sin( √ λt) ˙x(0)/ √ λ. DERIVATION. ˙x = y, ˙y = −λx and in matrix form as ˙x ˙y = 0 −1 λ 0 x y = A x y and because A has eigenvalues ±i √ λ, the new coordinates move as a(t) = ei √ λt a(0) and b(t) = e−i √ λt b(0). Writing this in the original coordinaes x(t) y(t) = S a(t) b(t) and fixing the constants gives x(t), y(t). EXAMPLE. THE SPINNER. The spinner is a rigid body attached to a spring aligned around the z-axes. The body can rotate around the z-axes and bounce up and down. The two motions are coupled in the following way: when the spinner winds up in the same direction as the spring, the spring gets tightend and the body gets a lift. If the spinner winds up to the other direction, the spring becomes more relaxed and the body is lowered. Instead of reducing the system to a 4D first order system, system d dt x = Ax, we will keep the second time derivative and diagonalize the 2D system d2 dt2 x = Ax, where we know how to solve the one dimensional case d2 dt2 v = −λv as v(t) = A cos( √ λt) + B sin( √ λt) with constants A, B depending on the initial conditions, v(0), ˙v(0). SETTING UP THE DIFFERENTIAL EQUATION. x is the angle and y the height of the body. We put the coordinate system so that y = 0 is the point, where the body stays at rest if x = 0. We assume that if the spring is winded up with an an- gle x, this produces an upwards force x and a momentum force −3x. We furthermore assume that if the body is at position y, then this produces a momentum y onto the body and an upwards force y. The differential equations ¨x = −3x + y ¨y = −y + x can be written as ¨v = Av = −3 1 1 −1 v. FINDING GOOD COORDINATES w = S−1 v is obtained with getting the eigenvalues and eigenvectors of A: λ1 = −2 − √ 2, λ2 = −2 + √ 2 v1 = −1 − √ 2 1 , v1 = −1 + √ 2 1 so that S = −1 − √ 2 −1 + √ 2 1 1 . SOLVE THE SYSTEM ¨a = λ1a,¨b = λ2b IN THE GOOD COORDINATES a b = S−1 x y . a(t) = A cos(ω1t) + B sin(ω1)t, ω1 = √ −λ1, b(t) = C cos(ω2t) + D sin(ω2)t, ω2 = √ −λ2. THE SOLUTION IN THE ORIGINAL COORDINATES. x(t) y(t) = S a(t) b(t) . At t = 0 we know x(0), y(0), ˙x(0), ˙y(0). This fixes the constants in x(t) = A1 cos(ω1t) + B1 sin(ω1t) +A2 cos(ω2t) + B2 sin(ω2t). The curve (x(t), y(t)) traces a Lyssajoux curve: -0.6 -0.4 -0.2 0.2 0.4 0.6 x[t] -1 -0.5 0.5 1 y[t] ASYMPTOTIC STABILITY. A linear system ˙x = Ax in the 2D plane is asymptotically stable if and only if det(A) > 0 and tr(A) < 0. PROOF. If the eigenvalues λ1, λ2 of A are real then both beeing negative is equivalent with λ1λ2 = det(A) > 0 and tr(A) = λ1 + λ2 < 0. If λ1 = a + ib, λ2 = a − ib, then a negative a is equivalent to λ1 + λ2 = 2a < 0 and λ1λ2 = a2 + b2 > 0.
  • 58. ASYMPTOTIC STABILITY COMPARISON OF DISCRETE AND CONTINUOUS SITUATION. The trace and the determimant are independent of the basis, they can be computed fast, and are real if A is real. It is therefore convenient to determine the region in the tr − det-plane, where continuous or discrete dynamical systems are asymptotically stable. While the continuous dynamical system is related to a discrete system, it is important not to mix these two situations up. Continuous dynamical system. Discrete dynamical system. Stability of ˙x = Ax (x(t + 1) = eA x(t)). Stability of x(t + 1) = Ax Stability in det(A) > 0, tr(A) > 0 Stability in |tr(A)| − 1 < det(A) < 1 Stability if Re(λ1) < 0, Re(λ2) < 0. Stability if |λ1| < 1,|λ2| < 1. PHASEPORTRAITS. (In two dimensions we can plot the vector field, draw some trajectories) -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 < 0 λ2 < 0, i.e A = −2 0 0 −3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 < 0 λ2 > 0, i.e A = −2 0 0 3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 > 0 λ2 > 0, i.e A = 2 0 0 3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = 0 λ2 < 0, i.e A = 0 0 0 −3 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = 0 λ2 = 0, i.e A = 0 0 1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = a + ib, a < 0 λ2 = a − ib, i.e A = −1 1 −1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = a + ib, a > 0 λ2 = a − ib, i.e A = 1 1 −1 0 -2 -1.5 -1 -0.5 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0.5 1 1.5 2 λ1 = ib λ2 = −ib, i.e A = 0 1 −1 0
  • 59. NONLINEAR DYNAMICAL SYSTEMS Math 21b, O. Knill SUMMARY. For linear systems ˙x = Ax, the eigenvalues of A determine the behavior completely. For nonlinear systems explicit formulas for solutions are no more available in general. It even also happen that orbits go go off to infinity in finite time like in ˙x = x2 with solution x(t) = −1/(t − x(0)). With x(0) = 1 it reaches infinity at time t = 1. Linearity is often too crude. The exponential growth ˙x = ax of a bacteria colony for example is slowed down due to lack of food and the logistic model ˙x = ax(1 − x/M) would be more accurate, where M is the population size for which bacteria starve so much that the growth has stopped: x(t) = M, then ˙x(t) = 0. Nonlinear systems can be investigated with qualitative methods. In 2 dimensions ˙x = f(x, y), ˙y = g(x, y), where chaos does not happen, the analysis of equilibrium points and linear approximation at those points in general allows to understand the system. Also in higher dimensions, where ODE’s can have chaotic solutions, the analysis of equilibrium points and linear approximation at those points is a place, where linear algebra becomes useful. EQUILIBRIUM POINTS. A point x0 is called an equilibrium point of ˙x = f(x) if f(x0) = 0. If x(0) = x0 then x(t) = x0 for all times. The system ˙x = x(6−2x−y), ˙y = y(4−x−y) for example has the four equilibrium points (0, 0), (3, 0), (0, 4), (2, 2). JACOBIAN MATRIX. If x0 is an equilibrium point for ˙x = f(x) then [A]ij = ∂ ∂xj fi(x) is called the Jacobian at x0. For two dimensional systems ˙x = f(x, y) ˙y = g(x, y) this is the 2 × 2 matrix A= ∂f ∂x (x, y) ∂f ∂y (x, y) ∂g ∂x (x, y) ∂g ∂y (x, y) . The linear ODE ˙y = Ay with y = x − x0 approximates the nonlinear system well near the equilibrium point. The Jacobian is the linear approximation of F = (f, g) near x0. VECTOR FIELD. In two dimensions, we can draw the vector field by hand: attaching a vector (f(x, y), g(x, y)) at each point (x, y). To find the equilibrium points, it helps to draw the nullclines {f(x, y) = 0}, {g(x, y) = 0}. The equilibrium points are located on intersections of nullclines. The eigenvalues of the Jacobeans at equilibrium points allow to draw the vector field near equilibrium points. This information is sometimes enough to draw the vector field by hand. MURRAY SYSTEM (see handout) ˙x = x(6 − 2x − y), ˙y = y(4 − x − y) has the nullclines x = 0, y = 0, 2x + y = 6, x + y = 5. There are 4 equilibrium points (0, 0), (3, 0), (0, 4), (2, 2). The Jacobian matrix of the system at the point (x0, y0) is 6 − 4x0 − y0 −x0 −y0 4 − x0 − 2y0 . Note that without interaction, the two systems would be logistic systems ˙x = x(6 − 2x), ˙y = y(4 − y). The additional −xy is the competition. Equilibrium Jacobean Eigenvalues Nature of equilibrium (0,0) 6 0 0 4 λ1 = 6, λ2 = 4 Unstable source (3,0) −6 −3 0 1 λ1 = −6, λ2 = 1 Hyperbolic saddle (0,4) 2 0 −4 −4 λ1 = 2, λ2 = −4 Hyperbolic saddle (2,2) −4 −2 −2 −2 λi = −3 ± √ 5 Stable sink USING TECHNOLOGY (Example: Mathematica). Plot the vector field: Needs["Graphics‘PlotField‘"] f[x_,y_]:={x(6-2x-y),y(5-x-y)};PlotVectorField[f[x,y],{x,0,4},{y,0,4}] Find the equilibrium solutions: Solve[{x(6-2x-y)==0,y(5-x-y)==0},{x,y}] Find the Jacobian and its eigenvalues at (2, 2): A[{x_,y_}]:={{6-4x,-x},{-y,5-x-2y}};Eigenvalues[A[{2,2}]] Plotting an orbit: S[u_,v_]:=NDSolve[{x’[t]==x[t](6-2x[t]-y[t]),y’[t]==y[t](5-x[t]-y[t]),x[0]==u,y[0]==v},{x,y},{t,0,1}] ParametricPlot[Evaluate[{x[t],y[t]}/.S[0.3,0.5]],{t,0,1},AspectRatio->1,AxesLabel->{"x[t]","y[t]"}]
  • 60. VOLTERRA-LODKA SYSTEMS are systems of the form ˙x = 0.4x − 0.4xy ˙y = −0.1y + 0.2xy This example has equilibrium points (0, 0) and (1/2, 1). It describes for example a tuna shark population. The tuna pop- ulation x(t) becomes smaller with more sharks. The shark population grows with more tuna. Volterra ex- plained so first the oscillation of fish populations in the Mediterrian sea. EXAMPLE: HAMILTONIAN SYS- TEMS are systems of the form ˙x = ∂yH(x, y) ˙y = −∂xH(x, y) where H is called the energy. Usu- ally, x is the position and y the mo- mentum. THE PENDULUM: H(x, y) = y2 /2 − cos(x). ˙x = y ˙y = − sin(x) x is the angle between the pendulum and y-axes, y is the angular velocity, sin(x) is the potential. (See homework). Hamiltonian systems preserve energy H(x, y) because d dt H(x(t), y(t)) = ∂xH(x, y) ˙x + ∂yH(x, y) ˙y = ∂xH(x, y)∂yH(x, y) − ∂yH(x, y)∂xH(x, y) = 0. Orbits stay on level curves of H. EXAMPLE: LIENHARD SYSTEMS are differential equations of the form ¨x + ˙xF (x) + G (x) = 0. With y = ˙x + F(x), G (x) = g(x), this gives ˙x = y − F(x) ˙y = −g(x) VAN DER POL EQUATION ¨x+(x2 − 1) ˙x + x = 0 appears in electrical engineering, biology or biochemistry. Since F(x) = x3 /3 − x, g(x) = x. ˙x = y − (x3 /3 − x) ˙y = −x Lienhard systems have limit cycles. A trajectory always ends up on that limit cycle. This is useful for engineers, who need oscillators which are stable under changes of parameters. One knows: if g(x) > 0 for x > 0 and F has exactly three zeros 0, a, −a, F (0) < 0 and F (x) ≥ 0 for x > a and F(x) → ∞ for x → ∞, then the corresponding Lienhard system has exactly one stable limit cycle. CHAOS can occur for systems ˙x = f(x) in three dimensions. For example, ¨x = f(x, t) can be written with (x, y, z) = (x, ˙x, t) as ( ˙x, ˙y, ˙z) = (y, f(x, z), 1). The system ¨x = f(x, ˙x) becomes in the coordinates (x, ˙x) the ODE ˙x = f(x) in four dimensions. The term chaos has no uniform definition, but usually means that one can find a copy of a random number generator embedded inside the system. Chaos theory is more than 100 years old. Basic insight had been obtained by Poincar´e. During the last 30 years, the subject exploded to its own branch of physics, partly due to the availability of computers. ROESSLER SYSTEM ˙x = −(y + z) ˙y = x + y/5 ˙z = 1/5 + xz − 5.7z LORENTZ SYSTEM ˙x = 10(y − x) ˙y = −xz + 28x − y ˙z = xy − 8z 3 These two systems are examples, where one can observe strange attractors. THE DUFFING SYSTEM ¨x + ˙x 10 − x + x3 − 12 cos(t) = 0 ˙x = y ˙y = −y/10 − x + x3 − 12 cos(z) ˙z = 1 The Duffing system models a metal- lic plate between magnets. Other chaotic examples can be obtained from mechanics like the driven pendulum ¨x + sin(x) − cos(t) = 0.
  • 61. FUNCTION SPACES/LINEAR MAPS, Math 21b, O. Knill FROM VECTORS TO FUNCTIONS. Vectors can be displayed in different ways:        1 2 5 3 2 6         1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 Listing the (i, vi) can also be interpreted as the graph of a function f : 1, 2, 3, 4, 5, 6 → R, where f(i) = vi. LINEAR SPACES. A space X in which we can add, scalar multiplications and where basic laws like commutativity, distributivity and associativity hold is called a linear space. Examples: • Lines, planes and more generally, the n-dimensional Euclidean space. • Pn, the space of all polynomials of degree n. • The space P of all polynomials. • C∞ , the space of all smooth functions on the line • C0 , the space of all continuous functions on the line. • C1 , the space of all differentiable functions on the line. • C∞ (R3 ) the space of all smooth functions in space. • L2 the space of all functions on the line for which f2 is integrable and ∞ ∞ f2 (x) dx < ∞. In all these function spaces, the function f(x) = 0 which is constantly 0 is the zero function. WHICH OF THE FOLLOWING ARE LINEAR SPACES? The space X of all polynomials of the form f(x) = ax3 + bx4 + cx5 The space X of all continuous functions on the unit interval [−1, 1] which vanish at −1 and 1. It contains for example f(x) = x2 − |x|. The space X of all smooth periodic functions f(x+1) = f(x). Example f(x) = sin(2πx)+ cos(6πx). The space X = sin(x) + C∞ (R) of all smooth functions f(x) = sin(x) + g, where g is a smooth function. The space X of all trigonometric polynomials f(x) = a0 + a1 sin(x) + a2 sin(2x) + ... + an sin(nx). The space X of all smooth functions on R which satisfy f(1) = 1. It contains for example f(x) = 1 + sin(x) + x. The space X of all continuous functions on R which satisfy f(2) = 0 and f(10) = 0. The space X of all smooth functions on R which satisfy lim|x|→∞ f(x) = 0. The space X of all continuous functions on R which satisfy lim|x|→∞ f(x) = 1. The space X of all smooth functions on R of compact support: for every f, there exists an interval I such that f(x) = 0 outside that interval. The space X of all smooth functions on R2 .
  • 62. LINEAR TRANSFORMATIONS. A map T between linear spaces is called lin- ear if T(x + y) = T(x) + T(y), T(λx) = λT(x), T(0) = 0. Examples: • Df(x)=f (x) on C∞ • Tf(x) = x 0 f(x) dx on C0 • Tf(x) = (f(0), f(1), f(2), f(3)) on C∞ . • Tf(x) = sin(x)f(x) on C∞ • Tf(x) = ( 1 0 f(x)g(x) dx)g(x) on C0 [0, 1]. WHICH OF THE FOLLOWING MAPS ARE LINEAR TRANSFORMATIONS? The map T(f) = f (x) on X = C∞ (T). The map T(f) = 1 + f (x) on X = C∞ (R). The map T(f)(x) = sin(x)f(x) on X = C∞ (R). The map T(f)(x) = f(x)/x on X = C∞ (R). The map T(f)(x) = x 0 f(x) dx on X = C([0, 1]). The map T(f)(x) = f(x + √ 2) on X = C∞ (R). The map T(f)(x) = ∞ ∞ f(x − s) sin(s) ds on C0 . The map T(f)(x) = f (x) + f(2) on C∞ (R). The map T(f)(x) = f (x) + f(2) + 1 on C∞ (R). The map T(f)(x, y, z) = fxx(x, y, z) + fyy(x, y, z) + fzz(x, y, z) + 1/(|x|)f(x, y, z) The map T(f)(x) = f(x2 ). The map T(f)(x) = f (x) − x2 f(x). The map T(f)(x) = f2 (x) on C∞ (R). EIGENVALUES, BASIS, KERNEL, IMAGE. Many concepts work also here. X linear space f, g ∈ X, f + g ∈ X, λf ∈ X, 0 ∈ X. T linear transformation T(f + g) = T(f) + T(g), T(λf) = λT(f), T(0) = 0. f1, f2, ..., fn linear independent i cifi = 0 implies fi = 0. f1, f2, ..., fn span X Every f is of the form i cifi. f1, f2, ..., fn basis of X linear independent and span. T has eigenvalue λ Tf = λf kernel of T {Tf = 0} image of T {Tf |f ∈ X}. Some concepts do not work without modification. Example: det(T) or tr(T) are not always defined for linear transformations in infinite dimensions. The concept of a basis in infinite dimensions has to be defined properly. INNER PRODUCT. The analogue of the dot product i figi for vectors (f1, f2, ..., fn), (g1, g2, ..., gn) is the integral 1 0 f(x)g(x) dx for functions on the interval [0, 1]. One writes (f, g) and ||f|| = (f, f). This inner product is defined on L2 or C∞ ([0, 1]). Example: (x3 , x2 ) = 1 0 x5 dx = 1/5. Having an inner product allows to define ”length”, ”angle” in infinite dimensions. One can use it to define projections, Gram-Schmidt orthogonalization etc.
  • 63. DIFFERENTIAL EQUATIONS, Math 21b, O. Knill LINEAR DIFFERENTIAL EQUATIONS WITH CONSTANT COEFFICIENTS. Df = Tf = f is a linear map on the space of smooth functions C∞ . If p(x) = a0 + a1x + ... + anxn is a polynomial, then p(D) = a0 + a1D + ... + anDn is a linear map on C∞ (R) too. We will see here how to find the general solution of p(D)f = g. EXAMPLE. For p(x) = x2 − x + 6 and g(x) = cos(x) the problem p(D)f = g is the differential equation f (x) − f (x) − 6f(x) = cos(x). It has the solution c1e−2x + c2e3x − (sin(x) + 7 cos(x))/50, where c1, c2 are arbitrary constants. How do we find these solutions? THE IDEA. In general, a differential equation p(D)f = g has many solution. For example, for p(D) = D3 , the equation D3 f = 0 has solutions (c0 +c1x+c2x2 ). The constants come from integrating three times. Integrating means applying D−1 but since D has as a kernel the constant functions, integration gives a one dimensional space of anti-derivatives (we can add a constant to the result and still have an anti-derivative). In order to solve D3 f = g, we integrate g three times. We will generalize this idea by writing T = p(D) as a product of simpler transformations which we can invert. These simpler transformations have the form (D − λ)f = g. FINDING THE KERNEL OF A POLYNOMIAL IN D. How do we find a basis for the kernel of Tf = f +2f +f? The linear map T can be written as a polynomial in D which means T = D2 − D − 2 = (D + 1)(D − 2). The kernel of T contains the kernel of D − 2 which is one-dimensional and spanned by f1 = e2x . The kernel of T = (D − 2)(D + 1) also contains the kernel of D + 1 which is spanned by f2 = e−x . The kernel of T is therefore two dimensional and spanned by e2x and e−x . THEOREM: If T = p(D) = Dn + an−1Dn−1 + ... + a1D + a0 on C∞ then dim(ker(T)) = n. PROOF. T = p(D) = (D − λj), where λj are the roots of the polynomial p. The kernel of T contains the kernel of D − λj which is spanned by fj(t) = eλj t . In the case when we have a factor (D − λj)k of T, then we have to consider the kernel of (D − λj)k which is q(t)eλt , where q is a polynomial of degree k − 1. For example, the kernel of (D − 1)3 consists of all functions (a + bt + ct2 )et . SECOND PROOF. Write this as A˙g = 0, where A is a n × n matrix and g = f, ˙f, · · · , f(n−1) T , where f(k) = Dk f is the k’th derivative. The linear map T = AD acts on vectors of functions. If all eigenvalues λj of A are different (they are the same λj as before), then A can be diagonalized. Solving the diagonal case BD = 0 is easy. It has a n dimensional kernel of vectors F = [f1, . . . , fn]T , where fi(t) = t. If B = SAS−1 , and F is in the kernel of BD, then SF is in the kernel of AD. REMARK. The result can be generalized to the case, when aj are functions of x. Especially, Tf = g has a solution, when T is of the above form. It is important that the function in front of the highest power Dn is bounded away from 0 for all t. For example xDf(x) = ex has no solution in C∞ , because we can not integrate ex /x. An example of a ODE with variable coefficients is the Sturm-Liouville eigenvalue problem T(f)(x) = a(x)f (x) + a (x)f (x) + q(x)f(x) = λf(x) like for example the Legendre differential equation (1 − x2 )f (x) − 2xf (x) + n(n + 1)f(x) = 0. BACKUP • Equations Tf = 0, where T = p(D) form linear differential equations with constant coefficients for which we want to understand the solution space. Such equations are called homogeneous. Solving differential equations often means finding a basis of the kernel of T. In the above example, a general solution of f + 2f + f = 0 can be written as f(t) = a1f1(t) + a2f2(t). If we fix two values like f(0), f (0) or f(0), f(1), the solution is unique. • If we want to solve Tf = g, an inhomogeneous equation then T −1 is not unique because we have a kernel. If g is in the image of T there is at least one solution f. The general solution is then f + ker(T). For example, for T = D2 , which has C∞ as its image, we can find a solution to D2 f = t3 by integrating twice: f(t) = t5 /20. The kernel of T consists of all linear functions at+b. The general solution to D2 = t3 is at + b + t5 /20. The integration constants parameterize actually the kernel of a linear map.
  • 64. THE SYSTEM Tf = (D − λ)f = g has the general solution ceλx + eλx x 0 e−λt g(t) dt . The solution f = (D − λ)−1 g is the sum of a function in the kernel and a special function. THE SOLUTION OF (D − λ)k f = g is obtained by applying (D − λ)−1 several times on g. In particular, for g = 0, we get: the kernel of (D − λ)k as (c0 + c1x + ... + ck−1xk−1 )eλx . THEOREM. The inhomogeneous p(D)f = g has an n-dimensional space of solutions in C∞ (R). PROOF. To solve Tf = p(D)f = g, we write the equation as (D − λ1)k1 (D − λ2)k2 · · · (D − λn)kn f = g. Since we know how to invert each Tj = (D − λj)kj , we can construct the general solution by inverting one factor Tj of T one after another. Often we can find directly a special solution f1 of p(D)f = g and get the general solution as f1 + fh, where fh is in the n-dimensional kernel of T. EXAMPLE 1) Tf = e3x , where T = D2 − D = D(D − 1). We first solve (D − 1)f = e3x . It has the solution f1 = cex + ex x 0 e−t e3t dt = c2ex + e3x /2. Now solve Df = f1. It has the solution c1 + c2ex + e3x /6 . EXAMPLE 2) Tf = sin(x) with T = (D2 − 2D + 1) = (D − 1)2 . We see that cos(x)/2 is a special solution. The kernel of T = (D − 1)2 is spanned by xex and ex so that the general solution is (c1 + c2x)ex + cos(x)/2 . EXAMPLE 3) Tf = x with T = D2 +1 = (D−i)(D+i) has the special solution f(x) = x. The kernel is spanned by eix and e−ix or also by cos(x), sin(x). The general solution can be written as c1 cos(x) + c2 sin(x) + x . EXAMPLE 4) Tf = x with T = D4 + 2D2 + 1 = (D − i)2 (D + i)2 has the special solution f(x) = x. The kernel is spanned by eix , xeix , e−ix , x−ix or also by cos(x), sin(x), x cos(x), x sin(x). The general solution can be written as (c0 + c1x) cos(x) + (d0 + d1x) sin(x) + x . THESE EXAMPLES FORM 4 TYPICAL CASES. CASE 1) p(D) = (D − λ1)...(D − λn) with real λi. The general solution of p(D)f = g is the sum of a special solution and c1eλ1x + ... + cneλnx CASE 2) p(D) = (D − λ)k . The general solution is the sum of a special solution and a term (c0 + c1x + ... + ck−1xk−1 )eλx CASE 3) p(D) = (D − λ)(D − λ) with λ = a + ib. The general solution is a sum of a special solution and a term c1eax cos(bx) + c2eax sin(bx) CASE 4) p(D) = (D − λ)k (D − λ)k with λ = a + ib. The general solution is a sum of a special solution and (c0 + c1x + ... + ck−1xk−1 )eax cos(bx) + (d0 + d1x + ... + dk−1xk−1 )eax sin(bx) We know this also from the eigenvalue problem for a matrix. We either have distinct real eigenvalues, or we have some eigenvalues with multiplicity, or we have pairs of complex conjugate eigenvalues which are distinct, or we have pairs of complex conjugate eigenvalues with some multiplicity. CAS SOLUTION OF ODE’s: Example: DSolve[f [x] − f [x] == Exp[3x], f[x], x] REMARK. (informal) Operator methods are also useful for ODEs with variable coefficients. For ex- ample, T = H − 1 = D2 − x2 − 1, the quantum harmonic oscillator, can be written as T = A∗ A = AA∗ + 2 with a creation operator A∗ = (D − x) and annihilation operator A = (D + x). (Hint: use the commutation relation Dx − xD = 1.) The kernel f0 = Ce−x2 /2 of A = (D + x) is also the kernel of T and so an eigenvector of T and H. It is called the vacuum. If f is an eigenvector of H with Hf = λf, then A∗ f is an eigenvector with eigenvalue λ + 2 . Proof. Because HA∗ − A∗ H = [H, A∗ ] = 2A∗ , we have H(A∗ f) = A∗ Hf + [H, A∗ ]f = A∗ λf + 2A∗ f = (λ + 2)(A∗ f). We obtain all eigenvectors fn = A∗ fn−1 of eigenvalue 1 + 2n by applying iteratively the creation operator A∗ on the vacuum f0. Because every function f with f2 dx < ∞ can be written uniquely as f = ∞ n=0 anfn, we can diagonalize H and solve Hf = g with f = n bn/(1 + 2n)fn, where g = n bnfn.
  • 65. FOURIER SERIES, Math 21b, O. Knill HOW DOES IT WORK? Piecewise smooth functions f(x) on [−π, π] form a linear space. The numbers ck = 1 2π π −π f(x)e−ikx dx are called the Fourier coefficients of f. We can recover f from these coefficients by f(x) = k ckeikx . When we write f = feven + fodd, where feven(x) = (f(x) + f(−x))/2 and fodd(x) = ((f(x) − f(−x))/2, then feven(x) = a0/ √ 2 + k=1 ak cos(kx) and fodd(x) = ∞ k=1 bk sin(kx). The sum of the cos-series for feven and the sin-series for fodd up gives the real Fourier series of f. Those coefficients are ak = 1 π π −π f(x) cos(kx) dx, a0 = 1 π π −π f(x)/ √ 2 dx, bk = 1 π π −π f(x) sin(kx) dx. While the complex expansion is more elegant, the real expansion has the advantage that one does not leave the real numbers. ck = 1 2π π −π f(x)e−ikx dx. f(x) = ∞ k=−∞ ckeikx . SUMMARY. a0 = 1 π π −π f(x)/ √ 2 dx ak = 1 π π −π f(x) cos(kx) dx bk = 1 π π −π f(x) sin(kx) dx f(x) = a0√ 2 + ∞ k=1 ak cos(kx) + ∞ k=1 bk sin(kx) EXAMPLE 1. Let f(x) = x on [−π, π]. This is an odd function (f(−x)+f(x) = 0) so that it has a sin series: with bk = 1 π π −π x sin(kx) dx = 1 π (x cos(kx)/k + sin(kx)/k2 |π −π) = 2(−1)k+1 /k we get x = ∞ k=1 2(−1)k+1 k sin(kx). For example, π/2 = 2(1 − 1/3 + 1/5 − 1/7...) a formula of Leibnitz. EXAMPLE 2. Let f(x) = cos(x)+1/7 cos(5x). This trigonometric polynomial is already the Fourier series. The nonzero coefficients are a1 = 1, a5 = 1/7. EXAMPLE 3. Let f(x) = |x| on [−π, π]. This is an even function f(−x) − f(x) = 0 so that it has a cos series: with a0 = 1/(2 √ 2), ak = 1 π π −π |x| cos(kx) dx = 2 π π 0 x cos(kx) dx = 2 π (+x sin(kx)/k + cos(kx)/k2 |π 0 ) = 2 π ((−1)k − 1)/k2 for k > 0. WHERE ARE FOURIER SERIES USEFUL? Examples: • Partial differential equations. PDE’s like ¨u = c2 u become ODE’s: ¨uk = c2 k2 uk can be solved as uk(t) = ak sin(ckt)uk(0) and lead to solutions u(t, x) = k ak sin(ckt) sin(kx) of the PDE. • Sound Coefficients ak form the frequency spec- trum of a sound f. Filters suppress frequencies, equalizers transform the Fourier space, compres- sors (i.e.MP3) select frequencies relevant to the ear. • Analysis: k ak sin(kx) = f(x) give explicit expres- sions for sums which would be hard to evaluate other- wise. The Leibnitz sum π/4 = 1−1/3+1/5−1/7+... (Example 1 above) is an example. • Number theory: Example: if α is irrational, then the fact that nα (mod1) are uniformly distributed in [0, 1] can be understood with Fourier theory. • Chaos theory: Quite many notions in Chaos the- ory can be defined or analyzed using Fourier theory. Examples are mixing properties or ergodicity. • Quantum dynamics: Transport properties of mate- rials are related to spectral questions for their Hamil- tonians. The relation is given by Fourier theory. • Crystallography: X ray Diffraction patterns of a crystal, analyzed using Fourier theory reveal the structure of the crystal. • Probability theory: The Fourier transform χX = E[eiX ] of a random variable is called characteristic function. Independent case: χx+y = χxχy. • Image formats: like JPG compress by cutting irrel- evant parts in Fourier space. WHY DOES IT WORK? One has a dot product on functions with f · g = 1 2π π −π f(x)g(x) dx. The functions ek = eikx serve as basis vectors. As in Euclidean space, where v · ek = vk is the k’th coordinate of v, the Fourier coefficient ck = ek · f = 1 2π π −π e−ikx f(x) dx is the k-th coordinate of f. In the same way as we wrote v = k vkek in Euclidean space, we have f(x) = k ckek(x). The vectors ek are orthonormal: 1 2π π −π ek(x)em(x) dx = 1 2π π −π e(m−k)ix dx which is zero for m = k and 1 for m = k. Also the functions 1/ √ 2, sin(nx), cos(nx) form an orthonormal basis with respect to the dot product (f, g) = 1 π π −π f(x)g(x) dx. The Fourier coefficients a0, an, bn are the coordinates of f in that basis. Fourier basis is not the only one. An other basis which has many applications are Wavelet expansions. The principle is the same.
  • 66. LENGTH, DISTANCE AND ANGLE. With a dot product, one has a length ||f||2 = f · f = 1 2π π −π |f(x)|2 dx and can measure distances between two functions as ||f − g|| and angles between functions f, g by cos(α) = f·g/(||f||||g||). Functions are called orthogonal if (f, g) = f·g = 0. For example, an even function f and an odd function g are orthogonal. The functions einx form an orthonormal family. The vectors √ 2 cos(kx), √ 2 sin(kx) form an orthonormal family, cos(kx) is in the linear space of even functions and sin(kx) in the linear space of odd functions. If we work with sin or cos Fourier series, we use the dot product f · g = 1 π π −π f(x)g(x) dx for which the functions 1/ √ 2, cos(kx), sin(kx) are orthonormal. REWRITING THE DOT PRODUCT. If f(x) = k ckeikx and g(x) = k dkeikx , then 1 2π π −π f(x)g(x) dx = 1 2π k,m cke−ikx dmeimx = k ckdk. The dot product is the sum of the product of the coordinates as in finite dimensions. If f and g are even, then f = f0/ √ 2 + ∞ k=1 fk cos(kx), g = g0/ √ 2 + k gk cos(kx) and 1 π π π f(x)g(x) dx = ∞ k=0 fkgk. If f and g are odd, and f = k fk sin(kx), g = ∞ k=1 gk sin(kx) then (f, g) = f · g = ∞ k=1 fkgk. Especially the Perseval equality 1 π π −π |f(x)|2 = ∞ k=1 f2 k holds. It can be useful also to find closed formulas for the sum of some series. EXAMPLE. f(x) = x = 2(sin(x) − sin(2x)/2 + sin(3x)/3 − sin(4x)/4 + ... has coefficients fk = 2(−1)k+1 /k and so 4(1 + 1/4 + 1/9 + ...) = 1 π π −π x2 dx = 2π2 /3 or 1 + 1/4 + 1/9 + 1/16 + 1/25 + ....) = π2 /6 . APPROXIMATIONS. -3 -2 -1 1 2 3 -0.2 0.2 0.4 0.6 0.8 1 1.2 If f(x) = k bk cos(kx), then fn(x) = n k=1 bk cos(kx) is an approximation to f. Be- cause ||f − fk||2 = ∞ k=n+1 b2 k goes to zero, the graphs of the functions fn come for large n close to the graph of the function f. The picture to the left shows an approximation of a piecewise continuous even function, the right hand side the values of the coefficients ak = 1 π π −π f(x) cos(kx) dx. 2 4 6 8 10 12 -0.1 0.1 0.2 0.3 0.4 SOME HISTORY. The Greeks approximation of planetary motion through epicycles was an early use of Fourier theory: z(t) = eit is a circle (Aristarchus system), z(t) = eit + eint is an epicycle (Ptolemaeus system), 18’th century Mathematicians like Euler, Lagrange, Bernoulli knew experimentally that Fourier series worked. Fourier’s (picture left) claim of the convergence of the series was con- firmed in the 19’th century by Cauchy and Dirichlet. For continuous functions the sum does not need to converge everywhere. However, as the 19 year old Fej´er (picture right) demonstrated in his theses in 1900, the coefficients still determine the function n−1 k=−(n−1) n−|k| n fkeikx → f(x) for n → ∞ if f is continuous and f(−π) = f(π). Partial differ- ential equations, i.e. the theory of heat had sparked early research in Fourier theory. OTHER FOURIER TRANSFORMS. On a finite interval one obtains a series, on the line an integral, on finite sets, finite sums. The discrete Fourier transformation (DFT) is important for applications. It can be determined efficiently by the (FFT=Fast Fourier transform) found in 1965, reducing the n2 steps to n log(n). Domain Name Synthesis Coefficients T = [−π, π) Fourier series f(x) = k ˆfkeikx ˆfk = 1 2π π −π f(x)e−ikx dx. R = (−∞, ∞)) Fourier transforms f(x) = ∞ ∞ ˆf(k)eikx dx ˆf(k) = 1 2π ∞ ∞ ˆf(k)e−ikx dx Zn = {1, .., n} DFT fm = n k=1 ˆfkeimk2π/n ˆfk = 1 n n m=1 fme−ikm2π/n All these transformations can be defined in dimension d. Then k = (k1, ..., kd) etc. are vectors. 2D DFT is for example useful in image manipulation. COMPUTER ALGEBRA. Packages like Mathematica have the discrete Fourier transform built in Fourier[{0.3, 0.4, 0.5}] for example, gives the DFT of a three vector. You can perform a simple Fourier analysis yourself by listening to a sound like Play[Sin[2000 ∗ x ∗ Floor[7 ∗ x]/12], {x, 0, 20}] ...
  • 67. HEAT AND WAVE EQUATION Math 21b, O. Knill NOTATION. We write Tx(x, t) and Tt(x, t) for the partial derivatives with respect to x or t. THE HEAT EQUATION. The temperature distribution T(x, t) in a metal bar [0, T] satisfies the heat equation Tt(x, t) = µTxx(x, t). At every point, the rate of change of the temperature is proportional to the second derivative of x . . . T(x, t) at x. The function T(t, x) is zero at both ends of the bar and f(x) = T(0, x) is a given initial temperature distribution. SOLVING IT. This partial differential equation (PDE) is solved by writing T(x, t) = u(x)v(t) which leads to ˙v(t)u(x) = v(t)u (x) or ˙v(t)/(µv(t)) = u (x)/u(x). Because the LHS does not depend on x and the RHS not on t, this must be a constant λ. The ODE u (x) = λu(x) is an eigenvalue problem D2 u = λu which has only solutions when λ = −k2 for integers k. The eigenfunctions of D2 are uk(x) = sin(kx). The second equation ˙v(t) = k2 µv is solved by v(t) = e− k2 µt so that u(x, t) = sin(kx)e−k2 t is a solution of the heat equation. Because linear combinations of solutions are solutions too, a general solution is of the form T(x, t) = ∞ k=1 ak sin(kx)e−k2 t . For fixed t this is a Fourier series. The coefficients ak are determined from f(x) = T(x, 0) = ∞ k=1 ak sin(kx), where ak = 2 π π 0 f(x) sin(kx). (The factor 2 comes from the fact that f should be thought to be extended to an odd function on [−π, π] and the integral from [−π, 0] is the same as the from [0, π].) FOURIER: (Solving the heat equation was the birth of Fourier theory) The heat equation Tt(t, x) = µTxx(t, x) with smooth T(x, 0) = f(x), T(0, 0) = T(π, 0) = 0 has the solution ∞ k=1 ak sin(kx)e−k2 µt with ak = 2 π π 0 f(x) sin(kx) dx. Remark. If T(0, 0) = a, T(π, 0) = b instead, then the solution is T(x, t) = T0(x, t)+a+bx/π, where T0(x, t) is the solution in the box using f0(x) = f(x)−(a+bx/π) which satisfies f(0) = f(π) = 0. (Because T(t, x) = a+bx/π is a solution of the heat equation, we can add it to any solution and get new solutions. This allows to tune the boundary conditions). EXAMPLE. f(x) = x on [−π/2, π/2 π-periodically continued has the Fourier coefficients ak = 2 π π/2 −π/2 x sin(kx) dx = 4/(k2 π)(−1)(k−1)/2 for odd k and 0 else. The solution is T(x, t) = ∞ m=0 4(−1)m (2m + 1)2π e−µ(2m+1)2 t sin((2m + 1)x) . The exponential term containing the time makes the function T(x, t) converge to 0: ”The bar cools.” The higher frequency terms are damped faster because ”Smaller disturbances are smoothed out faster.” VISUALIZATION. We can just plot the graph of the function T(x, t) or plot the temperature distribution for different times ti. 0 0.2 0.4 0.6 0.8 1 0 2 4 6 0 0.5 1 1.5 0 0.2 0.4 0.6 0.8 1
  • 68. 2000 4000 6000 8000 10000 -50 -25 25 50 75 100 DERIVATION OF THE HEAT EQUATION. The temperature T measures the kinetic energy distribution of atoms, and electrons in the bar. Each particle makes a random walk. We can model that as follows: assume we have n adjacent cells containing particles and in each time step, a fraction of the particles moves randomly either to the right or to the left. If Ti(t) is the energy of particles in cell i at time t, then the energy of particles at time t+1 is ρ 2 (Ti−1(t)−2Ti(t)+Ti+1)(t)). This is a discrete version of the second derivative T (x, t) ∼ (T(x + dx, t) − 2T(x, t) + T(x + dx, t))/(dx2 ). FOURIER TRANSFORM=DIAGONALISATION OF D. Fourier theory actually diagonalizes the linear map D: write f = n ˆfneinx = S−1 f and Sf = ˆf = (. . . , c−1, c0, c1, . . .). SDS−1 ˆf = SD n ˆfneinx = S n n ˆfneinx == (. . . , i(−1)c−1, 0c0, ic1, . . .). This means that differentiation changes the Fourier coefficients as cn → incn. From this we see that the diagonalisation of D2 is multiplying with −n2 . Now, if ˙u = D2 u, then ˙ˆun = −µn2 ˆun so that ˆun(t) = e−n2 t ˆu)n(0). The partial differential equation is transformed into a sequence of ordinary differential equations. Differentiation of the function f S ← ˆf D ↓ ↓ M Df S−1 → M ˆf becomes multiplica- tion with the diago- nal matrix M =           · · · · · · · · 2i 0 0 0 0 · · 0 i 0 0 0 · · 0 0 0 0 0 · · 0 0 0 i 0 · · 0 0 0 0 2i · · · · · · · ·           WAVE EQUATION. The wave equation Ttt(x, t) = c2 Txx(x, t) is solved in a similar way as the heat equation: writing T(x, t) = u(x)v(t) gives ¨vu = c2 vu or ¨v/(c2 v) = u /u. Because the left hand is independent of x and the right hand side is independent of t, we have ¨v/(c2 v) = u /u = −k2 = const. The right equation has solutions uk(x) = sin(kx). Now, vk(t) = ak cos(kct) + bk sin(kct) solves ¨v = −c2 k2 v so that Tk(x, t) = uk(x)vk(t) = sin(kx)(ak cos(kct) + bk sin(kct)) is a solution of the wave equation. General solutions can be obtained by taking superpositions of these waves T(x, t) = ∞ k=1 ak(sin(kx) cos(kct) + bk sin(kx) sin(kct). The coefficients ak, bk are obtained from T(x, 0) = ∞ k=1 ak sin(kx) and ˙T(x, 0) = ∞ k=1 kcbk sin(kx). Unlike in the heat equation both T(x, 0) and ˙T(x, 0) have to be specified. WAVE EQUATION: The wave equation Ttt = c2 Txx with smooth T(x, 0) = f(x), ˙T(x, 0) = g(x), T(0, 0) = T(π, 0) = 0 has the solution ∞ k=1 ak sin(kx) cos(kct) + bk sin(kx) sin(kct) with ak = 2 π π 0 f(x) sin(kx) dx and kcbk = 2 π π 0 g(x) sin(kx) dx. QUANTUM MECHANICS = MA- TRIX MECHANICS A particle in a box is described by a linear map L = −D2 ¯h2 /(2m) acting on func- tions on [0, π]. In the Fourier pic- ture, this linear map is M = ¯h2 2m           · · · · · · · · −4 0 0 0 0 · · 0 −1 0 0 0 · · 0 0 0 0 0 · · 0 0 0 −1 0 · · 0 0 0 0 −4 · · · · · · · ·           The diagonal entries are the eigenvalues of L as usual after diagonalisa- tion. Quantum mech. observables are symmetric linear maps: for example: position, X : f → xf, momentum P : f → i¯h f, kinetic energy f → P 2 /2m = −¯h2 ∆/(2m), Coulomb potential energy f → (4π 0)−1 e2 /rf. HIGHER DIMENSIONS. The heat or wave equations can be solved also in higher dimensions using Fourier theory. For example, on a matellic plate [−π, π] × [−π, π] the temperature distribution T(x, y, t) satisfies Tt = µ(Txx + Tyy). The Fourier series of a function f(x, y) in two variables is n,m cn,meinx+my , where cn,m = (2π)−2 π −π π −π f(x, y)e−inx e−imy dx dy. The Gaussian blur filter applied to each color coordinate of a picture acts very similar than the heat flow.
  • 69. LAPLACE EQUATION (informal) Math 21b, O. Knill LAPLACE EQUATION. The linear map T → ∆T = Txx + Tyy for smooth functions in the plane is called the Laplacian. The PDE ∆T = 0 is called the Laplace equation. A solution T is in the kernel of ∆ and called harmonic. For example T(x, y) = x2 − y2 is harmonic. One can show that T(x, y) = Re((x + iy)n or T(x, y) = Im((x + iy)n ) are all harmonic. DIRICHLET PROBLEM: find a function T(x, y) on a region Ω which satisfies ∆T(x, y) = 0 inside Ω and which is prescribed function f(x, y) at the boundary. This boundary value problem can be solved explicitly in many cases. IN A DISC. {x2 + y2 ≤ 1}. If f(t) = T(cos(t), sin(t)) is prescribed on the boundary and T satisfies ∆T = 0 inside the disc, then T can be found via Fourier theory: If f(t) = a0 + ∞ n=1 an cos(nt) + bn sin(nt), then T(x, y) = a0 + n anRe((x + iy)n ) + bnIm((x+iy)n ) satisfies the Laplace equation and T(cos(t), sin(t)) = f(t) on the boundary of the disc. PROOF. If f = a0 is constant on the boundary then T(x, y) = a. If f(t) = sin(nt) on the boundary z = eit = x+iy then T(x, y) = Im(zn ) and if f(t) = cos(nt) on the boundary then T(x, y) = Re(zn ) is a solution. EXAMPLE. Re(z2 ) = x2 − y2 + i(2xy) gives the two harmonic functions f(x, y) = x2 − y2 and g(x, y) = 2xy: they satisfy (∂2 x + ∂2 y)f = 0. ESPECIALLY: the mean value property for harmonic functions T(0, 0) = 1 2 2π 0 f(t) dt holds. POISSON FORMULA: With z = x + iy = reiθ and T(x, y) = f(z), the general solution is given also by the Poisson integral formula: f(reiθ ) = 1 2π 2π 0 1 − r2 r2 − 2r cos(θ − φ) + 1 f(eiφ ) dφ . PHYSICAL RELEVANCE. If the temperature or charge distribution on the boundary of a region Ω, then T(x, y) is the stable temperature or charge distribution inside that region. In the later case, the electric field inside Ω is then U. An other application is that the velocity v of an ideal incompressible fluid satisfies v = U, where U satisfies the Laplace differential equation ∆U = 0 in the region. SQUARE, SPECIAL CASE. We solve first the case, when T vanishes on three sides x = 0, x = π, y = 0 and T(x, π) = f(x). Separation of variables gives then T(x, y) = n an sin(nx) sinh(ny), where the coefficients an are obtained from T(x, π) = n bn sin(nx) = n an sin(nx) sinh(nπ), where bn = 2 π π 0 f(x) sin(nx) dx are the Fourier coefficients of f. The solution is therefore Tf (x, y) = ∞ n=1 bn sin(nx) sinh(ny)/ sinh(nπ) SQUARE GENERAL CASE Solutions to general boundary conditions can be obtained by adding the solutions in the following cases: Tf (x, π − y) solves the case, when T vanishes on the sides x = 0, x = π, y = π and is f on y = 0. The function Tf (y, x) solves the case, when T vanishes on the sides y = 0, y = π, x = 0 and Tf (π − y, x) solves the case, when T vanishes on y = 0, y = π and x = π. The general solution is T(x, y) = Tf (x, y) + Tg(x, π − y) +Th(y, x) + Tk(π − y, x)
  • 70. A GENERAL REGION. For a general region, one uses numerical methods. One pos- sibility is by conformal transportation: to map the region into the the disc using a complex map F, which maps the region Ω into the disc. Solve then the problem in the disc with boundary value T(F −1 (x, y)) on the disc. If S(x, y) is the solution there, then T(x, y) = S(F −1 (x, y)) is the solution in the region Ω. The picture shows the example of the Joukowski map z → (z + 1/z)/2 which has been used in fluid dynamics (N.J. Joukowski (1847-1921) was a Russian aerodynamics researcher.) A tourists view on related PDE topics POISSON EQUATION. The Poisson equation ∆f = g in a square [0, π]2 can be solved via Fourier theory: If f(x, y) = k,m an,m cos(nx) cos(my) and g(x, y) = k,m bn,m cos(nx) cos(my), then fxx + fyy = n,m −(n2 + m2 )an,m cos(nx) cos(my) = k,m bn,m cos(nx) cos(my), then an,m = −bn,m/(n2 − m2 ). The poisson equation is important in electrostatics, for example to determine the electromagnetic field when the charge and current distribution is known: ∆U(x) = −(1/ 0)ρ, then E = U is the electric field to the charge distribution ρ(x). EIGENVALUES. The functions sin(nx) sin(ny) are eigenfunctions for the Laplace operator fxx + fyy = n2 f on the disc. For a general bounded region Ω, we can look at all smooth functions which are zero on the boundary of Ω. The possi- ble eigenvalues of ∆f = λf are the possible ener- gies of a particle in Ω. COMPLEX DYNAMICS. Also in complex dy- namics, harmonic functions appear. Finding properties of complicated sets like the Mandel- brot set is done by mapping the exterior to the outside of the unit circle. If the Mandelbrot set is charged then the contour lines of equal poten- tial can be obtained as the corresponding contour lines in the disc case (where the lines are circles). SCHR¨ODINGER EQUATION. If H is the energy operator, then i¯h ˙f = Hf is called the Schr¨odinger equa- tion. If H = −¯h2 ∆/(2m), this looks very much like the heat equation, if there were not the i. If f is an eigenvalue of H, then i¯h ˙f = λf and f(t) = ei¯ht f(0). In the heat equation, we would get f(t) = e−µ(t) f(0). The evolution of the Schr¨odinger equation is very smilar to the wave equation. QUANTUM CHAOS studies the eigenvalues and eigenvectors of the Laplacian in a bounded region. If the billiard in the region is chaotic, the study of the eigensystem is called quantum chaos. The eigenvalue problem ∆f = λf and the billiard problem in the region are related. A famous open problem is whether two smooth convex regions in the plane for which the eigenvalues λj are the same are equivalent up to rotation and translation. In a square of size L we know all the eigenvalues π2 L2 (n2 + m2 ). The eigenvectors are sin(pi L nx) sin(pi L mx). We have explicit formulas for the eigenfunctions and eigenvalues. On the classical level, for the billiard, we have also a complete picture, how a billiard path will look like. The square is an example, where we have no quantum chaos.