Linear Algebra

Tensors

Linear algebra is the most basic and fundamental part of machine learning. Any data can be represented using tensors e.g. vectors and matrices.
Don’t worry if the below diagram doesn’t make sense now, I will demonstrate how actual football related data can fit into these tensors.

Tensor Transposition

Tensor transposition is needed in machine learning because it allows us to change the order of the dimensions of a tensor for example from a
3x2 matrix to a 2x3 matrix. This is necessary in various tasks such as matrix multiplication which is used heavily in machine learning.

Example

Suppose we have a matrix A that represents the number of goals scored by each player, with dimensions (matches, goals). We can transpose
this matrix to obtain a new matrix AT with dimensions (goals, matches).

Player Apps Goals

1 Luis Díaz26, AM(L) 7 3

2 Mohamed Salah30, AM(CLR),FW 27 12

3 Thiago31, M(C) 14 -

4 Alisson30, GK 28 -

5 Darwin Núñez23, AM(CL),FW 17 8

6 Virgil van Dijk31, D(C) 23 3

7 Trent Alexander-Arnold24, D(R),M(R) 24 1

8 Roberto Firmino31, M(CLR),FW 12 8

9 Cody Gakpo23, AM(CL),FW 9 4

 10 Andrew Robertson29, D(L),M(L) 21 -

11 Joe Gomez25, D(CLR) 14 -

12 Ibrahima Konaté23, D(C) 9 -

13 Fabinho29, D(CR),DMC 21(5) -

14 Harvey Elliott20, AM(CR) 17(10) 1

15 Stefan Bajcetic18, M(C) 6(5) 1

16 Konstantinos Tsimikas26, D(L) 7(9) -

17 Diogo Jota26, AM(CLR),FW 6(6) -

18 Jordan Henderson32, D(C),M(CLR) 16(9) -

19 James Milner37, D(LR),M(CLR) 6(16) -

20 Nathaniel Phillips26, D(C) 1(1) -

21 Fábio Carvalho20, AM(CL) 4(8) 2

22 Joël Matip31, D(C) 10(2) -

23 Alex Oxlade Chamberlain29, M(CLR) 4(5) 1

24 Curtis Jones22, AM(CL) 2(6) -

25 Naby Keïta28, M(C) 3(5) -

26 Ben Doak17, Forward 0(2) -

27 Bobby Clark18, Midfielder 0(1) -

When we convert the above table to a matrix containing just the apps and goals data. This gives us a 27x2 dimension matrix:
The transpose will look like this 2x27 matrix:

Here’s how to do it in Python:

 import numpy as np

A = np.array([[7, 3],
[27, 12],
[14, 0],
[28, 0],
[17, 8],
[23, 3],
[24, 1],
[12, 8],
[9, 4],
[21, 0],
[14, 0],
[9, 0],
[21, 0],
[17, 1],
[6, 1],
[7, 0],
[16, 0],
[6, 0],
[1, 0],
[4, 2],
[10, 0],
[4, 1],
[2, 0],
[3, 0],
[0, 0],
[0, 0]])

A_T = np.transpose(A)

Basic Tensor Arithmetic

Basic tensor arithmetic operations, such as addition and multiplication, are needed in machine learning to combine tensors, perform element-
wise operations, and compute the weighted sum of tensors. These operations are fundamental to many machine learning algorithms, such as
neural networks, where we need to combine inputs and weights to compute activations.

Example

Suppose we have two matrices A and B that represent the number of goals and assists made by each player in a football team, respectively, with
dimensions (players, matches). We can combine these matrices using element-wise addition to obtain a new matrix C with dimensions (players,
matches), where each element represents the total number of points earned by each player.

Let’s say we want to calculate overall goal contributions of Liverpool players. Pretend goals and assists were two separate vectors:

Player Goals

1 Mohamed Salah30, AM(CLR),FW 12

2 Darwin Núñez23, AM(CL),FW 8

3 Roberto Firmino31, M(CLR),FW 8

4 Cody Gakpo23, AM(CL),FW 4

5 Luis Díaz26, AM(L) 3

6 Virgil van Dijk31, D(C) 3

7 Fábio Carvalho20, AM(CL) 2

8 Alex Oxlade Chamberlain29, M(CLR) 1

9 Harvey Elliott20, AM(CR) 1

10 Stefan Bajcetic18, M(C) 1

11 Trent Alexander-Arnold24, D(R),M(R) 1

12 Alisson30, GK -

13 Andrew Robertson29, D(L),M(L) -

14 Ben Doak17, Forward -

15 Bobby Clark18, Midfielder -

16 Curtis Jones22, AM(CL) -

17 Diogo Jota26, AM(CLR),FW -

18 Fabinho29, D(CR),DMC -

19 Ibrahima Konaté23, D(C) -

20 James Milner37, D(LR),M(CLR) -

21 Joe Gomez25, D(CLR) -

22 Joël Matip31, D(C) -

23 Jordan Henderson32, D(C),M(CLR) -

24 Konstantinos Tsimikas26, D(L) -

25 Naby Keïta28, M(C) -

26 Nathaniel Phillips26, D(C) -

27 Thiago31, M(C) -

Player Assists

1 Mohamed Salah30, AM(CLR),FW 7

2 Darwin Núñez23, AM(CL),FW 3

3 Roberto Firmino31, M(CLR),FW 4

4 Cody Gakpo23, AM(CL),FW -

5 Luis Díaz26, AM(L) 2

6 Virgil van Dijk31, D(C) -

7 Fábio Carvalho20, AM(CL) -

8 Alex Oxlade Chamberlain29, M(CLR) -

9 Harvey Elliott20, AM(CR) 2

10 Stefan Bajcetic18, M(C) -

11 Trent Alexander-Arnold24, D(R),M(R) 2

12 Alisson30, GK 1

13 Andrew Robertson29, D(L),M(L) 6

 14 Ben Doak17, Forward -

15 Bobby Clark18, Midfielder -

16 Curtis Jones22, AM(CL) -

17 Diogo Jota26, AM(CLR),FW 3

18 Fabinho29, D(CR),DMC -

19 Ibrahima Konaté23, D(C) -

20 James Milner37, D(LR),M(CLR) 1

21 Joe Gomez25, D(CLR) -

22 Joël Matip31, D(C) -

23 Jordan Henderson32, D(C),M(CLR) 1

24 Konstantinos Tsimikas26, D(L) 4

25 Naby Keïta28, M(C) -

26 Nathaniel Phillips26, D(C) -

27 Thiago31, M(C) -

A = Goals, B = Assists

A = np.array([[12,8,8,4,3,3,2,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]])

B = np.array([[7,3,4,0,2,0,0,0,2,0,2,1,6,0,0,0,3,0,0,1,0,0,1,4,0,0,0]])

A_B = A + B
A_B

If two tensors have the same size, operations are often by default applied element-wise. This is not matrix multiplication, but is rather called the
Hadamard product or simply the element-wise product.

This can also be applied to other arithmetic such as subtraction, multiplication & division.

Reduction

Reduction operations are needed in machine learning to aggregate the values of a tensor along one or more dimensions. This is useful in various
applications, such as computing the loss function of a neural network, where we need to sum the errors over all samples in a batch.

Example

Suppose we have a matrix A that represents the number of successful passes made by each player in each match, with dimensions (players,
matches). We can compute the total number of successful passes made by the team in each match by summing the elements of each column of
the matrix A.
 import torch

// A_B = [19, 11, 12, 4, 5, 3, 2, 1, 3, 1, 3, 1, 6, 0, 0,

0,
3, 0, 0, 1, 0, 0, 1, 4, 0, 0, 0]

new_tensor = torch.tensor(A_B)

torch.sum(new_tensor)

Output: tensor(80)

The same rules apply to a matrix:

new_2d_tensor = torch.tensor([[1,2,3],[4,5,6]])
new_2d_tensor

Output: tensor([5, 7, 9])

The Dot Product

The dot product is needed in machine learning for various tasks, such as computing the similarity between two vectors, projecting a vector onto
another vector, and performing matrix multiplication. The dot product is essential for computing the weighted sum of inputs to a neuron in a
neural network.

A vector is basically an arrow in space pointing towards some direction. Looking at the below image the dot product of vector A (green) and
vector B (Blue) can be thought of as the projection (shadow) line i.e. we are looking at the “shadow” of one vector projected onto another. The
closer the two vectors are in size and direction the bigger the projection line is. The further they are the smaller the projection line is.

In the case of orthogonal (perpendicular) vectors the dot product will be 0 as they are in going in different directions (90 degree angle), for
example:
 y = [-2, 3]
x = [3, 2]

dot_product = y[0] * x[0] + y[1] * x[1]

print("The dot product of y and x is:", dot_product)

Output: The dot product of y and x is: 0

Meaning the vector y will have no shadow over vector x.

Example

Suppose we have two vectors u and v that represent the playing style of two football players, with components representing metrics such as
dribbling ability, shooting accuracy, and passing accuracy. We can compute the dot product of these vectors to obtain a measure of how similar
their playing styles are.

For example, the matrix for Messi could be:

Dribbling ability Shooting accuracy Passing accuracy

95 90 85

And the matrix for Ronaldo could be:

Dribbling ability Shooting accuracy Passing accuracy

90 92 80

To determine how similar the skills of Messi and Ronaldo are, we can calculate the dot product of their matrices. We do this by multiplying the
corresponding entries of the matrices and then summing the results.

The dot product of the matrices for Messi and Ronaldo is:

(95 * 90) + (90 * 92) + (85 * 80) = 16175

This means that the skills of Messi and Ronaldo are somewhat similar, with a dot product of 16175. However, we can also see that they have
some differences in their skills, as they are not identical matrices.

Now, let's say we want to compare Messi to another football player, Neymar. Neymar's matrix might look like this:

Dribbling ability Shooting accuracy Passing accuracy

 90 87 90

The dot product of Messi's and Neymar's matrices is:

(95 * 90) + (90 * 87) + (85 * 90) = 16215

This shows that the skills of Messi and Neymar are quite similar, with a dot product of 16215. In this case, we can see that the vectors in their
matrices are more aligned than those of Messi and Ronaldo.

In summary, the dot product can be used to measure the similarity of different football players based on their skills in certain areas. By
representing the skills of each player as a vector in a matrix, we can compare the direction of the vectors and compute their dot product to
determine how similar their skills are.

The Frobenius Norm

The Frobenius norm is needed in machine learning to measure the magnitude or “size” of a tensor or matrix. This is essential in various tasks
such as regularisation, where we need to add a penalty term to the loss function to prevent overfitting.

Example

Suppose we have a matrix A that represents the performance of a football team based on various metrics such as goals scored, assists made,
and shots taken, with dimensions (matches, metrics). We can compute the Frobenius norm of this matrix by summing the squared values of each
element in the matrix.

Goals_Scored Assists_Made Shots_Taken

Match_1 2 1 5

Match_2 1 2 4

Match_3 3 4 7

Match_4 1 0 2

When converted to a matrix:

To compute the Frobenius norm of this matrix, we need to square each element in the matrix, sum them up, and take the square root of the
result. We can do this using numpy's norm function with the ord='fro' option:

fro_norm = np.linalg.norm(A, ord='fro')

Output: fro_norm = 10.723805294763608

The Frobenius norm of a matrix is a measure of its magnitude or "size" that takes into account all the individual entries in the matrix. Specifically,
it is the square root of the sum of the squares of all the entries in the matrix.

In the case of a football team's performance matrix, the entries might represent various metrics such as goals scored, assists made, and shots
taken. The Frobenius norm of this matrix reflects the overall magnitude of the team's performance across all of these metrics. A higher Frobenius
norm indicates that the team performed better overall across all metrics, while a lower Frobenius norm indicates poorer performance.
So, if we have two football teams and their performance matrices A and B, with Frobenius norms of fro_norm_A and fro_norm_B,
respectively, then we can say that:

If fro_norm_A > fro_norm_B, then team A performed better overall across all metrics than team B.
If fro_norm_A < fro_norm_B, then team B performed better overall across all metrics than team A.
If fro_norm_A = fro_norm_B, then the two teams had similar overall performances across all metrics.

Matrix Multiplication

Matrix multiplication is necessary in machine learning for various tasks such as neural networks, linear regression, and PCA. It allows us to
transform data into a different space and compute the weighted sum of inputs to a neuron.

Example

The columns of tensor A need to match the rows of tensor B. In the example below, we're multiplying a 4x3 matrix A with a 3x1 vector x
to obtain a 4x1 vector that represents the weighted sum of the team's performance in each metric.

Matrix A with a vector:

Suppose we have a matrix A that represents the performance of a football team based on various metrics such as goals scored, assists made,
and shots taken, with dimensions (matches, metrics), and a vector x that represents the weight of each metric in predicting the team's
performance, with dimensions (metrics, 1). We can compute the product of A and x to obtain a new vector y with dimensions (matches, 1), where
each element represents the weighted sum of the corresponding metrics for each match.

A = [[2, 1, 4],
[3, 2, 1],
[1, 0, 2],
[2, 1, 3]]

x = [[0.5],
[0.3],
[0.2]]

Output:

Ax = [[2*0.5 + 1*0.3 + 4*0.2],

[3*0.5 + 2*0.3 + 1*0.2],
[1*0.5 + 0*0.3 + 2*0.2],
[2*0.5 + 1*0.3 + 3*0.2]]

= [[2.1],
[2.1],
[0.9],
[1.9]]

Matrix A with matrix B:

Suppose we have two matrices A and B that represent the performance of two football teams based on various metrics such as goals scored,
assists made, key passes, and shots taken, with dimensions (matches, metrics). We can use matrix multiplication to compute the dot product of
the team's performance vectors for each match. For example, suppose we have the following matrices A and B:
 A = [[2, 1, 5, 1],
[1, 2, 4, 2],
[3, 4, 7, 1],
[1, 0, 2, 3]]

B = [[1, 3, 4, 1],
[0, 2, 1, 2],
[2, 1, 3, 1],
[1, 2, 0, 1]]

We can compute the dot product of the performance vectors for each match as follows:

A*B = [[(2*1) + (1*0) + (5*2) + (1*1), (2*3) + (1*2) + (5*1) + (1*2),

(2*4) + (1*1) + (5*3) + (1*0), (2*1) + (1*2) + (5*1) + (1*1)],
[(1*1) + (2*0) + (4*2) + (2*1), (1*3) + (2*2) + (4*1) + (2*2),
(1*4) + (2*1) + (4*3) + (2*0), (1*1) + (2*2) + (4*1) + (2*1)],
[(3*1) + (4*0) + (7*2) + (1*1), (3*3) + (4*2) + (7*1) + (1*2),
(3*4) + (4*1) + (7*3) + (1*0), (3*1) + (4*2) + (7*1) + (1*1)],
[(1*1) + (0*0) + (2*2) + (3*1), (1*3) + (0*2) + (2*1) + (3*2),
(1*4) + (0*1) + (2*3) + (3*0), (1*1) + (0*2) + (2*1) + (3*1)]]

C = [[13, 18, 27, 10],

[11, 15, 16, 11],
[18, 30, 40, 19],
[8, 11, 10, 6]]

The matrix product A*B represents the weighted combination of the three metrics (goals scored, assists made, and shots taken) for each match
played by the football team. Each element in the resulting matrix C represents the weighted sum of the corresponding elements in matrix A and
matrix B.

For example, the element in the first row and first column of matrix C (13) represents the weighted sum of the number of goals scored, assists
made, key passes and shots taken in the first match, where the weightings are given by the elements in the first row of matrix A and the first
column of matrix B. Similarly, the element in the second row and third column of matrix C (16) represents the weighted sum of the metrics for the
third match, where the weightings are given by the second row of matrix A and the third column of matrix B.

Symmetric and Identity Matrices

A symmetric matrix is a matrix that remains the same after its transpose. Symmetric matrices are often used to represent the covariance matrix
between different variables i.e. how related features are with each other.

I will cover this in more detail when we reach eigenvectors and eigenvalues.
 import numpy as np

# Define a symmetric matrix A

A = np.array([[1, 2, 3],
[2, 4, 5],
[3, 5, 6]])

# Print the transpose of A

print(A.T)

Output:

array([[1, 2, 3],
[2, 4, 5],
[3, 5, 6]])

An identity matrix is a square matrix with ones on the diagonal and zeros elsewhere.

In other words, the diagonal elements of the identity matrix are equal to 1, and all other elements are equal to 0. In linear algebra, any matrix
multiplied by an identity matrix of the appropriate size is equal to the original matrix. That is, if A is an m x n matrix, then AI = IA = A, where I is
the appropriate identity matrix.

It is important for various machine learning techniques which you will see later.

Example

import numpy as np

# create a 3x3 identity matrix

I = np.eye(3)

# create a sample 3x3 matrix

A = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])

# multiply A by I
B = np.dot(A, I)

print("A:")
print(A)

print("I:")
print(I)

print("B = A x I:")
print(B)
Output:

A:
[[1 2 3]
[4 5 6]
[7 8 9]]

I:
[[1 0 0]
[0 1 0]
[0 0 1]]

B = A x I:
[[1 2 3]
[4 5 6]
[7 8 9]]

Matrix Inversion

Inversing a matrix is used to solve linear systems. In linear algebra, Ax = b is a standard way of representing a system of linear equations. Here,
A is a standard matrix of coefficients:

Suppose we also have a vector x containing the number of goals, shots, and assists, respectively, for a given player, and a vector b representing
the total number of points scored by that player. Then we can express the relationship between x and b as follows: Ax = b

where A is the coefficient matrix and b and x are column vectors given by:

x = | goals |
| shots |
| assists |

b = | points |

To solve for x, we can use the inverse of A, which we denote by A-1. Multiplying both sides of the equation by A-1 yields:

1. Ax = b
2. A-1 Ax = A-1 b # (A-1)A = Identity matrix i.e. 1
3. Ix = A-1 b # Ix = 1*x = x
4. x = A-1 b

In Python code, we can find the inverse of A and solve for x as follows:
 import numpy as np

A = np.array([[2, 3, 1], [1, 2, 2], [3, 1, 1]])

b = np.array([9, 10, 8])

# Find the inverse of A

A_inv = np.linalg.inv(A)

# Solve for x
x = np.dot(A_inv, b)

# Print the solution

print("Goals per game: {:.2f}".format(x[0]))
print("Shots per game: {:.2f}".format(x[1]))
print("Assists per game: {:.2f}".format(x[2]))

This yields the solution:

Goals per game: 2.33

Shots per game: 2.33
Assists per game: 1.67

So the player scored approximately 2.33 goals, took 6.67 shots, and made 1.67 assists per game, on average, to score a total of 10 match
performance points.

Diagonal Matrices

Diagonal matrices are needed in machine learning for various tasks such as scaling or rotating data. They are also used in principal component
analysis to decorrelate the data and in linear regression to perform feature selection.

Suppose we have a diagonal matrix D that represents the weight of different metrics when analysing a player's performance, with dimensions
(metrics, metrics). We can use this matrix to adjust the weight of different metrics when analysing a player's performance

import numpy as np

# Define a diagonal matrix representing the weight of different metrics

# Here, we have four metrics: goals scored, assists, successful passes,
and interceptions
D = np.diag([0.3, 0.2, 0.1, 0.4])

Output of the diagonal matrix D:

 # Define a matrix of player data, with dimensions (players, metrics)
# Here, we have three players: Messi, Ronaldo, and Neymar
# And we are measuring their performance in the four metrics defined
above
player_data = np.array([
[50, 20, 1000, 50],
[40, 30, 800, 40],
[30, 25, 1200, 60]
])

# Compute a weighted average of each player's performance, using the

diagonal matrix
weighted_data = np.dot(player_data, D)

# Print the weighted data for each player

for i, player in enumerate(['Messi', 'Ronaldo', 'Neymar']):
print(f"{player}'s weighted data:", weighted_data[i])

Output:

Messi's weighted data: [ 15. 4. 120. 20.]

Ronaldo's weighted data: [ 12. 6. 80. 16.]
Neymar's weighted data: [ 9. 5. 120. 24.]

Here, we can see the weighted data for each player, where each row represents a player's performance in the four metrics weighted by the
corresponding values in the diagonal matrix D. For example, Messi's weighted data shows that his 50 goals scored are given a weight of 0.3, his
20 assists are given a weight of 0.2, his 1000 successful passes are given a weight of 0.1, and his 50 interceptions are given a weight of 0.4. The
weighted data for Ronaldo and Neymar are computed similarly.

In the example code I provided, we used a diagonal matrix D to represent the weights of different metrics when analysing a player's
performance in football. This allows us to adjust the importance of each metric and compute a weighted average of the player's
performance, taking into account the relative importance of each metric.

In machine learning, we might use weighted data in a similar way to adjust the importance of different features when training a model or
making predictions. For example, if we are trying to predict the likelihood of a customer churning, we might assign higher weights to
features that are strong predictors of churn (such as the number of customer service calls) and lower weights to features that are less
predictive (such as the customer's name).