CHAPTER 4

FUNCTION APPROXIMATION
 Definition of Regression
 Regression analysis is an important tool in statistics and data science. It helps us

understand and measure how variables are related and how it will affect on each other.

 Regression analysis is super important for understanding data and making smart

decisions.

 Regression analysis aims to find the best model that shows how one variable depends

on others. We do this by drawing a line or curve through the data points, so it matches

the actual results as closely as possible.

 We call the differences between the actual and predicted values residuals.
 Definition of Regression
 Given 𝑛 data points 𝑥1 , 𝑦1 , 𝑥2 , 𝑦2 , … , 𝑥𝑛 , 𝑦𝑛 , we should find the best fit to the data

𝑦 = 𝑓(𝑥)

 The Residual at each points is 𝐸𝑖 = 𝑦𝑖 − 𝑓(𝑥𝑖 )

 Linear Regression
 Given 𝑛 data points 𝑥1 , 𝑦1 , 𝑥2 , 𝑦2 , … , 𝑥𝑛 , 𝑦𝑛 , we should find the best fit to the data

𝑦 = 𝑎0 + 𝑎1 𝑥

 Does minimizing 𝑛
𝑖=1 𝐸𝑖 work as a criterion?
 Linear Regression
 Example:

 Given the data points 2,4 , 3,6 , 2,6 , 3,8 , best fit the data to a straight line using

the criterion of minimizing 𝑖=1 𝐸𝑖 ?

𝑛

𝒙 𝒚
2 4
3 6
2 6
3 8
 Linear Regression
 Example:

 To fit a straight line, we choose 2 points

2,4 , 3,8

 4 = 𝑎0 + 𝑎1 (2)

 8 = 𝑎0 + 𝑎1 (3)
𝒙 𝒚 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 𝑬 = 𝒚 − 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅
 𝑦 = 4𝑥 − 4 will be the regression line
2 4 4 0
3 6 8 −2
2 6 4 2
3 8 8 0
4

𝐸𝑖 = 0
𝑖=1
 Linear Regression
 Example:

 To fit a straight line, we choose 2 points

2,6 , 3,6

 6 = 𝑎0 + 𝑎1 (2)

 6 = 𝑎0 + 𝑎1 (3)
𝒙 𝒚 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 𝑬 = 𝒚 − 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅
 𝑦 = 6 will be the regression line
2 4 6 −2
3 6 6 0
2 6 6 0
3 8 6 2
4

𝐸𝑖 = 0
𝑖=1
 Linear Regression
 Example:

 4
𝑖=1 𝐸𝑖 = 0 for both regression models 𝑦 = 4𝑥 − 4 and 𝑦 = 6

 The sum of the residuals is minimized, in this case it is zero, but the regression model

is not unique

 Thus the criterion of minimizing the sum of the residuals is a bad criterion
 Linear Regression
 Will minimizing 𝑛
𝑖=1 |𝐸𝑖 | work better?
 Linear Regression
 Example:

 Given the data points 2,4 , 3,6 , 2,6 , 3,8 , best fit the data to a straight line using

the criterion of minimizing 𝑛

𝑖=1 |𝐸𝑖 |?

𝒙 𝒚
2 4
3 6
2 6
3 8
 Linear Regression
 Example:

 To fit a straight line, we choose 2 points

2,4 , 3,8

 4 = 𝑎0 + 𝑎1 (2)

 8 = 𝑎0 + 𝑎1 (3)
𝒙 𝒚 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 𝑬 = 𝒚 − 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅
 𝑦 = 4𝑥 − 4 will be the regression line
2 4 4 0
3 6 8 −2
2 6 4 2
3 8 8 0
4

𝐸𝑖 = 4
𝑖=1
 Linear Regression
 Example:

 To fit a straight line, we choose 2 points

2,6 , 3,6

 6 = 𝑎0 + 𝑎1 (2)

 6 = 𝑎0 + 𝑎1 (3)
𝒙 𝒚 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 𝑬 = 𝒚 − 𝒚𝒑𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅
 𝑦 = 6 will be the regression line
2 4 6 −2
3 6 6 0
2 6 6 0
3 8 6 2
4

𝐸𝑖 = 4
𝑖=1
 Linear Regression
 Example:

 4
𝑖=1 𝐸𝑖 = 4 for both regression models 𝑦 = 4𝑥 − 4 and 𝑦 = 6

 The sum of the absolute residuals has been made as small as possible, in this case it is

four, but the regression model is not unique

 Thus the criterion of minimizing the sum of the absolute residuals is also a bad

criterion
 Linear Regression
 The least squares criterion minimizes the sum of the square of the residuals in the

model, and also produces a unique line.

𝒏 𝟐 𝒏
 𝑺𝒓 = 𝒊=𝟏 𝑬𝒊 = 𝒊=𝟏(𝒚𝒊 − 𝒂 𝟎 − 𝒂 𝟏 𝒙𝒊 ) 𝟐
 Linear Regression
𝒏 𝟐 𝒏
 𝑺𝒓 = 𝒊=𝟏 𝑬𝒊 = 𝒊=𝟏(𝒚𝒊 − 𝒂 𝟎 − 𝒂 𝟏 𝒙𝒊 ) 𝟐

 To find 𝒂𝟎 and 𝒂𝟏 we minimize 𝑺𝒓 with respect to 𝒂𝟎 and 𝒂𝟏 :

𝜕𝑆𝑟 𝑛
 =2 𝑖=1 𝑦𝑖 − 𝑎0 − 𝑎1 𝑥𝑖 −1 = 0
𝜕𝑎0

𝜕𝑆𝑟 𝑛
 =2 𝑖=1 𝑦𝑖 − 𝑎0 − 𝑎1 𝑥𝑖 −𝑥𝑖 = 0
𝜕𝑎1

 Or we have:
𝑛 𝑛 𝑛
 𝑖=1 𝑎0 + 𝑖=1 𝑎1 𝑥𝑖 = 𝑖=1 𝑦𝑖
𝑛 𝑛 𝑛
 𝑖=1 𝑎0 𝑥𝑖 + 𝑖=1 𝑎1 𝑥𝑖 ² = 𝑖=1 𝑦𝑖 𝑥𝑖
 Linear Regression
𝒏 𝟐 𝒏
 𝑺𝒓 = 𝒊=𝟏 𝑬𝒊 = 𝒊=𝟏(𝒚𝒊 − 𝒂 𝟎 − 𝒂 𝟏 𝒙𝒊 ) 𝟐

 Solving for 𝒂𝟎 and 𝒂𝟏 :

𝑛 𝑛 𝑛 𝑛
𝑖=1 𝑦𝑖 𝑥𝑖 − 𝑖=1 𝑥𝑖 𝑖=1 𝑦𝑖
 𝑎1 = 2
𝑛 𝑛 𝑛
𝑖=1 𝑥𝑖 −( 𝑖=1 𝑥𝑖 )²

𝑛 2 𝑛 𝑛 𝑛
𝑖=1 𝑥𝑖 𝑖=1 𝑦𝑖 − 𝑖=1 𝑥𝑖 𝑖=1 𝑥𝑖 𝑦𝑖
 𝑎0 = 𝑛 𝑥 2− 𝑛 𝑥 2
= 𝑦 − 𝑎1 𝑥
𝑛 𝑖=1 𝑖 𝑖=1 𝑖
 Linear Regression
 Example:

 The torque, T needed to turn the torsion spring of a mousetrap through an angle, is

given below. Find the constants for the model given by T = 𝑘1 + 𝑘2 𝜃

𝑨𝒏𝒈𝒍𝒆, 𝜃 𝑻𝒐𝒓𝒒𝒖𝒆, 𝑻
𝑹𝒂𝒅𝒊𝒂𝒏𝒔 𝑵−𝒎
0.698132 0.188224
0.959931 0.209138
1.134464 0.230052
1.570796 0.250965
1.919862 0.313707
 Linear Regression
 Example:

 The following table shows the summations needed for the calculations of the constants

in the regression model.

𝑨𝒏𝒈𝒍𝒆, 𝜃 𝑻𝒐𝒓𝒒𝒖𝒆, 𝑻 𝜃² 𝑻𝜃
𝑹𝒂𝒅𝒊𝒂𝒏𝒔 𝑵−𝒎 𝑹𝒂𝒅𝒊𝒂𝒏𝒔2 𝐍 − 𝐦 − 𝑹𝒂𝒅𝒊𝒂𝒏𝒔
0.698132 0.188224 0.487388 0.131405
0.959931 0.209138 0.921468 0.200758
1.134464 0.230052 1.2870 0.260986
1.570796 0.250965 2.4674 0.394215
1.919862 0.313707 3.6859 0.602274

6.2831 1.1921 8.8491 1.5896

 Linear Regression

 Example:

𝑨𝒏𝒈𝒍𝒆, 𝜃 𝑻𝒐𝒓𝒒𝒖𝒆, 𝑻 𝜃² 𝑻𝜃
5 5 5 5
𝑖=1 𝑇𝑖 𝜃𝑖 − 𝑖=1 𝜃𝑖 𝑖=1 𝑇𝑖
𝑹𝒂𝒅𝒊𝒂𝒏𝒔 𝑵−𝒎 𝑹𝒂𝒅𝒊𝒂𝒏𝒔2 𝐍 − 𝐦 − 𝑹𝒂𝒅𝒊𝒂𝒏𝒔
 𝑘2 = 2
0.698132 0.188224 0.487388 0.131405
5 5𝑖=1 𝜃𝑖 −( 5𝑖=1 𝜃𝑖 )²
0.959931 0.209138 0.921468 0.200758
1.134464 0.230052 1.2870 0.260986
5(1.5896)−(6.2831)(1.1921) −2
 𝑘2 = = 9.6091 × 10 1.570796 0.250965 2.4674 0.394215
5 8.8491 −(6.2831)²
1.919862 0.313707 3.6859 0.602274

5 5
𝑖=1 𝑇𝑖 𝑖=1 𝜃𝑖
 𝑘1 = 𝑇 − 𝑘2 𝜃 = − 𝑘2 6.2831 1.1921 8.8491 1.5896
𝑛 𝑛

1.1921 9.6091×10−2 6.2831

 𝑘1 = − = 1.1767 × 10−1
5 5
 Linear Regression
 Example:

5 5𝑖=1 𝑇𝑖 𝜃𝑖 − 5𝑖=1 𝜃𝑖 5
𝑖=1 𝑇𝑖
 𝑘2 = 2
5 5𝑖=1 𝜃𝑖 −( 5𝑖=1 𝜃𝑖 )²

5(1.5896)−(6.2831)(1.1921)
 𝑘2 = = 9.6091 × 10−2
5 8.8491 −(6.2831)²

5 5
𝑖=1 𝑇𝑖 𝑖=1 𝜃𝑖
 𝑘1 = 𝑇 − 𝑘2 𝜃 = − 𝑘2
𝑛 𝑛

1.1921 9.6091×10−2 6.2831

 𝑘1 = − = 1.1767 × 10−1
5 5