Non-Linear Regression in R

Non-Linear Regression is a statistical method that is used to model the relationship between a dependent variable and one of the independent variable(s). In non-linear regression, the relationship is modeled using a non-linear equation. This means that the model can capture more complex and non-linear relationships between the variables, but also requires more computational resources and a more sophisticated approach to estimate the model's parameters.

We will implement non-linear regression model in R programming language. Before starting, make sure the following packages are installed:

install.packages("minpack.lm")
install.packages("ggplot2")

Examples of Non-Linear Regression

We will look at some example implementation of Non-Linear Regression in R using different models like exponential, polynomial (quadratic and cubic) and visualize them.

Example 1

This example demonstrates exponential regression in R using the ggplot2 and nls packages. We begin by loading the necessary libraries and generating sample data for the independent variable x and the dependent variable y. The model is then fitted using the nls() function, which performs non-linear least squares estimation. In this case, we define the relationship between x and y using the formula:

y= a\cdot\exp(b\cdot x)

This represents an exponential function with parameters a and b. Initial values for these parameters are provided via the start argument as list(a = 4, b = 2) to guide the fitting process.

library(minpack.lm)
library(ggplot2)

x <- c(0, 1, 2, 3, 4, 5)
y <- c(1, 2, 4, 8, 16, 32)

start_values <- c(a=4, b=2)
fit <- nls(y ~ a * exp(b * x),
           start = start_values,
           algorithm = "port",
           control = nls.control(maxiter = 1000))
summary(fit)

Output:

Formula: y ~ a * exp(b * x)

Parameters:
Estimate Std. Error t value Pr(>|t|)
a 1.000e+00 6.277e-14 1.593e+13 <2e-16 ***
b 6.931e-01 1.331e-14 5.206e+13 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.247e-13 on 4 degrees of freedom

Algorithm "port", convergence message: absolute function convergence (6)

Plot the Exponential Regression line with points

ggplot(data.frame(x, y), aes(x, y)) +
  geom_point() +
  geom_line(aes(x, predict(fit, newdata = data.frame(x)))) +
  ggtitle("Exponential Regression") +
  xlab("x") +
  ylab("y")

Output:

Example 2

This example illustrates polynomial regression of degree 2 in R, using the ggplot2 package for visualization and the lm() function from base R to fit the model. The independent variable x is defined as a sequence of integers from 1 to 10, while the dependent variable y is generated using the equation:

y=x^2+x+2+\text{random noise}

Random noise is introduced using the rnorm() function with a mean of 0 and a standard deviation of 10 to simulate real-world variability. A data frame df is created containing the x and y values. The polynomial regression model is then fitted using lm(y ~ poly(x, 2), data = df), where poly(x, 2) indicates a second-degree polynomial.

#data 
x <- 1:10
y <- x^2 + x + 2 + rnorm(10, 0, 10)
df <- data.frame(x, y)
#fitting the model
fit <- lm(y ~ poly(x, 2), data = df)
summary(fit)

Output:

Call:
lm(formula = y ~ poly(x, 2), data = df)

Residuals:
Min 1Q Median 3Q Max
-9.975 -5.664 2.832 5.523 6.885

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 47.140 2.395 19.686 2.18e-07 ***
poly(x, 2)1 85.981 7.573 11.354 9.21e-06 ***
poly(x, 2)2 18.278 7.573 2.414 0.0465 *
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 7.573 on 7 degrees of freedom
Multiple R-squared: 0.9506, Adjusted R-squared: 0.9365
F-statistic: 67.37 on 2 and 7 DF, p-value: 2.676e-05

Plot the regression line

library(ggplot2)
#plotting the model
ggplot(df, aes(x, y)) +
  geom_point() +
  geom_line(aes(x, predict(fit))) +
  ggtitle("Polynomial Regression")

Output:

Example 3

This example illustrates cubic regression in R, using the ggplot2 package for visualization and the lm() function from base R to fit the model. The independent variable x is defined as a sequence of integers from 1 to 10, while the dependent variable y is generated using the equation:

y = x^3 - 2x^2 + x + 2 + \text{random noise}

Random noise is introduced using the rnorm() function with a mean of 0 and a standard deviation of 10 to simulate real-world variability.

x <- 1:10
y <- x^3 - 2 * x^2 + x + 2 + rnorm(10, 0, 10)
df <- data.frame(x, y)
fit <- lm(y ~ poly(x, 3), data = df)
summary(fit)

Output:

Call:
lm(formula = y ~ poly(x, 3), data = df)

Residuals:
Min 1Q Median 3Q Max
-11.268 -7.677 -4.150 7.141 16.517

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 234.036 3.826 61.169 1.28e-09 ***
poly(x, 3)1 748.529 12.099 61.867 1.20e-09 ***
poly(x, 3)2 328.759 12.099 27.172 1.64e-07 ***
poly(x, 3)3 61.231 12.099 5.061 0.00231 **
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 12.1 on 6 degrees of freedom
Multiple R-squared: 0.9987, Adjusted R-squared: 0.998
F-statistic: 1530 on 3 and 6 DF, p-value: 4.86e-09

Plot the regression line

#plotting the regression line
library(ggplot2)
ggplot(df, aes(x, y)) +
  geom_point() +
  geom_line(aes(x, predict(fit))) +
  ggtitle("Cubic Regression")

Output:

Example 4

This example illustrates quadratic regression in R, using the ggplot2 package for visualization and the lm() function from base R to fit the model. The independent variable x is defined as a sequence of integers from 1 to 10, while the dependent variable y is generated using the equation:

y = x^2 + 2x + 2 + \text{random noise}

Random noise is introduced using the rnorm() function with a mean of 0 and a standard deviation of 10 to simulate real-world variability

x <- 1:10
#quadratic equation
y <- x^2 + 2 * x + 2 + rnorm(10, 0, 10)
df <- data.frame(x, y)

fit <- lm(y ~ poly(x, 2), data = df)
summary(fit)

Output:

Call:
lm(formula = y ~ poly(x, 2), data = df)

Residuals:
Min 1Q Median 3Q Max
-13.2189 -7.5274 0.0416 5.6517 16.7008

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 51.901 3.384 15.339 1.21e-06 ***
poly(x, 2)1 102.037 10.700 9.536 2.92e-05 ***
poly(x, 2)2 19.363 10.700 1.810 0.113
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 10.7 on 7 degrees of freedom
Multiple R-squared: 0.9308, Adjusted R-squared: 0.9111
F-statistic: 47.11 on 2 and 7 DF, p-value: 8.699e-05

Plot the regression line

#import libraries
library(ggplot2)
#plotting the model
ggplot(df, aes(x, y)) +
  geom_point() +
  geom_line(aes(x, predict(fit))) +
  ggtitle("Quadratic Regression")

Output: