Correlation and Simple Regression
46 likes44,174 views
This document provides an overview of a data analysis course covering various statistical techniques including correlation, regression, hypothesis testing, clustering, and time series analysis. The course covers descriptive statistics, data exploration, probability distributions, simple and multiple linear regression analysis, logistic regression analysis, and model building for credit risk analysis. Notes are provided on correlation calculation and its properties. Assumptions and interpretations of linear regression are also summarized. The document is intended as a high-level overview of topics covered in the course rather than an in-depth treatment.
![What is Correlation
• It is a measure of association(linear association only)
• Formula for correlation coefficient
r is the ratio of variance together and product of separate variances
Venkat Reddy
Data Analysis Course
r= cov(XY)/sd(x)*sd(y)
r = [n(xy) – (x)(y)] / {[n(x2) – (x)2][n(y2) – (y)2]}0.5
Where n is the number of data pairs, x is the independent
variable and y the dependent variable.
6](https://image.slidesharecdn.com/correlationregression-130320062639-phpapp01/85/Correlation-and-Simple-Regression-6-320.jpg)
![Simple Linear Regression Analysis
• If you know something about X, this knowledge helps you predict
something about Y.
Venkat Reddy
Data Analysis Course
• Variables:
• X = Independent Variable (we provide this)
• Y = Dependent Variable (we observe this)
• Parameters:
• β0 = Y-Intercept
• β1 = Slope
• ε ~ Normal Random Variable (με = 0, σε = ???) [Noise] 22](https://image.slidesharecdn.com/correlationregression-130320062639-phpapp01/85/Correlation-and-Simple-Regression-22-320.jpg)
Correlation and Simple Regression
- 1. Data Analysis Course Correlation and Regression(Version-1) Venkat Reddy
- 2. Data Analysis Course • Data analysis design document • Introduction to statistical data analysis • Descriptive statistics • Data exploration, validation & sanitization • Probability distributions examples and applications • Simple correlation and regression Venkat Reddy Data Analysis Course analysis • Multiple liner regression analysis • Logistic regression analysis • Testing of hypothesis • Clustering and decision trees • Time series analysis and forecasting • Credit Risk Model building-1 2 • Credit Risk Model building-2
- 3. Note • This presentation is just class notes. The course notes for Data Analysis Training is by written by me, as an aid for myself. • The best way to treat this is as a high-level summary; the actual session went more in depth and contained other Venkat Reddy Data Analysis Course information. • Most of this material was written as informal notes, not intended for publication • Please send questions/comments/corrections to [email protected] or [email protected] • Please check my website for latest version of this document -Venkat Reddy 3
- 4. Contents • What is Correlation • Correlation calculation • Properties of correlation • What is Regression Venkat Reddy Data Analysis Course • Assumptions • Meaning of Beta • Least squares Coefficient estimation • Goodness of fit • Output interpretation 4
- 5. What is need of correlation? • What happens to Sweater sales with increase in temperature? • What is the strength of association between them? • Ice-cream sales v.s temperature ? • What is the strength of association between them? Venkat Reddy Data Analysis Course • Which one of these two is stronger? How to quantify the association? 5
- 6. What is Correlation • It is a measure of association(linear association only) • Formula for correlation coefficient r is the ratio of variance together and product of separate variances Venkat Reddy Data Analysis Course r= cov(XY)/sd(x)*sd(y) r = [n(xy) – (x)(y)] / {[n(x2) – (x)2][n(y2) – (y)2]}0.5 Where n is the number of data pairs, x is the independent variable and y the dependent variable. 6
- 7. Type of relationship Linear relationships Curvilinear relationships Y Y Venkat Reddy Data Analysis Course X X Y Y 7 X X
- 8. Type of relationship Strong relationships Weak relationships Y Y Venkat Reddy Data Analysis Course X X Y Y 8 X X
- 9. Type of relationship No relationship Y Venkat Reddy Data Analysis Course X Y 9 X
- 10. Range of Correlation • In case of exact positive linear Correlation = +1 Dependent variable relationship the value of r is 25 ___. 20 • In case of a strong positive 15 linear relationship, the value of 10 12 14 16 18 20 Independent variable r will be close to ___. • In case of exact negative linear Correlation = -1 relationship the value of r is ___. Dependent variable 25 • In case of a strong negative 20 linear relationship, the value of r 15 10 12 14 16 18 20 will be close to ___. Independent variable Data Analysis Course Venkat Reddy 10
- 11. Range of Correlation Correlation = 0 Dependent variable 30 In case of a weak relationship the 25 value of r will be close to ___. 20 15 10 0 2 4 6 8 10 12 Independent variable Correlation = 0 In case of nonlinear relationship Dependent variable 30 the value of r will be close to 20 ___. 10 0 0 2 4 6 8 10 12 Independent variable Data Analysis Course Venkat Reddy 11
- 12. Strength of Association • Correlation 0 No linear association • Correlation 0 to 0.25 Negligible positive association • Correlation 0.25-0.5 Weak positive Venkat Reddy Data Analysis Course association • Correlation 0.5-0.75 Moderate positive association • Correlation >0.75 Very Strong positive association • What are the limits for negative correlation 12
- 13. Properties of Correlation • -1 ≤ r ≤ +1 • r=0 represents no linear relationship between the two variables • Correlation is unit free Venkat Reddy Data Analysis Course Limitations: • Though r measures how closely the two variables approximate a straight line, it does not validly measures the strength of nonlinear relationship • When the sample size, n, is small we also have to be careful with the reliability of the correlation • Outliers could have a marked effect on r 13
- 14. Lab • Create these tables and find correlations 3 1 2 X Y X Y 10 14 -31 900 X Y X Y 17 25 -25 625 1 2 1 2 22 23 Venkat Reddy Data Analysis Course -24 576 2 4 2 2.9 21 31 -19 361 3 3 3 3 24 29 -13 169 34 60 -6 36 25 19 -1 1 31 35 3 9 45 45 10 100 33 38 11 121 14 196 60 50 15 225 46 56 24 576 47 45 24 576 48 70 29 841 50 750 14
- 15. Correlation - Limitations 3 1 2 X Y X Y 10 14 -31 900 X Y X Y 17 25 -25 625 1 2 1 2 22 23 -24 576 2 4 2 2.9 21 31 -19 361 3 3 3 3 24 29 -13 169 34 60 -6 36 r= 0.5 r= 0.9 25 19 -1 1 31 35 3 9 45 45 Venkat Reddy Data Analysis Course 10 100 11 121 Correlation is a measure 33 38 14 196 of linear relationship 60 50 15 225 46 56 r = -0.12 only 47 45 24 576 24 576 48 70 r= 0.44 29 841 50 750 r= 0.86 • Example1: Y is both decreased and increased when X is increased. Correlation is -0.12, but this is not an appropriate measure of association • Example-2: Correlation changed from 0.5 to 0.9 with a small change in the data. ‘r’ is not reliable when n is small • Example-3: Correlation between X and Y is 0.44, correlation between X & Y is 0.86 if we 15 exclude outlier
- 16. Correlation vs. Possible Relationships Between Variables • Direct cause and effect, that is x cause y or water causes plant to grow. • Both cause and effect, that y cause x or coffee consumption causes nervousness as well nervous people have more coffee. Venkat Reddy Data Analysis Course • Relationship caused by third variable; Death due to drowning and soft drink consumption during summer. Both variables are related to heat and humidity (third variable). –This is dangerous (Why?) • Coincidental relationship; Increase in the number of people exercising and increase in the number of people committing crimes. –This is even more dangerous (Why?) 16 • Correlation measures association and not causation.
- 17. Lab -Correlation • Download stock price data from here • Find the correlation between IBM open & Intel open price • Can we apply correlation on this data? Draw a scatter plot between the stocks –What is the type of relationship Venkat Reddy Data Analysis Course • When Intel stock open price increased • What happened to Microsoft open price? • What happened to IBM open price? • Is there any correlation between closing price of three stocks? • Correlation between the stocks with respect to change in price? 17
- 18. Correlation Practice • Download Age vs Blood Pressure data • Is there any association between age and blood pressure? • Quantify the association between age and blood pressure. • How strong is the association between age and blood Venkat Reddy Data Analysis Course pressure? • What is your observation? As age increases does blood pressure increase or decrease? • Are there any outliers? Is your measure of association reliable? Is it high/low due to outliers? • What is the final verdict, in the given sample did you see a strong/moderate/no association between Age & BP 18
- 19. Regression Data Analysis Course 19 Venkat Reddy
- 20. Why Regression? Last 30 days data for a KFC shop in a Mall. Number of visitors v.s burgers sold (download it from here) Venkat Reddy Data Analysis Course Number of visitors are expected to be 6500 tomorrow. How many burgers will be sold? 20
- 21. Regression • Regression analysis is used to predict the value of one variable (the dependent variable) on the basis of other variables (the independent variables). • In correlation, the two variables are treated as equals. In regression, one variable is considered independent (=predictor) variable (X) and the other the dependent (=outcome) variable Y. Venkat Reddy Data Analysis Course • Dependent variable: denoted Y • Independent variables: denoted X1, X2, …, Xk • Above model is referred to as simple linear regression. We would be interested in estimating β0 and β1 from the data we collect. 21
- 22. Simple Linear Regression Analysis • If you know something about X, this knowledge helps you predict something about Y. Venkat Reddy Data Analysis Course • Variables: • X = Independent Variable (we provide this) • Y = Dependent Variable (we observe this) • Parameters: • β0 = Y-Intercept • β1 = Slope • ε ~ Normal Random Variable (με = 0, σε = ???) [Noise] 22
- 23. Assumptions of linear regression- When Can I fit the linear regression line Linear regression assumes that… 1. The relationship between X and Y is linear: Imagine a quadratic(parabolic) relation ship between X & Y. Does it make sense to fit a straight line through this data. 2. Y is distributed normally at each value of X: Imagine Y=0 whenever x is a multiple of 5. does it make sense to fit a straight line Venkat Reddy Data Analysis Course through this data. At each X, Y is normally distributed, which means at each X, Y value is around its overall mean value 3. The variance of Y at every value of X is the same (homogeneity of variances): Imagine data in the form of a cone, as we move away from origin the variance in Y is increasing drastically. Does it make sense to fit a straight line through this data? 4. The observations are independent: There is already one trend in the data If the observations are dependent. One trend line is not 23 sufficient to model in this case
- 24. Regression line y y β0 β1x ε Observed Value of y for xi Venkat Reddy Data Analysis Course εi Slope = β1 Predicted Value Random Error of y for xi for this x value Intercept = β0 xi x 24
- 25. Meaning of Beta • Beta1 denotes the slope. What is slope? A slope of 2 means that every 1- unit change in X yields a 2-unit change in Y • Beta1 is the estimated change in the average value of y as a result of a one-unit change in x • Beta0 is the estimated average value of y when the value of x is zero Venkat Reddy Data Analysis Course y β0 β1x ε y Observed Value of y for xi εi Slope = β1 Predicted Random Error for Value of y for xi this x value 25 Intercept = β0 xi
- 26. Least squares Estimation • X: x1, x2, x3, x4, x5, x6, x7,…….. • Y:y1, y2, y3, y4, y5, y6, y7……. • Imagine a line through all the points • Deviation from each point (residual or error) Venkat Reddy Data Analysis Course • Square of the deviation • Minimizing sum of squares of deviation e 2 (y y) ˆ 2 (y (b0 b1x)) 2 b0 and b1 are obtained by finding the values of b0 and b1 that 26 minimize the sum of the squared residuals
- 27. Lab: Burger Example • Download burger data • What is the mathematical relation between number of visitors and number of burgers sold? • If what is the increase in the burger sales for every 100 Venkat Reddy Data Analysis Course visitors? • If 5000 visitors are expected tomorrow, how much stock should I keep? • If I want to sell 1000 burgers, how many visitors should I expect? • Is it a good plan to increase the burger production above 3000? • How reliable is this mathematical equation? 27
- 28. How good is my regression line? • Take a regression line; Estimate y by substituting xi from data; Is it exactly same as yi? • Remember no line is perfect • There is always some error in the estimation Venkat Reddy Data Analysis Course • Unless there is comprehensive dependency between predictor and response, there is always some part of response(Y) that can’t be explained by predictor (x) • So, total variance in Y is divided into two parts, • Variance that can be explained by x, using regression • Variance that can’t be explained by x 28
- 29. Explained and Unexplained Variation • Total variation is made up of two parts: SST SSE SSR • Total sum of Sum of Squares Sum of Squares Error Regression Venkat Reddy Data Analysis Course Squares SST ( y y) 2 SSE ( y y) ˆ 2 SSR ( y y)2 ˆ SST :Measures the variation of the yi values around their mean y SSE : Variation attributable to factors other than the relationship between x and y SSR: Explained variation attributable to the relationship between x and y 29
- 30. Explained and Unexplained Variation y yi SSE = (yi - i )2 y y _ SST = (yi - y)2 Venkat Reddy Data Analysis Course y _ _ SSR = (yi - y)2 _ y y 30 Xi x
- 31. Coefficient of determination • A good fit will have • SSE (Minimum or Maximum?) • SSR (Minimum or Maximum?) • SSR/SSE(Minimum or Maximum?) • The coefficient of determination is the portion of the total Venkat Reddy Data Analysis Course variation in the dependent variable that is explained by variation in the independent variable • The coefficient of determination is also called R-squared and is denoted as R2 SSR R 2 where 0 R2 1 SST 31 In the single independent variable case, the coefficient of determination is equal to square of simple correlation coefficient
- 32. Type of relationship y R2 = 1 Perfect linear relationship between x and y: Venkat Reddy Data Analysis Course x R2 = 1 100% of the variation in y is explained by variation in x y x 32 R2 = +1
- 33. Type of relationship y 0 < R2 < 1 Venkat Reddy Weaker linear relationship between x Data Analysis Course and y: x Some but not all of the variation in y is y explained by variation in x 33 x
- 34. Type of relationship R2 = 0 y Venkat Reddy No linear relationship between x and Data Analysis Course y: The value of Y does not depend on x. (None of the variation in y is explained x by variation in x) R2 = 0 34
- 35. Lab • How good is the regression line for burger example? • How good is the regression line? • What is total sum of squares? How much model explained? • What percentage of variation in Y(burgers sold) is explained by Venkat Reddy Data Analysis Course X(visitors)? • In the age vs blood pressure example, estimate bp for a given age • How good is the regression line? • What is total sum of squares? How much model explained? • What percentage of variation in Y(blood pressure) is explained by X(age)? 35
- 36. Standard Error of Estimate • The standard deviation of the variation of observations around the regression line is estimated by SSE s Venkat Reddy Data Analysis Course n k 1 Where SSE = Sum of squares error n = Sample size k = number of independent variables in the model 36
- 37. The Standard Deviation of the Regression Slope • The standard error of the regression slope coefficient (b1) is estimated by sε sε sb1 Venkat Reddy Data Analysis Course (x x) 2 ( x) x n 2 2 where: sb1 = Estimate of the standard error of the least squares slope SSE = Sample standard error of the estimate 37 sε n2
- 38. Comparing Standard Errors Variation of observed y values Variation in the slope of regression lines from the regression line from different possible samples y y Venkat Reddy Data Analysis Course smalls x smallsb1 x y y 38 large s x large sb1 x
- 39. Significance testing… Slope ˆ Distribution of slope ~ Tn-2(β,s.e.( )) Venkat Reddy Data Analysis Course H0: β1 = 0 (no linear relationship) H1: β1 0 (linear relationship does exist) ˆ 0 Tn-2= s.e.( ) ˆ Reject or accept the null hypothesis based on above test statistic 39 value.
- 40. Lab • In burger & BP examples • What is the standard error of estimates • What is the standard error estimate of beta • Compare two standard errors Venkat Reddy Data Analysis Course • Which one of these two model is reliable? • Are the coefficients significant? 40
- 41. When can I NOT fit a linear regression line The relationship between X and Y is Y is NOT distributed normally at each value of X NOT linear Venkat Reddy Data Analysis Course The observations are NOT independent Variance of Y at every value of X is NOT same 41
- 42. Venkat Reddy Konasani Manager at Trendwise Analytics [email protected] [email protected] Venkat Reddy Data Analysis Course +91 9886 768879 42