Intro to Logistic Regression
- 1. Logistic Regression Jacquelyn Victoria & Tamer Wahba 1
- 2. Slide Ownership Jacquelyn Victoria - 3 to 9 Tamer Wahba - 10 to 15 2
- 3. Regression Analysis + Classification How can we predict a nominal class using regression analysis? Consider a binary class: Each instance x is a vector of feature values Our output values or class labels are restricted to 0 or 1, i.e. f(x) ∈ {0, 1} We need an h(x) where: 0 < h(x) < 1 We need a function which exhibits this behavior 3
- 4. Logistic Functions Sigmoid Function σ(x) Asymptotes at y = 1 and y = 0 Easy to specify threshold (σ(0) = .5) Results are P(y=1) As a result: Where θ is a vector of weights 4
- 5. Cost Function Need to find hθ(x) that is a logistic function that represents our data Need to find θ to fit our data -log(1-x)-log(x) 5
- 6. Gradient Descent In order to find the minimum, we can use the partial derivative of J(θ) do { }until θ converges Where α is the learning rate (almost always between 0 and 1, .1-.3 usually a good range) 6
- 7. Maximum Likelihood Estimation 7 do { }until θ converges Can also be calculated using: Iteratively Reweighted Least Squares Multinomial data uses Softmax Regression
- 8. Interpreting hypothesis 8 Recall that σ(0) = .5 and that hθ(x) = σ(θTx) x1 x2
- 9. Interpreting hθ I want to create a model to give me the probability that I will pass a test given how many hours I have studied Hours 0.50 0.75 1.00 1.25 1.50 1.75 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 4.00 4.25 4.50 4.75 5.00 5.50 Pass 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 Using this generated model, calculate my probability of passing given I have studied 3 hours P(passing| study time = 3) = .61 9source
- 10. Logistic Regression Compared to Other Classifiers Naive Bayes Support Vector Machines Decision Trees 10
- 11. vs Decision Tree Assumptions DT: decision boundaries parallel to axes LR: one smooth boundary Decision trees can be used when there are multiple decision boundaries 11
- 12. Feature Weights NB: each set independently depending on class LR: together such that decision function tends to be high for positive classes and low for negative classes Correlated features have no effect on logistic regression vs Naive Bayes 12
- 13. vs Support Vector Machine 13 Both attempt to find hyperplane separating training samples SVM: find the solution with maximum margin LR: find any solution that separates the instances SVM is a hard classified while LR is probabilistic
- 14. Advantages Works well with diagonal decision boundaries Does not give undue weight to correlated features Probabilistic outcomes 14 Requires large sample size for stable results Disadvantages
- 15. Use Cases Categorical outcomes Large sample data Minimal preprocessing 15
- 16. For more info... Helpful links to go into more depth with Logistic Regression Stanford Open Course (Logit regression section) Logit Regression Tutorial (exercises in MATLAB) Logit Regression Tutorial (no code) How to use Logit Regression in Python How to use Logit Regression in R How to use Logit Regression in Java using Weka 16
Editor's Notes
- #5: hθ(x) = σ(θTx)
- #8: We’re trying to find the most likely theta given our test instances. This can be nondeterministic, Also need stopping criteria.
- #9: We can actually add terms such as x1^2, or x1*x2^4 to make a non-linear
- #10: The first entry in our instance vector is always one, due to the weight of the intercept When we say theta transpose x - this means that we transpose theta and then multiply it, using matrix multiplication,
- #14: Draw SVM diagram on board