Master Machine Learning Algorithms Discover How

They Work And Implement Them From Scratch V11

Jason Brownlee download

https://ebookbell.com/product/master-machine-learning-algorithms-
discover-how-they-work-and-implement-them-from-scratch-v11-jason-
brownlee-35111876

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Master Machine Learning Algorithms Discover How They Work And

Implement Them From Scratch 11 Jason Brownlee

https://ebookbell.com/product/master-machine-learning-algorithms-
discover-how-they-work-and-implement-them-from-scratch-11-jason-
brownlee-42525092

Master Machine Learning Algorithms Discover How They Work And

Implement Them From Scratch V11 Jason Brownlee

https://ebookbell.com/product/master-machine-learning-algorithms-
discover-how-they-work-and-implement-them-from-scratch-v11-jason-
brownlee-5653674

Master Machine Learning Algorithms Discover How They Work And

Implement Them From Scratch V112 Jason Brownlee

https://ebookbell.com/product/master-machine-learning-algorithms-
discover-how-they-work-and-implement-them-from-scratch-v112-jason-
brownlee-7257896

Machine Learning Master Supervised And Unsupervised Learning

Algorithms With Real Examples English Edition Doshi

https://ebookbell.com/product/machine-learning-master-supervised-and-
unsupervised-learning-algorithms-with-real-examples-english-edition-
doshi-34734256
Machine Learning Algorithms From Scratch With Python V17 Jason
Brownlee

https://ebookbell.com/product/machine-learning-algorithms-from-
scratch-with-python-v17-jason-brownlee-11613838

The Master Algorithm How The Quest For The Ultimate Learning Machine
Will Remake Our World Pedro Domingos

https://ebookbell.com/product/the-master-algorithm-how-the-quest-for-
the-ultimate-learning-machine-will-remake-our-world-pedro-
domingos-5215906

The Master Algorithm How The Quest For The Ultimate Learning Machine
Will Remake Our World Pedro Domingos

https://ebookbell.com/product/the-master-algorithm-how-the-quest-for-
the-ultimate-learning-machine-will-remake-our-world-pedro-
domingos-22876592

The Master Algorithm How The Quest For The Ultimate Learning Machine
Will Remake Our World Pedro Domingos Domingos

https://ebookbell.com/product/the-master-algorithm-how-the-quest-for-
the-ultimate-learning-machine-will-remake-our-world-pedro-domingos-
domingos-23401480

Machine Learning For Algorithmic Trading Master As A Pro Applied

Artificial Intelligence And Python To Predict Systematic Strategies
For Options And Stock Learn Datadriven Finance Using Keras Mark Broker
Jason Test
https://ebookbell.com/product/machine-learning-for-algorithmic-
trading-master-as-a-pro-applied-artificial-intelligence-and-python-to-
predict-systematic-strategies-for-options-and-stock-learn-datadriven-
finance-using-keras-mark-broker-jason-test-46472194
 ������������������������
����������
���������������������������
���������������������������

��������������
Jason Brownlee

Master Machine Learning Algorithms

Discover How They Work and Implement Them From Scratch
 i

Master Machine Learning Algorithms

© Copyright 2016 Jason Brownlee. All Rights Reserved.
Edition, v1.1
http://MachineLearningMastery.com
Contents

Preface iii

I Introduction 1
1 Welcome 2
1.1 Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Algorithm Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Book Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 What This Book is Not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 How To Best Use this Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

II Background 6
2 How To Talk About Data in Machine Learning 7
2.1 Data As you Know It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Statistical Learning Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Computer Science Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Models and Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Algorithms Learn a Mapping From Input to Output 11

3.1 Learning a Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Learning a Function To Make Predictions . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Techniques For Learning a Function . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Parametric and Nonparametric Machine Learning Algorithms 13

4.1 Parametric Machine Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Nonparametric Machine Learning Algorithms . . . . . . . . . . . . . . . . . . . 14
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 Supervised, Unsupervised and Semi-Supervised Learning 16

5.1 Supervised Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Unsupervised Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

ii
 iii

5.3 Semi-Supervised Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

6 The Bias-Variance Trade-Off 19

6.1 Overview of Bias and Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
6.2 Bias Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.3 Variance Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.4 Bias-Variance Trade-Off . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

7 Overfitting and Underfitting 22

7.1 Generalization in Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.2 Statistical Fit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.3 Overfitting in Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.4 Underfitting in Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.5 A Good Fit in Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.6 How To Limit Overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

III Linear Algorithms 25

8 Crash-Course in Spreadsheet Math 26
8.1 Arithmetic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
8.2 Statistical Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
8.3 Random Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
8.4 Flow Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
8.5 More Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

9 Gradient Descent For Machine Learning 30

9.1 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
9.2 Batch Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
9.3 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
9.4 Tips for Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
9.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

10 Linear Regression 34
10.1 Isn’t Linear Regression from Statistics? . . . . . . . . . . . . . . . . . . . . . . . 34
10.2 Many Names of Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 34
10.3 Linear Regression Model Representation . . . . . . . . . . . . . . . . . . . . . . 35
10.4 Linear Regression Learning the Model . . . . . . . . . . . . . . . . . . . . . . . 35
10.5 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
10.6 Making Predictions with Linear Regression . . . . . . . . . . . . . . . . . . . . . 37
10.7 Preparing Data For Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . 37
10.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
 iv

11 Simple Linear Regression Tutorial 40

11.1 Tutorial Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
11.2 Simple Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
11.3 Making Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
11.4 Estimating Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
11.5 Shortcut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
11.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

12 Linear Regression Tutorial Using Gradient Descent 46

12.1 Tutorial Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
12.2 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
12.3 Simple Linear Regression with Stochastic Gradient Descent . . . . . . . . . . . . 47
12.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

13 Logistic Regression 51
13.1 Logistic Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
13.2 Representation Used for Logistic Regression . . . . . . . . . . . . . . . . . . . . 52
13.3 Logistic Regression Predicts Probabilities . . . . . . . . . . . . . . . . . . . . . . 52
13.4 Learning the Logistic Regression Model . . . . . . . . . . . . . . . . . . . . . . . 53
13.5 Making Predictions with Logistic Regression . . . . . . . . . . . . . . . . . . . . 54
13.6 Prepare Data for Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . 54
13.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

14 Logistic Regression Tutorial 56

14.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
14.2 Logistic Regression Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
14.3 Logistic Regression by Stochastic Gradient Descent . . . . . . . . . . . . . . . . 57
14.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

15 Linear Discriminant Analysis 61

15.1 Limitations of Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 61
15.2 Representation of LDA Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
15.3 Learning LDA Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
15.4 Making Predictions with LDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
15.5 Preparing Data For LDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
15.6 Extensions to LDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
15.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

16 Linear Discriminant Analysis Tutorial 65

16.1 Tutorial Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
16.2 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
16.3 Learning The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
16.4 Making Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
16.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
 v

IV Nonlinear Algorithms 71
17 Classification and Regression Trees 72
17.1 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
17.2 CART Model Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
17.3 Making Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
17.4 Learn a CART Model From Data . . . . . . . . . . . . . . . . . . . . . . . . . . 74
17.5 Preparing Data For CART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
17.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

18 Classification and Regression Trees Tutorial 76

18.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
18.2 Learning a CART Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
18.3 Making Predictions on Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
18.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

19 Naive Bayes 82
19.1 Quick Introduction to Bayes’ Theorem . . . . . . . . . . . . . . . . . . . . . . . 82
19.2 Naive Bayes Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
19.3 Gaussian Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
19.4 Preparing Data For Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
19.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

20 Naive Bayes Tutorial 88

20.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
20.2 Learn a Naive Bayes Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
20.3 Make Predictions with Naive Bayes . . . . . . . . . . . . . . . . . . . . . . . . . 91
20.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

21 Gaussian Naive Bayes Tutorial 93

21.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
21.2 Gaussian Probability Density Function . . . . . . . . . . . . . . . . . . . . . . . 94
21.3 Learn a Gaussian Naive Bayes Model . . . . . . . . . . . . . . . . . . . . . . . . 95
21.4 Make Prediction with Gaussian Naive Bayes . . . . . . . . . . . . . . . . . . . . 96
21.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

22 K-Nearest Neighbors 98
22.1 KNN Model Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
22.2 Making Predictions with KNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
22.3 Curse of Dimensionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
22.4 Preparing Data For KNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
22.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

23 K-Nearest Neighbors Tutorial 102

23.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
23.2 KNN and Euclidean Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
23.3 Making Predictions with KNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
23.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
 vi

24 Learning Vector Quantization 106

24.1 LVQ Model Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
24.2 Making Predictions with an LVQ Model . . . . . . . . . . . . . . . . . . . . . . 107
24.3 Learning an LVQ Model From Data . . . . . . . . . . . . . . . . . . . . . . . . . 107
24.4 Preparing Data For LVQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
24.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

25 Learning Vector Quantization Tutorial 110

25.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
25.2 Learn the LVQ Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
25.3 Make Predictions with LVQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
25.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

26 Support Vector Machines 115

26.1 Maximal-Margin Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
26.2 Soft Margin Classifier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
26.3 Support Vector Machines (Kernels) . . . . . . . . . . . . . . . . . . . . . . . . . 116
26.4 How to Learn a SVM Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
26.5 Preparing Data For SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
26.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

27 Support Vector Machine Tutorial 119

27.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
27.2 Training SVM With Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . 120
27.3 Learn an SVM Model from Training Data . . . . . . . . . . . . . . . . . . . . . 121
27.4 Make Predictions with SVM Model . . . . . . . . . . . . . . . . . . . . . . . . . 123
27.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

V Ensemble Algorithms 125

28 Bagging and Random Forest 126
28.1 Bootstrap Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
28.2 Bootstrap Aggregation (Bagging) . . . . . . . . . . . . . . . . . . . . . . . . . . 127
28.3 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
28.4 Estimated Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
28.5 Variable Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
28.6 Preparing Data For Bagged CART . . . . . . . . . . . . . . . . . . . . . . . . . 129
28.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

29 Bagged Decision Trees Tutorial 130

29.1 Tutorial Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
29.2 Learn the Bagged Decision Tree Model . . . . . . . . . . . . . . . . . . . . . . . 131
29.3 Make Predictions with Bagged Decision Trees . . . . . . . . . . . . . . . . . . . 132
29.4 Final Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
29.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
 vii

30 Boosting and AdaBoost 136

30.1 Boosting Ensemble Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
30.2 Learning An AdaBoost Model From Data . . . . . . . . . . . . . . . . . . . . . 136
30.3 How To Train One Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
30.4 AdaBoost Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
30.5 Making Predictions with AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . 138
30.6 Preparing Data For AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
30.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

31 AdaBoost Tutorial 140

31.1 Classification Problem Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
31.2 Learn AdaBoost Model From Data . . . . . . . . . . . . . . . . . . . . . . . . . 141
31.3 Decision Stump: Model #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
31.4 Decision Stump: Model #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
31.5 Decision Stump: Model #3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
31.6 Make Predictions with AdaBoost Model . . . . . . . . . . . . . . . . . . . . . . 147
31.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

VI Conclusions 149
32 How Far You Have Come 150

33 Getting More Help 151

33.1 Machine Learning Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
33.2 Forums and Q&A Websites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
33.3 Contact the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
Preface

Machine learning algorithms dominate applied machine learning. Because algorithms are such
a big part of machine learning you must spend time to get familiar with them and really
understand how they work. I wrote this book to help you start this journey.
You can describe machine learning algorithms using statistics, probability and linear algebra.
The mathematical descriptions are very precise and often unambiguous. But this is not the
only way to describe machine learning algorithms. Writing this book, I set out to describe
machine learning algorithms for developers (like myself). As developers, we think in repeatable
procedures. The best way to describe a machine learning algorithm for us is:

1. In terms of the representation used by the algorithm (the actual numbers stored in a file).

2. In terms of the abstract repeatable procedures used by the algorithm to learn a model
from data and later to make predictions with the model.

3. With clear worked examples showing exactly how real numbers plug into the equations
and what numbers to expect as output.

This book cuts through the mathematical talk around machine learning algorithms and
shows you exactly how they work so that you can implement them yourself in a spreadsheet,
in code with your favorite programming language or however you like. Once you possess this
intimate knowledge, it will always be with you. You can implement the algorithms again and
again. More importantly, you can translate the behavior of an algorithm back to the underlying
procedure and really know what is going on and how to get the most from it.
This book is your tour of machine learning algorithms and I’m excited and honored to be
your tour guide. Let’s dive in.

Jason Brownlee
Melbourne, Australia
2016

viii
 Part I

Introduction

1
Chapter 1

Welcome

Welcome to Master Machine Learning Algorithms. This book will teach you 10 powerful machine
learning algorithms from scratch.
Developers learn best with a mixture of algorithm descriptions and practical examples.
This book was carefully designed to teach developers about machine learning algorithms. The
structure includes both procedural descriptions of machine learning algorithms and step-by-step
tutorials that show you exactly how to plug-in numbers into the various equations and exactly
what numbers to expect on the other side. This book was written to pull back the curtain
on machine learning algorithms for you so that nothing is hidden. After reading through the
algorithm descriptions and tutorials in this book you will be able to:

1. Understand and explain how the top machine learning algorithms work.

2. Implement algorithm prototypes in your language or tool of choice.

This book is your guided tour to the internals of machine learning algorithms.

1.1 Audience
This book was written for developers. It does not assume a background in statistics, probability
or linear algebra. If you know a little statistics and probability it can help as we will be talking
about concepts such as means, standard deviations and Gaussian distributions. Don’t worry if
you are rusty or unsure, you will have the equations and worked examples to be able to fit it all
together.
This book also does not assume a background in machine learning. It helps if you know
the broad strokes, but the goal of this book is to teach you machine learning algorithms from
scratch. Specifically, we are concerned with the type of machine learning where we build models
in order to make predictions on new data called predictive modeling. Don’t worry if this is
new to you, we will get into the details of the types of machine learning algorithms soon.
Finally, this book does not assume that you know how to code or code well. You can follow
along all of the examples in a spreadsheet. In fact you are strongly encouraged to follow along
in a spreadsheet. If you’re a programmer, you can also port the examples to your favorite
programming language as part of the learning process.

2
1.2. Algorithm Descriptions 3

1.2 Algorithm Descriptions

The description and presentation of algorithms in this book was carefully designed. Each
algorithm is described in terms of three key properties:

1. The representation used by the algorithm in terms of the actual numbers and structure
that could be stored in a file.

2. The procedure used by the algorithm to learn from training data.

3. The procedure used by the algorithm to make predictions given a learned model.

There will be very little mathematics used in this book. Those equations that are included
were included because they are the very best way to get an idea across. Whenever possible,
each equation will also be described textually and a worked example will be provided to show
you exactly how to use it.
Finally, and most importantly, every algorithm described in this book will include a step-by-
step tutorial. This is so that you can see exactly how the learning and prediction procedures
work with real numbers. Each tutorial is provided in sufficient detail to allow you to follow
along in a spreadsheet or in a programming language of your choice. This includes the raw input
data and the output of each equation including all of the gory precision. Nothing is hidden or
held back. You will see it all.

1.3 Book Structure

This book is broken into four parts:

1. Background on machine learning algorithms.

2. Linear machine learning algorithms.

3. Nonlinear machine learning algorithms.

4. Ensemble machine learning algorithms.

Let’s take a closer look at each of the five parts:

1.3.1 Algorithms Background

This part will give you a foundation in machine learning algorithms. It will teach you how all
machine learning algorithms are connected and attempt to solve the same underlying problem.
This will give you the context to be able to understand any machine learning algorithm. You
will discover:

ˆ Terminology used in machine learning when describing data.

ˆ The framework for understanding the problem solved by all machine learning algorithms.

ˆ Important differences between parametric and nonparametric algorithms.

1.3. Book Structure 4

ˆ Contrast between supervised, unsupervised and semi-supervised machine learning prob-

lems.

ˆ Error introduced by bias and variance the trade-off between these concerns.

ˆ Battle in applied machine learning to overcome the problem of overfitting data.

1.3.2 Linear Algorithms

This part will ease you into machine learning algorithms by starting with simpler linear algorithms.
These may be simple algorithms but they are also the important foundation for understanding
the more powerful techniques. You will discover the following linear algorithms:

ˆ Gradient descent optimization procedure that may be used in the heart of many machine
learning algorithms.

ˆ Linear regression for predicting real values with two tutorials to make sure it really sinks
in.

ˆ Logistic regression for classification on problems with two categories.

ˆ Linear discriminant analysis for classification on problems with more than two categories.

1.3.3 Nonlinear Algorithms

This part will introduce more powerful nonlinear machine learning algorithms that build upon
the linear algorithms. These are techniques that make fewer assumptions about your problem
and are able to learn a large variety of problem types. But this power needs to be used carefully
because they can learn too well and overfit your training data. You will discover the following
nonlinear algorithms:

ˆ Classification and regression trees the staple decision tree algorithm.

ˆ Naive Bayes using probability for classification with two tutorials showing you useful ways
this technique can be used.

ˆ K-Nearest Neighbors that do not require any model at all other than your dataset.

ˆ Learning Vector Quantization which extends K-Nearest Neighbors by learning to compress

your training dataset down in size.

ˆ Support vector machines which are perhaps one of the most popular and powerful out of
the box algorithms.
1.4. What This Book is Not 5

1.3.4 Ensemble Algorithms

A powerful and more advanced type of machine learning algorithm are ensemble algorithms.
These are techniques that combine the predictions from multiple models in order to provide
more accurate predictions. In this part you will be introduced to two of the most used ensemble
methods:

ˆ Bagging and Random Forests which are among the most powerful algorithms available.

ˆ Boosting ensemble and the AdaBoost algorithm that successively corrects the predictions
of weaker models.

1.4 What This Book is Not

ˆ This is not a machine learning textbook. We will not be going into the theory behind why
things work or the derivations of equations. This book is about teaching how machine
learning algorithms work, not why they work.

ˆ This is not a machine learning programming book. We will not be designing machine
learning algorithms for production or operational use. All examples in this book are for
demonstration purposes only.

1.5 How To Best Use this Book

This book is intended to be read linearly from one end to the other. Reading this book is not
enough. To make the concepts stick and actually learn machine learning algorithms you need to
work through the tutorials. You will get the most out of this book if you open a spreadsheet
along side the book and work through each tutorial.
Working through the tutorials will give context to the representation, learning and prediction
procedures described for each algorithm. From there, you can translate the ideas to your own
programs and to your usage of these algorithms in practice.
I recommend completing one chapter per day, ideally in the evening at the computer so you
can immediately try out what you have learned. I have intentionally repeated key equations
and descriptions to allow you to pick up where you left off from day to day.

1.6 Summary
It is time to finally understand machine learning. This book is your ticket to machine learning
algorithms. Next up you will build a foundation to understand the underlying problem that all
machine learning algorithms are trying to solve.
 Part II

Background

6
Chapter 2

How To Talk About Data in Machine

Learning

Data plays a big part in machine learning. It is important to understand and use the right
terminology when talking about data. In this chapter you will discover exactly how to describe
and talk about data in machine learning. After reading this chapter you will know:
ˆ Standard data terminology used in general when talking about spreadsheets of data.

ˆ Data terminology used in statistics and the statistical view of machine learning.

ˆ Data terminology used in the computer science perspective of machine learning.

This will greatly help you with understanding machine learning algorithms in general. Let’s
get started.

2.1 Data As you Know It

How do you think about data? Think of a spreadsheet. You have columns, rows, and cells.

Figure 2.1: Data Terminology in Data in Machine Learning.

ˆ Column: A column describes data of a single type. For example, you could have a column
of weights or heights or prices. All the data in one column will have the same scale and
have meaning relative to each other.
ˆ Row: A row describes a single entity or observation and the columns describe properties
about that entity or observation. The more rows you have, the more examples from the
problem domain that you have.

7
Discovering Diverse Content Through
Random Scribd Documents
Miss Carlton shuddered.
"I can't believe that, dear. Because the higher you are, the farther
you have to fall!"
"But you have all that chance to regain control of your plane,"
insisted her niece. "Crashes practically always come on the ground—
it's very rare indeed that two planes crash in the air, even when they
are flying in Army formation."
"How soon do you think you'll get your license?" inquired Miss
Carlton, showing that Linda's words had made no impression at all
upon her. She was anxious to get away now; Spring City was
becoming very hot.
"Any time now," replied the girl, her eyes shining with anticipation.
"I have done all the required solo flying—and more too."
"Solo flying? Do you mean you've been up alone? Without even
Ralph?"
"Yes, of course! And I love it, Aunt Emily! Oh, if you could just try it
once, you'd never be afraid again. It is the most wonderful sensation
—up in the skies, all alone! Free as a bird!" She paused abruptly,
smiling at her own enthusiasm. She did not often talk like this to
anybody, though there was a great deal of poetry in her make-up.
"Well, dear, I'm glad you like it," said Miss Carlton, in a matter-of-
fact tone. "But don't overdo it. And don't go in for any stunts."
Ralph Clavering, who had been making it his habit to come over to
see Linda every evening, now that all his other friends had gone
away, arrived on the porch in time to hear Miss Carlton's admonition.
He was about to say something, for he was very proud of his
successful "acrobatic flying," when he caught Linda's frown of
warning. Of course there was no use of worrying the timid woman,
who was worried enough already.
He sat on the railing, dangling his legs, and carelessly lighting a
cigarette, as if he were very much at home.
"Linda's little 'Pursuit' is a daisy, Miss Carlton," he said. "It really has
a most marvelous motor—and all sorts of safety devices. There's not
a thing for you to worry about.... I wish I had one like it!"
Linda regarded him sympathetically. It was hard luck that his father,
with all his money, refused to buy Ralph a plane! But he had been
promised one the following year—if he graduated from college
without any conditions. Evidently Mr. Clavering was using it as a spur
to his son's ambition, for Ralph had never been keen about his
studies. Good times came first with him; besides, he argued, what
was the use of learning to make money, when his father already had
more than they could spend?
"What are you children going to do this evening?" asked Miss
Carlton, though it was nine o'clock now, and there wouldn't be much
evening left, for Linda insisted upon going to bed early.
"I'd like to map out our trip to Green Falls," the latter replied. "And
then we could show our plan to Lieutenant Kingsberry, and see
where the airports are located along the way, in case we have to
land."
"Why not Taylor?" inquired Ralph, teasingly, for he knew that Linda
did not care much about her instructor.
She gave the boy a withering look.
"Well, then—Redhead? He ought to know. By the way, I never see
you talking to him, Linda!"
"I never get a chance. He's always busy, and besides, you're usually
with me. I guess he's too shy to intrude."
Nevertheless, she decided that she must have one talk with Ted
Mackay before she left the school, to clear up matters that had
never been discussed. All during the next week she watched for her
opportunity, but it did not come until her final day at the school—the
day when she received her license as a private pilot.
Wild with joy at her success, she asked where Ted was, and ran over
to the hangar where he happened to be working. For once, Ralph
was not with her; he had not yet landed the plane he had been
flying.
"Mr. Mackay!" she cried joyously—she was afraid to call him "Ted"
now, for he seemed like such a stranger. "I'm a real pilot! I can fly
my own plane now, wherever I want to go!"
The young man came over solemnly and shook hands with her.
"May I be the first to congratulate you?" he asked.
"Not the first. Lieutenant Kingsberry has done so already. But, of
course, in a way he doesn't count."
"And this is only your beginning, I know!" he said, his blue eyes
sparkling with enthusiasm. "You're going to a ground school in the
fall—as we used to talk about—aren't you?"
"Yes, I hope so." She hesitated, and looked down at the ground,
digging the toe of a dainty slipper—entirely feminine, in spite of her
flyer's costume—into the dust. She felt shy, and embarrassed; it was
so hard to hurt Ted, and yet she didn't dare disobey her father.
"Ted," she said, finally, "could I have just one little talk with you, to
clear things up—before I go away?"
"I've been longing for it," he confessed, eagerly. "But I'd decided
that you were through with me, on account of my actions that night
you got your plane—when I sneaked over to see it. One of the boys
heard it roaring over our heads, and ran out to see where it was
landing. So, when he came back with the news that it was in your
field, I knew it must be yours. When I went over to see it myself—I
—I was hoping you'd come out alone—and we could gloat over it
together! And then all that crowd showed up, and your aunt too—I
was sure it was she—and I just lost my nerve and ran. It looked
pretty queer, I guess."
"No, only why didn't you come to the house first?" she inquired.
"I was afraid the butler would say, 'Miss Carlton is not at home'—the
way the rich young ladies' butlers always do in the novels."
"Only we haven't any butler," laughed Linda.
"Well, you have a strict aunt—and a father that's made of steel!"
"Don't!" cried the girl, in an offended tone. "You mustn't say a word
against my father, or I never will talk to you. But that brings me to
what I wanted to say.... My father has no time for you, on account
of your father. It seems that a man by the same name worked for
him on the ranch in Texas—and was untrustworthy. Could that have
been your father?"
"I'm afraid it was," admitted Ted, sadly.
"So you see why he selected Mr. Taylor to teach me to fly...." Tears
almost came into her eyes, as she saw how sorrowful Ted was
looking. "I think it's absurd, myself," she admitted. "But I suppose
Daddy means it for the best.... I'm—not to be friends with you,
Ted.... And, oh, I'm so sorry!"
"I'm sorry too, Linda," the boy said slowly. "But somehow I never
believed we could be real friends. I'm not like you—I don't believe in
fairy stories."
"What do you mean?"
"I mean that the poor young man, who has a disgrace to live down,
isn't likely to be friends with the rich, beautiful girl—in real life.... So
I guess it's good-by...." He held out his hand.
"Oh, but I'll at least see you again!" she protested. "Tomorrow I'm
going to fly my plane over here and back—all by myself!"
"That's wonderful—I wish I could be here to see you do it," he
answered regretfully. "But unfortunately I am leaving myself
tomorrow. I'm taking a job as salesman for a plane construction
company in Kansas City."
"Congratulations!" cried Linda, pleased at his advancement. "Well,
good luck—and good-by!"
"And, by the way," he added, "I want to thank you for wearing my
poor little flowers at your class-day. I saw you—through the fence. I
was so glad they held the affair out-of-doors!"
"Then they were from you?" she asked, ashamed that she had
forgotten to thank him. "I thought so, but I wasn't sure. I meant to
ask you. They were lovely."
"I am going to give you a card of my firm," said Ted, reaching into
his pocket. "So that you will know where I am, in case you need any
help with your Arrow.... You—you—don't mind?"
"I'll be very thankful to have it," she reassured him. "You know, Ted,
I have an awful lot of confidence in you!"
And, with a final pressure of her hand, he turned to go, and she,
looking about, saw Ralph Clavering walking towards her.
"What's the big idea?" he asked her, when he reached her side, and
Ted had disappeared. "Holding hands with Red?" His tone was
irritable.
"I was just saying good-by," she explained. "He's leaving tomorrow
for a job in Kansas City."
"Flying?"
"Naturally."
"Well, we'll be flying away soon, too," he added, more cheerfully. "I
had a letter from Kit this morning, and she wants us surely at Green
Falls for July thirty-first. It's the Midsummer Ball, and the big event
of the season—socially. She told me to tell you and Miss Carlton to
be sure not to miss it."
"Oh, I'll be ready by Saturday," replied Linda. "Aunt Emily has been
doing all the shopping, so I hardly need to do anything.... By the
way, did Kit give you any gossip about the crowd?"
"Let me see," muttered Ralph, as he took her arm possessively while
they walked across the field, in the hope that Ted Mackay would see
them. "She did have quite a bit to say—but it was mostly about
Maurry."
"Maurice Stetson? What's he been doing?"
"Rushing Kit, evidently. And she seems to like it.... And she said
Harry Smith has a life-guard's job, and is spending all his spare time
with Lou."
"I haven't heard from Lou in ages," remarked Linda. "But I guess it's
partly my fault. I haven't had time to answer her letters." Then,
changing the subject, as they came out to the road where Linda's
car was parked, "You're going to fly up with me in the 'Pursuit,'
aren't you, Ralph?"
"Surest thing! We'll fly everywhere together—from now on. Just like
Mr. and Mrs. Lindy!"
"Only we won't!" she answered abruptly, laughing at him.
As they stepped up to the roadster, they almost fell over a man who
came out from a shabby coupé in front of theirs. He had evidently
been leaning over, fixing something.
"Want any help?" asked Ralph, though Linda knew he hadn't the
slightest idea of giving any.
"No, thanks," muttered the man, without looking up. "Engine
trouble."
"Engine trouble!" repeated Linda, sympathetically. Then, turning to
Ralph. "Suppose something like that should happen to us—on the
way to Green Falls!"
"Well, it won't!" replied Ralph reassuringly. "The motor's just about
perfect in that little plane of yours! No—but I tell you what, Linda,
you better bring your gun along. That crazy sister of mine expects
me to bring her pearls up for the Midsummer Ball!"
"Real pearls—at a summer resort!" cried Linda, as she slipped the
key into her lock, and started her engine. "She's taking an awful
chance!"
"That's what I think. But of course they're insured. And so long as
she's succeeded in getting Dad's permission, it's not my business to
stop her.... By the way, it's a fancy-dress affair. What sort of costume
will you wear?"
"I don't know. I guess I'll leave it to Aunt Emily."
But when she got back home, she forgot all about pearls and
dresses and mid-summer balls. Nothing mattered to her, but the
glorious fact that at last she was a real flyer!
 Chapter VII
The Flight to Green Falls

The first thing that Linda thought of when she opened her eyes the
following morning was the glorious fact that she was now a real
pilot. She could take her plane anywhere—to Green Falls, to her
father's ranch in Texas, wherever she wanted to go—and nobody
could stop her. The freedom of the world and of the skies was hers.
But she had no intention of taking it any farther than the Spring City
Flying School that day. She would spend the morning there,
watching one of the licensed mechanics give it a thorough
inspection, in readiness for the flight to Green Falls on the following
day.
She wished that it might be Ted Mackay who would go over the
plane. She had such confidence in his knowledge, his thoroughness.
Besides, it would be fun to spend the morning with him, asking him
questions, and talking things over.
Naturally, that was impossible. When Linda reached the field she
found that Ted already had gone, and a number of changes had
been made. H. B. Taylor was now first-ranking instructor, and the
young man who had been acting as office boy, or orderly, or
whatever they chose to call him, had passed his course and was
promoted to the rank of instructor. Another man took his place—an
older man this time, and Linda thought probably it was the poor
fellow who had been having engine trouble with his shabby coupé
the preceding day. Everything seemed different, and Linda was
somehow glad that she was leaving. The place would never be the
same to her without Ted Mackay.
About noon she received the mechanic's O.K. upon her plane, and
flew home in time for lunch. Her aunt had finished packing, and was
as excited as a child about going to Green Falls, and again taking up
their customary social life among their friends.
"I have bought a new flying suit for you, dear," she said to her niece,
as the girl entered the library. "Unwrap it and see how you like it."
Linda eagerly unfastened the strings and lifted out a pair of white
flannel knickers, with a jaunty blue sweater and helmet of knitted
silk, just the color of her eyes. The whole costume was charming,
and a lovely change from the dark riding breeches she had been
using for flying.
"It's perfect, Aunt Emily!" she cried, realizing for the first time that
she had never cared for what she was now wearing. "And it was so
sweet of you to think of getting it for me!"
"I never could see why girls have to look masculine," replied her
aunt. "Of course I can understand that skirts are impractical, but
they make these suits so pretty now-a-days. And I want you to look
nice the very first minute you arrive at Green Falls. First impressions
are always so important and there is sure to be a crowd there to
greet you."
Linda was only too delighted to wear it the next day, which dawned
clear and warm for her flight. Miss Carlton left early in the morning,
by train, so that she would be at Green Falls in plenty of time to
welcome the flyers.
Ralph came over for Linda about half-past nine. Carrying their lunch,
the young people started on their first real adventure in the air.
The young man, too, wore a new suit of spotless white flannel, and,
as they walked, tall and slender and straight, they made perhaps the
best-looking pair of flyers in America. But neither was conscious of
that; both were too much excited about their first trip in the air to
give even a passing thought to their appearances.
"Are you sure that you have the precious necklace?" asked Linda, as
they made their way across the field in back of her house.
"Yes, indeed," answered Ralph. "I went to the safe-deposit vault this
morning to get it. That was one reason why I didn't want to start
early. I had to wait for the bank to open."
"Kit would be horribly disappointed if we didn't bring it," returned
Linda. "I honestly think she loves those pearls as much as I do my
'Pursuit'!"
"Queer taste," remarked the boy. "If I had them, I'd sell them and
buy a biplane!"
"Of course you would," said Linda approvingly. "Even if you do insist
upon talking baby-talk!"
"Baby-talk?"
"Certainly. 'Buy a biplane'—sounds like 'Bye, Bye, Baby,' doesn't it?"
Ralph smiled, but they both forgot immediately what they were
saying, for they were beside the plane now, ready to start on their
flight. Linda was not at all nervous about the journey, only thrilled
and happy. She climbed into the cockpit with the same assurance
that she entered her car, and her take-off was just as easy, just as
natural. It seemed now as if she piloted the biplane by instinct; with
the sureness of a bird it rose into the air to a gradual height of
fifteen hundred feet. For she had been cautioned again and again
that there was safety in height.
They flew along without any attempt at conversation, for it was
difficult to hear above the roar of the motor. But Linda was so happy
that she hummed softly to herself, and most of the time she was
smiling. Ralph, with a map in his lap, kept a close watch on the
compass.
For some time they did not see any other planes in the sky, and then
at last one came into view. As it drew closer, it occurred to Linda to
wonder whether she was being followed.
"Who do you suppose that is?" shouted Ralph, above the noise of
the motor.
"I think it's somebody from our school—maybe Taylor," she replied.
"Perhaps Dad ordered them to follow us—for safety—or maybe it
was Ted Mackay's idea."
As the plane drifted off to one side, they thought no more about the
matter. But it was noon now; the sun stood high overhead, and both
of the young people were astonished to find how hungry they were.
"I want to try a couple of stunts before we eat," Linda told Ralph.
"You're game, aren't you?"
"Surest thing!" replied the boy, with delight. "We've got plenty of
height—and a spectator too, for that matter." The other plane had
just come back into sight.
Linda's eyes were shining with excitement, yet inside she was
perfectly cool. Hadn't she made inside loops and Immelman turns
often at school, and didn't she know exactly what to do? With
perfect poise, she swung the plane into a loop, and completed it
without any difficulty. Pleased with her success, she tried it again
and again.
"You must think you're Laura Ingalls!" shouted Ralph, catching his
breath. "Trying to beat her record?"
"Hardly," smiled Linda, for the famous aviatrix he mentioned held the
record at that time with nine hundred and eighty consecutive inside
loops, at a speed of four and a half loops a minute.
The plane was righted now, but Linda suddenly noticed that Ralph
was acting awfully queer, hanging over the side, and hunting
frantically in the pockets of the sweater which he had put over the
seat. She believed he must be ill; certainly his face was ghastly
white.
"Ralph!" she cried, fearfully. "What's the matter?"
"I've lost the necklace!" he screamed in terror. "Must have fallen out
of my pocket!"
"Oh!" wailed Linda, aghast at the meaning of his words. "Are you
sure?"
"Positive!"
"Then we'll land immediately. We're over a field, so we ought to be
able to find it. Now—keep your eye on the compass!"
Gradually, and with easy skill, she turned the biplane into the wind
and descended, finally coming down into a large flat field, evidently
a pasture ground for some horses. Ralph was the first to jump out.
"We went a little south to land," he said, "so it must have dropped
up there."
"Was it in a box?" questioned Linda.
"Yes, fortunately. A white velvet box, inside a larger pasteboard one,
with three rubber bands around it. That ought to make it easier to
find."
Linda, however, had her doubts; the field was so big! Besides, what
proof had Ralph that he had lost it at that particular minute—when
she was making her loops. She remembered that he had taken off
his sweater an hour ago, when he felt too warm, and had carelessly
hung it over the side, forgetful of the precious box in its pocket. That
was the trouble with being so rich! Many times she had noticed how
heedless both Kitty and Ralph were about valuables.
They walked silently across the field, their eyes on the ground, their
minds filled with remorse. Ten minutes passed, and they had not
found it.
"Let's go back and eat our lunch," suggested Ralph, consulting his
watch. "It's almost one o'clock, and we'll feel better if we eat. After
all, we have plenty of time—Green Falls is only about twenty miles
farther. We could search all afternoon, if necessary."
"Yes, only Aunt Emily would nearly die of anxiety. She'd be sure we
had been killed, if we didn't arrive before supper."
They went back to the plane and took out the dainty lunch which
Miss Carlton's cook had packed that morning for them. But, hungry
though they were, the meal was not the pleasant picnic they had
been hoping for. Both were too unhappy to enjoy what they were
eating.
Presently the noise of a motor overhead attracted their attention,
and, looking up, they saw a plane in descent. When it was low
enough to identify, they knew that it was the one that had been
following them.
"It's the 'Waco' from our school!" cried Linda. "I recognize it now. He
must think we're in trouble. I wonder who's piloting?"
The plane made a rather poor landing at the far end of the field,
perhaps half a mile away. They could distinguish a man getting out
of the cockpit, but of course at that distance they could not identify
him. However, he seemed to be coming slowly towards them.
As he advanced nearer and nearer Linda noticed that he wore an
ordinary suit of clothing—not a flyer's uniform, and he kept his hand
in his pocket. But she still did not recognize him—unless he was that
new man the school had taken on the preceding day. Once he
stooped over, as if he were picking something up, and Linda's heart
beat wildly with hope. Could it be that he had found the necklace?
Apparently, though, it was only a plant that he had pulled up by the
roots, for when he straightened himself, he seemed to be examining
its leaves.
"In trouble?" he shouted, as soon as he was within hearing distance.
Ralph jumped up and ran towards him, shaking his head in the
negative.
"No trouble with the plane," he replied. "But we've lost a little box—
with a necklace in it. You haven't seen it, have you?"
"Why, yes," answered the man slowly, "I did pick up a box." And he
put his other hand in his pocket, and drew out the very article.
Fortunately it had not been broken; even the rubber bands were still
tightly around it. He handed it to Ralph.
"Oh, thank you a thousand times!" cried Linda, too relieved to
believe her eyes. "The necklace was a graduation present to this
man's sister, and she values it very highly!"
"Well, if that's all, I'll be off," said the man, as he watched Ralph put
the box into his pocket.
"No, I must reward you," insisted the boy, taking out a twenty-dollar
bill. "And by the way, you're from the Spring City Flying School,
aren't you? We recognized the plane."
The other nodded, and seemed in a hurry to be off. Already he was
twenty feet away.
"It was awfully nice of you to follow us, and look after us," called
Linda, "but really we don't need protection. We're getting along
finely!"
But the man was running now, and could hardly have heard what
Linda was saying. In a couple of minutes they heard the motor start,
and with a clumsy take-off, the plane ascended.
"A queer cuss," remarked Ralph. "And I can't see that he's much of a
flyer. You and I are both better—by a long shot.... But anyhow,
we've got the necklace!" He put his arms around Linda and hugged
her, and she was too happy to protest. What a miracle it was to have
found it!
"That will teach me a lesson," said Ralph, as he helped Linda gather
up the lunch. "I'm going to be more careful now. I've put the
necklace in my most inside pocket!"
"And I'm not going in for any more acrobatics for a while," added
Linda.
They climbed into the cockpit, and started the motor without
wasting any more time. Half an hour later they made a graceful
landing at Green Falls' Airport, for a group of a hundred spectators
to witness and admire.
 Chapter VIII
The Robbery

"Let's don't say anything about our little mishap," whispered Linda,
as the flying couple got out of their plane. "For one thing, I'd just as
soon not boast about stunts in front of Aunt Emily. She would be
worried all the more."
"And I'm not any too proud of the fact that I was so careless about a
valuable necklace," returned Ralph. "So we'll keep it our secret."
There was no time for further words. Everybody rushed at them,
shouting joyous welcomes. Louise was the first to kiss Linda—then
all the others, and finally her aunt.
"Thank Heaven you're safe!" cried the latter. "I couldn't eat a bite of
lunch, I was so uneasy."
"Of course we're safe," assured Ralph. "And maybe if we'd come by
motor, we should have had an accident. There was a big smash-up—
two automobiles—outside of Spring City this morning."
"Isn't the air up here wonderful!" exclaimed Miss Carlton. "After that
stuffy town of ours!"
"I think the airport is wonderful," replied Linda, "for so small a place.
But as for the air—well, don't forget Auntie dear, that Ralph and I
have been having marvelous air—up in the skies!"
"Hope you didn't give him the air," remarked Maurice Stetson,
solemnly.
Kitty Clavering gave the young man a withering look, and inquired of
the flyers when they might hope for rides. "Oh, I don't mean today,"
she added, "for I know you must both be nearly dead."
"Not a bit of it!" denied Linda, who still looked as fresh as a flower in
her becoming blue and white suit. "But it's supposed to be wise to
have a mechanic go over your plane each time you fly. Just a
precaution, you see."
"A very good rule to follow," commented Miss Carlton. "Now
everybody get into their cars, and we'll go over to our bungalow for
some ginger-ale and sandwiches."
"Just a moment, please!" interrupted a voice at her elbow, and
everyone turned to see a newspaper man with a camera. "Pictures,
please!"
Linda and Ralph smilingly agreed, and their friends stepped aside.
Then they all piled into the three machines that were waiting for
them; while the strangers who had been watching commented on
the beautiful biplane, and the handsome couple who had been flying
it, and wondered whether they were married.
"Did you bring my necklace, Ralph?" asked Kitty Clavering, as he got
into her roadster with her and Maurice.
"Surest thing!" he replied, as if nothing at all had happened on the
way. Reaching into his pocket, he pulled out the pasteboard box,
with the French jeweler's name engraved on the lid.
"Thanks a lot," she replied. "Maurry, you take care of it till we get
home, so long as you're sitting in the middle. Mind you don't lose it!
I think as much of that as Linda does of her plane."
"But not as much of it as you do of me?" asked the youth, flippantly.
"A thousand times more! Like the old question people always ask
married men: 'If your mother and your wife were drowning, which
one would you save?' Well, if you and the necklace were drowning,
I'd go after my necklace!"
"Righto. Necklaces, no matter how valuable, have never been known
to swim. I do."
It was only a five minute ride from the airport to Miss Carlton's
bungalow, so Kitty waited until they had all gone inside the pleasant
living-room to open her box, and gaze at her beloved treasure once
more.
"I'm dying to see it again," she said, as she took the box from
Maurice's hand. "If I had my way, I wouldn't keep it in a safe-deposit
vault. I like it where I can look at it."
She took off the rubber bands and opened the box, displaying the
velvet case inside. But when she unfastened the clasp, her
expression of delight changed abruptly to one of horror. The case
was empty!
Her exclamation of distress was pitiful to hear. Her dearest
possession—gone!
"Ralph!" she cried with torturing accusation. "Ralph! Are you teasing
me?"
Her brother's face became ghastly white.
"What—what's wrong—Kit?" he stammered.
"My necklace! Oh, what has happened?" She burst out crying.
Everybody crowded around and gazed in consternation at the empty
box, looking questioningly at Ralph, to see whether it could possibly
be intended as a joke. But he did not need to tell them of his
innocence; he looked almost as stricken as his sister. He knew now
that it had been stolen by the man who pretended to be a pilot! And
he had actually made twenty dollars out of Ralph besides, for the
transaction! What fools they had been, never to open the box!
"It's all my fault!" cried Linda, contritely. "My silly, foolish,
childishness, for wanting to show off!"
Nobody of course had any idea what she was talking about—nobody
except Ralph.
"No! No! It was mine!" he protested. "My carelessness!"
"Then you both knew!" exclaimed Kitty, raising her head, which she
had buried on Linda's shoulder while she sobbed. "Oh, how cruel,
not to prepare me!"
"On my honor, we didn't!" averred Ralph, and from the look on his
face, his sister knew that he was telling the truth.
"Explain what you meant, then," she commanded.
"Let me tell you," put in Linda. "But sit down, Kit dear. You're liable
to faint.... You see, we were robbed, and too foolish to suspect it.
We even paid the robber twenty dollars for doing the job."
"So you said," Kitty remarked, impatiently. "Do you mean that you
saw somebody take it—right under your eyes?" She had dropped
down on the couch, and her pale little face was pitiful to see. The
tears still ran down her cheeks, washing tiny rivers through the
powder. Luckily she was not a girl who used rouge, or she would
have looked ridiculous. As it was, she gave the appearance of a very
unhappy child.
"Exactly!" explained Linda. "Or rather, we might have, if we had had
sense enough to realize it. I wanted to try a couple of loops, and we
started quite high, but by the time we had finished, we were over an
open field. It was then that Ralph suddenly realized that the box had
dropped out of his pocket when the plane was on its side. So we
decided to land, and search the field."
"And somebody had already picked it up?" demanded Dot, excitedly.
"No. Another airplane—I had noticed it before—landed soon after we
came down. The pilot walked over and asked us if we were in
trouble."
"And you stupids told him all about the fifty-thousand-dollar
necklace!" cried Louise, in disgust.
"No, we didn't! We were smart enough to know that wouldn't be
wise. We thought we knew him, though—we had seen him at the
Spring City Flying School. But we did tell him we had lost a necklace,
and he said he had picked something up. As a matter of fact, we
had noticed him stoop over."
"And you took it and thanked him, and never looked inside!" cried
Kitty.
"I'm afraid you're right," admitted Ralph. "We thought he was a
friend, following us for our protection, at the orders of the school."
"Well, then, why was he following you?" demanded Kitty,
incredulously.
"He must have overheard us talking about the necklace," answered
Linda slowly, for she was trying to think the thing out. "Yes—that is
what I believe he was doing all the time, Ralph. Now I remember—
the day we got our licenses!"
"You mean you went around the school shouting the news that you
were carrying pearls to Green Falls in an airplane?" asked the
unhappy girl.
"Of course not! Only the men at the bank—the safe-deposit vault—
really knew about it. And of course they're absolutely trustworthy!
Except maybe this one man—who was fixing his car outside the
aviation field. We never thought he was listening—why we couldn't
even see him!"
"Children," interrupted Miss Carlton, who had been patiently waiting
to serve the refreshments, "wouldn't you all feel better if you ate
something? Then we can discuss what are the best steps to take to
capture the thief."
They agreed, but Linda and Ralph and Kitty were all extremely
nervous; they hated to lose any time. Ralph decided to telephone to
a lawyer at once in Spring City, to put expert detectives on the job,
and to get in touch with the Flying School.
"Lucky the necklace was insured," remarked Maurice Stetson, as he
drank his ginger-ale.
"Yes, but Dad will never get me another!" moaned Kitty,
disconsolately. "He'll say I was careless, and invest the insurance in
bonds, to be kept in trust till I'm older—or something like that." She
started to cry afresh. "And I only wore the necklace twice—at
graduation and at the class dance!"
Linda watched her sorrow with more than sympathy—with remorse.
It was her fault, she was sure! Of course she couldn't imagine caring
so much for a pearl necklace, when such lovely imitations were
made, but it wasn't her place to judge. Kitty probably wouldn't
understand why she loved her Arrow so much.
Slowly, painfully, she came to her decision. She rose and went over
to the couch where Kitty was sitting, and crowded in between the
latter and Dot.
"It's my fault, Kit," she said, "and of course I can't pay for it—but I
can help. I'm—I'm—going to sell my airplane, and—give you the
money. Then you can start buying a new one—a couple of pearls at
a time."
Kitty squeezed her hand affectionately.
"You're a dear, Linda, but I couldn't possibly let you do that. Besides,
it was really Ralph's fault."
"Of course it was!" put in the young man, returning from making his
telephone call. "But we're going to catch that thief!" he announced,
with conviction. "I've just been talking with Lieutenant Kingsberry at
the field, and he says that fellow didn't even have a license, that
they only took him on temporarily, as sort of errand boy. And he
deliberately stole that plane!"
"I thought he was about the poorest pilot I ever saw!" cried Linda,
jumping up excitedly at this piece of news. "He'll probably crash,
sooner or later.... Ralph!" Her eyes were shining with inspiration....
"Let's go out after him—ourselves!"
"Lieutenant Kingsberry is broadcasting the news all over—to all the
airports," replied the young man. "Everybody will be watching for
him. Do you think there would be any use in our going?"
"Yes! Yes! We might be just the ones to spot him! Oh, come on!"
"But haven't you had enough flying for today, Linda?" inquired Miss
Carlton, anxiously.
"We won't go far, Auntie dear," answered the girl. "Just around to
the nearest airports, and see if anybody has any information. The
practice of landing and taking-off again will be good for us both....
And you needn't worry one bit!... Now, who'll drive us over to our
'Pursuit'?"
"'Pursuit' is right," remarked Maurice. "Your plane has the right
name, Linda!"
Louise immediately offered her services, and in less than five
minutes the young pilots had washed their faces and were ready to
start. Ten minutes later they climbed into the cockpit on the runway
of the airport, and, this time with Ralph at the controls, they took off
for the nearest airport.
Ralph was delighted to be piloting a plane again, and in his
enthusiasm he almost forgot the seriousness of his mission. A king
of the air, he thought, and his lips were smiling. But Linda could not
forget so easily.
Like most young men, he loved going fast, and as soon as he was
high enough, he let the plane out to her maximum speed. Over the
clouds they sailed, at a rate of seventy miles an hour, yet they did
not seem to be traveling fast. Linda had no sense of danger, yet it
was the first flight she had ever made that she did not thoroughly
enjoy, for, unlike Ralph, she could not for one moment forget Kitty's
tragedy.
Twenty minutes, however, was all that was needed to reach their
first port, and Ralph, not quite so skilled or so careful as Linda,
made, nevertheless a pretty landing. It was a large field, evidently
designed for amateur sport flyers, and there were a number of
licensed mechanics in readiness to greet new arrivals.
Ralph lost no time in telling his story to the first man who came
forward. Had they any information so far? he inquired.
"Only of a wreck about fifteen miles away," replied the latter. "That
may be your man—if, as you say, he is not an experienced pilot."
"Can you give us directions?" put in Linda excitedly.
"Certainly," replied the other, taking a map from his pocket, and
indicating the position of the wreck. "We've already sent a doctor
and a nurse—and telephoned for an ambulance." Marking the spot,
he handed the map to Ralph.
Jumping into the plane at once, Linda took control, for she felt surer
of herself than of her companion in an emergency. The boy was so
absent-minded, so likely to forget things in his excitement.
Their destination was a field again, but not a large one, this time,
and already a small crowd, gathered from passing automobiles, had
collected. Here landing was not so easy as in the airports designed
for that very purpose. But the girl knew just what she was doing,
and she handled the situation with a dexterity that would have
brought credit to a far more experienced pilot.
Over against an embankment, its wings smashed to pieces, a plane
was lying on its side, mutely testifying to the truth of the mechanic's
statement.
"There's the wreck!" cried Ralph, as he and Linda stepped on the
ground. "Do you think it's the Waco?"
Grabbing her companion's arm, Linda ran forward eagerly. When
they were within fifty yards of it, she knew that it was the very plane
they were seeking.
"It is! Oh, Ralph! Even the license number—so I'm sure! Remember?
Look! Do you suppose that man was killed?"
"Would serve him right!" muttered the boy, resentfully. "Stealing a
necklace, and crashing a plane that wasn't his! But let's go over and
have a peep at him—there's the ambulance."
The crowd, which was still gathering, although the field was in an
isolated spot, was being held back by a policeman, for the
ambulance was ready to start. Ralph dashed forward, anxious to get
a look at the thief before it departed.
"Not that we could claim the necklace now," he explained to Linda,
whose arm he was holding, "for we haven't any proofs of our
ownership. But at least we could warn the cop to look out for it."
"Back! Back!" shouted the officer, for the driver was tooting his horn.
"Oh, please wait a minute!" begged Linda. "Please let me see the
man who is inside!"
The policeman regarded the girl doubtfully, but she was so eager in
her pleading that he thought perhaps she had a good reason.
Perhaps the man inside the ambulance meant something to her; he
decided to grant her request.
"Take a look, miss," he agreed. "But be quick about it."
Stepping ahead of Ralph, Linda climbed upon the back step of the
car, and peered anxiously into it, past the white-clad interne, to the
unconscious figure on the stretcher. Suddenly she started violently,
and clung to the door of the ambulance for support. It was
incredible, impossible! Her knees shook, her hands fell to her side,
and she swayed backward in a faint. In an instant Ralph's arms were
around her; he carried her out of the crowd.
The unconscious man in the ambulance was none other than Ted
Mackay!
 Chapter IX
Suspicions

Someone from the crowd handed Ralph a cup filled with water, and
before they had gone half a dozen steps, Linda had recovered
consciousness. She dropped down to the ground and stared
questioningly about her.
"What was it, my dear?" asked Ralph gently, as he held the water to
her lips. "Was the man hurt so horribly?"
"No—it wasn't that," replied Linda slowly, remembering all that had
happened. "It was just—oh, Ralph! I hate to tell you!"
"Please tell me, Linda," he begged.
She looked about her for a moment. The ambulance had gone, and
the crowd, seeing that the girl was all right, began to withdraw,
some to examine the shattered plane, others to go back to their cars
parked along the roadside. There was nobody listening now, so she
decided to answer Ralph's question.
"It wasn't our thief at all," she said. "It was—Ted Mackay."
"Ted Mackay?" he repeated, as if he could not believe his ears.
"Yes."
"Then how do you explain it? That couldn't have been Mackay we
met on that field—Mackay disguised, or anything?"
"No. He wasn't tall enough. And he had black hair. Oh, Ralph, I'm
sure of that!"
"Then how do you explain it?"
"I don't explain it," she said weakly.
He said nothing more, but he knew that she was not only terribly
disappointed in not being able to trace the necklace, but that she
was entertaining grave doubts about Mackay's part in the whole
miserable affair. Were he and this thief in partnership, playing a
wicked game, and had Ted hired the man because he would not let
them know his part in the robbery?
But there was no use talking about that now, for Ralph realized that
Linda was almost ready to collapse. Drawing her arm through his, he
led her silently back to the Pursuit, and put her into the cockpit,
indicating that he would pilot them back to Green Falls. Not a word
did she utter during the entire flight homeward; she drooped
listlessly back in her seat, with an expression of disappointment and
despair on her face. How she wished that she had not come!
No one was waiting for them at the airport, so they took a taxi to
Miss Carlton's bungalow. They found the latter on the porch, with
only Kitty and Maurice beside her.
"Any news?" demanded the girl, jumping out of the hammock, and
rushing down the steps before the taxi had been stopped.
"Some news, yes," replied Linda, while Ralph paid the driver. "But
I'm afraid it doesn't mean much. Ralph will tell you all about it."
But the young man was not willing to tell his story until he had
asked Miss Carlton to take care of Linda.
"She fainted at the field," he explained. "The hot sun and the crowd,
I expect." He did not want to speak of Ted Mackay before her, while
she felt so ill. "So if you'll take Linda up to her room, Miss Carlton,
I'll tell Kitty what I know—and tell you later."
The words aroused Linda's aunt immediately, and she lost interest in
the necklace temporarily. What were a few pearls, anyway, in
comparison to her precious girl? She hurried her off to bed, and
Ralph turned to Kitty and Maurice.
"You see it was this way," he began, and Kitty stamped her foot in
exasperation.
"Don't be so slow, Ralph!" she commanded.
"Why, here comes Linda's father!" interrupted Maurice, as another
taxi stopped at the bungalow. "What do you think of that?"
Kitty looked vexed. Another interruption! But Ralph was already on
his feet, greeting him, and explaining the absence of Linda and her
aunt.
"And I was just going to tell Kitty about our pursuit of the thief," he
added, "so if you care to hear the story, Mr. Carlton, perhaps you will
sit here with us?"
The older man was glad to comply with the request. Naturally,
anything that was connected with Linda's first flights was of
paramount interest to him.
So, in spite of Kitty's impatience, her brother began the story with
the day that he and Linda received their licenses, and ended it with
the latter's identification of Ted Mackay, unconscious on the stretcher
in the ambulance.
"Mackay!" repeated Mr. Carlton, shaking his head knowingly. "So he
was the brains of the crime!"
"I'm afraid so, sir. And I'm afraid that's what made Linda faint."
"Of course it is! She believed in that fellow. But I warned her not to
trust him. You see his father worked for me out in Texas and he's an
unprincipled fellow. Stole from everybody—not only myself, but even
the rest of the help. And got into a mix-up with some Mexicans, and
turned them against me.... Yes, it must run in the family. The father
may even be in on this necklace robbery. I don't know where he is
now."
"That explains a good deal," mused Ralph, who had been listening
thoughtfully. "I never did like Ted Mackay." He would not admit even
to himself that jealousy was the main reason for this dislike.
"Besides, Linda probably told him about the Midsummer Ball, and
our carrying Kit's necklace to Green Falls. I thought it was funny if
that other chap caught on so quickly."
"Did Linda see much of Mackay while she was at the school?" her
father asked, sharply.
"I can't say that, although I wasn't always with her. Towards the end
of our time we did so much solo flying, that when I was up in the air
I didn't know where she was, although she was usually up too—in
another plane. But one time I did find her in a pretty intimate
conversation—and that was right before we left. She probably told
him then."
"Too bad! Too bad!" muttered Mr. Carlton, regretfully. He was
wishing now that he had sent Linda to some other flying school.
At this moment, Miss Carlton, having left Linda asleep in her room,
came out on the porch to see her young guests. She showed no
surprise at finding her brother; for fifteen years she had been
accustomed to having him drop in when least expected, without a
moment's notice.
"Well, Tom," was all that she said, as she presented her cheek for
his brotherly kiss. "I suppose these children have told you the
news."
"Yes, and if you don't mind, Emily, I think I'll drive over with them to
see Mr. Clavering," he added, for the young people had all risen, and
were showing signs of departure. "I'd like to have a talk with him—
at least if you'll excuse me."
"Certainly," replied his sister. "And will you be back in time for
dinner?"
"I'll come home in half an hour," stated her brother, laughing, for he
always teased her about her insistence upon his promptness.
It was natural that he should want to meet Kitty's parents, that he
might at least offer to do his part in trying to recover or make good
the girl's loss.
But Mr. Clavering seemed to take the matter almost lightly.
"Of course it's too bad," he said, "but as long as it is only a theft,
and not an injury to one of the children, I think it's foolish to worry.
And, after all, we may get insurance."
"May get insurance?" repeated Mr. Carlton, frowning. "Why shouldn't
you get it? I thought that was what insurance was for!"
"I'm afraid ordinary insurance will not cover travel by air," explained
the other man.
At these words his daughter burst into tears. Her last hope was
gone!
"I never thought of that," said Mr. Carlton, gravely. "That makes a
difference.... Well, Mr. Clavering, in that case, I guess we had better
divide the obligation. I'll raise my twenty-five thousand—the
necklace was worth fifty, I understand—as soon as I can."
"You'll do nothing of the sort!" protested the other, firmly. "Your
daughter was not the least bit at fault. It was natural for her to try
her stunts—she wouldn't be human if she didn't! I put the whole
blame upon Ralph."
"No! No——"
"Yes, yes! I won't hear anything else. But we'll wait and give the
detectives time. If we have caught the leader, as you and Ralph
think, it ought to be an easy matter to locate the accomplice. At
least, provided Mackay doesn't die."
"That's true!" exclaimed Ralph. "I never thought of that. We better
get over to the hospital to see him as soon as possible."
"How about tomorrow morning?" suggested Mr. Carlton. "I'd like to
go with you, my boy—I've had some experience in dealing with
criminals, ever since the episode with Mackay's father."
"I'll be delighted to have you," replied Ralph. "And in the meantime,
I'll call my detective and put him on the other man's trail."
So while Linda slept peacefully at home, her father and her best boy
friend made plans to verify their suspicions against Ted Mackay, lying
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com