MachineLearningSparkML.pptx
- 2. Machine Learning with Spark MLlib Manuel Martín Márquez Antonio Romero Marin Joeri Hermans Hadoop Tutorials
- 3. Machine Learning (ML) • ML is a branch of artificial intelligence: • Uses computing based systems to make sense out of data • Extracting patterns, fitting data to functions, classifying data, etc • ML systems can learn and improve • With historical data, time and experience • Bridges theoretical computer science and real noise data. 3
- 5. Supervised and Unsupervised Learning • Unsupervised Learning • There are not predefined and known set of outcomes • Look for hidden patterns and relations in the data • A typical example: Clustering 5 0.0 0.5 1.0 1.5 2.0 2.5 2 4 6 Petal.Length Petal.Width irisCluster$cluster 1 2 3
- 6. Supervised and Unsupervised Learning • Supervised Learning • For every example in the data there is always a predefined outcome • Models the relations between a set of descriptive features and a target (Fits data to a function) • 2 groups of problems: • Classification • Regression 6
- 7. Supervised Learning • Classification • Predicts which class a given sample of data (sample of descriptive features) is part of (discrete value). • Regression • Predicts continuous values. 7 100.0 0.0 0.0 0.0 96.0 4.0 4.0 0.0 96.0 setosa versicolor virginica setosa versicolor virginica Actual Predicted 0 25 50 75 100 Percent
- 8. Machine Learning as a Process Define Objectives Data Preparation Model Building Model Evaluation Model Deployment 8 - Define measurable and quantifiable goals - Use this stage to learn about the problem - Normalization - Transformation - Missing Values - Outliers - Data Splitting - Features Engineering - Estimating Performance - Evaluation and Model Selection - Study models accuracy - Work better than the naïve approach or previous system - Do the results make sense in the context of the problem
- 9. ML as a Process: Data Preparation 9 • Needed for several reasons • Some Models have strict data requirements • Scale of the data, data point intervals, etc • Some characteristics of the data may impact dramatically on the model performance • Time on data preparation should not be underestimated • Missing Values • Error Values • Different Scales • Dimensionality • Types Problems • Many others Raw Data • Scaling • Centering • Skewness • Outliers • Missing Values • Errors Data Transfor mation Modeling phase Data Ready
- 10. ML as a Process: Feature engineering 10 • Determine the predictors (features) to be used is one of the most critical questions • Some times we need to add predictors • Reduce Number: • Fewer predictors more interpretable model and less costly • Most of the models are affected by high dimensionality, specially for non-informative predictors • Binning predictors Wrappers Multiple models adding and removing parameter Algorithms that use models as input and performance as output Genetics Algorithms Filters Evaluate the relevance of the predictor Based normally on correlations
- 11. ML as a Process: Model Building 11 • Data Splitting • Allocate data to different tasks • model training • performance evaluation • Define Training, Validation and Test sets • Feature Selection (Review the decision made previously) • Estimating Performance • Visualization of results – discovery interesting areas of the problem space • Statistics and performance measures • Evaluation and Model selection • The ‘no free lunch’ theorem no a priory assumptions can be made • Avoid use of favorite models if NEEDED
- 12. • Thank you 11/8/2022 Document reference 12
Editor's Notes
- #6: ML methods fall into two learning types Unsupervised Suppose you want to segment your customers into general categories of people with similar buying patterns.
- #7: More formally fits data to a function or a function approximation
- #8: More formally fits data to a function or a function approximation
- #9: More formally fits data to a function or a function Adding Roles
- #10: Add Examples
- #11: Random Forest (tree based) MARS and LASSO internally perform predictor selection Add Examples
- #12: there is no one single model that will works better than any other a priory