Download as PDF, PPTX
Uploaded bySigOpt
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning / AI Models
The document discusses the challenges of tuning deep learning models and compares various hyperparameter tuning methods, emphasizing the advantages of Bayesian global optimization. It highlights how optimal learning can improve the efficiency of parameter configuration and model performance in AI applications, using case studies with CNNs and other frameworks. SigOpt is presented as a solution to optimize training processes and reduce error rates without expert time wasted on manual tuning.
More Related Content
Similar to MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning / AI Models
MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning / AI Models
- 1. BAYESIAN GLOBAL OPTIMIZATION UsingOptimal Learning to Deep Learning / AI Models Scott Clark [email protected]
- 2. OUTLINE 1. Why isTuning Models Hard? 2. Comparison of Tuning Methods 3. Bayesian Global Optimization 4. Deep Learning Examples
- 3. Deep Learning /AI is extremely powerful Tuning these systems is extremely non-intuitive
- 4. https://www.quora.com/What-is-the-most-important-unresolved-problem-in-machine-learning-3 What is themost important unresolved problem in machine learning? “...we still don't really know why some configurations of deep neural networks work in some case and not others, let alone having a more or less automatic approach to determining the architectures and the hyperparameters.” Xavier Amatriain, VP Engineering at Quora (former Director of Research at Netflix)
- 5.
- 6. TUNABLE PARAMETERS INDEEP LEARNING
- 7. TUNABLE PARAMETERS INDEEP LEARNING
- 8.
- 9.
- 10. STANDARD TUNING METHODS Parameter Configuration ? GridSearch Random Search Manual Search - Weights - Thresholds - Window sizes - Transformations ML / AI Model Testing Data Cross Validation Training Data
- 11. OPTIMIZATION FEEDBACK LOOP ObjectiveMetric Better Results REST API New configurations ML / AI Model Testing Data Cross Validation Training Data
- 12.
- 13. … the challengeof how to collect information as efficiently as possible, primarily for settings where collecting information is time consuming and expensive. Prof. Warren Powell - Princeton What is the most efficient way to collect information? Prof. Peter Frazier - Cornell How do we make the most money, as fast as possible? Scott Clark - CEO, SigOpt OPTIMAL LEARNING
- 14. ● Optimize objectivefunction ○ Loss, Accuracy, Likelihood ● Given parameters ○ Hyperparameters, feature/architecture params ● Find the best hyperparameters ○ Sample function as few times as possible ○ Training on big data is expensive BAYESIAN GLOBAL OPTIMIZATION
- 15. SMBO Sequential Model-Based Optimization HOWDOES IT WORK?
- 16. 1. Build GaussianProcess (GP) with points sampled so far 2. Optimize the fit of the GP (covariance hyperparameters) 3. Find the point(s) of highest Expected Improvement within parameter domain 4. Return optimal next best point(s) to sample GP/EI SMBO
- 17.
- 18.
- 19. overfit good fitunderfit GAUSSIAN PROCESSES
- 20.
- 21.
- 22. ● Classify moviereviews using a CNN in MXNet SIGOPT + MXNET
- 23. TEXT CLASSIFICATION PIPELINE ML/ AI Model (MXNet) Testing Text Validation Accuracy Better Results REST API Hyperparameter Configurations and Feature Transformations Training Text
- 24. TUNABLE PARAMETERS INDEEP LEARNING
- 25. ● Comparison ofseveral RMSProp SGD parametrizations STOCHASTIC GRADIENT DESCENT
- 26.
- 27. Grid Search RandomSearch ? TUNING METHODS
- 28. SPEED UP #2:RANDOM/GRID -> SIGOPT
- 29. CONSISTENTLY BETTER ANDFASTER
- 30. ● Classify housenumbers in an image dataset (SVHN) SIGOPT + TENSORFLOW
- 31. COMPUTER VISION PIPELINE ML/ AI Model (Tensorflow) Testing Images Cross Validation Accuracy Better Results REST API Hyperparameter Configurations and Feature Transformations Training Images
- 32.
- 33. ● All convolutionalneural network ● Multiple convolutional and dropout layers ● Hyperparameter optimization mixture of domain expertise and grid search (brute force) SIGOPT + NEON http://arxiv.org/pdf/1412.6806.pdf
- 34. COMPARATIVE PERFORMANCE ● Expertbaseline: 0.8995 ○ (using neon) ● SigOpt best: 0.9011 ○ 1.6% reduction in error rate ○ No expert time wasted in tuning
- 35. SIGOPT + NEON http://arxiv.org/pdf/1512.03385v1.pdf ●Explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions ● Variable depth ● Hyperparameter optimization mixture of domain expertise and grid search (brute force)
- 36. COMPARATIVE PERFORMANCE Standard Method ●Expert baseline: 0.9339 ○ (from paper) ● SigOpt best: 0.9436 ○ 15% relative error rate reduction ○ No expert time wasted in tuning
- 37.
- 38. OPTIMIZATION FEEDBACK LOOP ObjectiveMetric Better Results REST API New configurations ML / AI Model Testing Data Cross Validation Training Data
- 39. SIMPLIFIED OPTIMIZATION Client Libraries ●Python ● Java ● R ● Matlab ● And more... Framework Integrations ● TensorFlow ● scikit-learn ● xgboost ● Keras ● Neon ● And more... Live Demo
- 40. DISTRIBUTED TRAINING ● SigOptserves as a distributed scheduler for training models across workers ● Workers access the SigOpt API for the latest parameters to try for each model ● Enables easy distributed training of non-distributed algorithms across any number of models
- 41.
- 42.