Mlse20190208
1 like•1,892 views
1. AE is a domain-agnostic platform for adaptive experimentation developed by Facebook. It allows using techniques like Bayesian optimization and A/B testing to iteratively improve systems based on experimental results. 2. AE supports sequential experimentation techniques including Bayesian optimization to automatically suggest promising parameter configurations. It also allows for human experts to manually select configurations. 3. The core API and backend services of AE are designed to handle large-scale experimentation workloads from companies like Facebook to help optimize systems like the DNN models and web interfaces.
Mlse20190208
- 1. AE: A domain-agnostic platform for adaptive experimentation 2019-02-08@ MLSE NuerIPS2018 Sys for ML
- 2. Yuta Kashino ( ) BakFoo, Inc. CEO Astro Physics /Observational Cosmology Zope / Python Realtime Data Platform for Enterprise PoC/Prototyping
- 3. AE: A domain-agnostic platform for adaptive experimentation Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham, Ashwin Murthy, Shaun Singh Facebook
- 4. AE: • Google Vizier / Ray.Tune ML + Python • Vizier: http://bit.ly/2DZoh4I • Ray.Tune: http://bit.ly/2DZ4D8U • FB (OSS • FB Learner Flow http://bit.ly/2DXHGD5 … • sequential experiment: BO,
- 5. AE: 1. 1. DNN Web UI A/B / HHVM JIT 2. HHVM JIT: PHP C++/LLVM http://bit.ly/2DYOqQO 2. human-in-the-loop 3. A/B 4.
- 6. 1. : BO • Bayesian Optimization • • Spearmint http://bit.ly/2E0pM2p • BayesOpt http://bit.ly/2DZ1mGB • MOE http://bit.ly/2DZOCiS • Hyperopt http://bit.ly/2DYIGXx • Optuna https://optuna.org/ • Ray.Tune http://bit.ly/2DZ4D8U • SigOpt https://sigopt.com/ • Vizier http://bit.ly/2DZoh4I • Kaggle Hyperopt
- 7. 1.seqential expriment • sequential expriment: • box 1992 http://bit.ly/2E0qPiR • Bayesian Optimization Bayesian Optimization
- 8. 1. :BO • • Gaussian Process(GP) • A/B • x: parameter config. • x*: param. • f: • Expected Improvement(EI) • EI http://bit.ly/2GA7Opj EI(x) = ∫ ∞ −∞ max(0,y − y*)p(y|x)dy y = f(x), y* = f(x*)
- 9. 1.Hyperopt • BO : Tree Parzen Estimator (TPE) • PFN http://bit.ly/2GtAm3s • • • fmin
- 10. 1.AE: Core API
- 12. 3.AE: • Bayesian Optimization • Constrained Bayesian Optimization with Noisy Experiments http://bit.ly/2E1pJTZ • Expected Improvement • botorch • PyTorch/GPyTorch •
- 13. 4.AE: • Runner Batch • Online/Offline • MySQL • AnalysisPlan Batch • AnalysisPlan • AnalysisPlan
- 14. AE: FB • HHVM JIT • PlanOut UI A/B • DNN • • FB Video • FB http://bit.ly/2GwMAbI
- 15. • AE Facebook
- 16. • AE Runner: Python shell • Ray + Tune • DNN Kubernetes • Kubeflow + Seldon • Polyaxon • 2018 ML http://bit.ly/2GwocH8 http://bit.ly/2GuCxEa
- 17. BakFoo, Inc. • Software 2.0 http://bit.ly/2DZ4z96 • Sofware 2.0 = ML + Software 1.0
- 18. BakFoo, Inc. • Software • Kubernetes • FaaS + • FaaS: AWS Lambda, Google Functions … • kubeflow polyaxon • DB/ • Distributed Functions ( ) •
- 20. BakFoo, Inc. NHK NMAPS: +
- 22. BakFoo, Inc.
- 23. BakFoo, Inc. : SNS +