Pitfalls of machine learning in production
- 1. Pitfalls of Machine Learning in production Antoine Sauray - Nantes Machine Learning Meetup - 7/10/19
- 2. Who am I ? Antoine Sauray 👋 @asauray Software engineer, Scala dev Founder of Hyperplan ML platform We are hosted at the Halle 6 in Ile de Nantes Previously worked at iAdvize as ML Engineer 4 years exp in mobile dev (Android and iOS native) 2
- 4. 4 Machine Learning is a small part of your system Machine Learning “Hidden Technical Debt in Machine Learning Systems” - 2015
- 5. 5 Machine Learning is a small part of your system Not Machine Learning “Hidden Technical Debt in Machine Learning Systems” - 2015
- 6. Machine Learning is hard to put in production O’Reilly survey - August 2018 6
- 7. “ML systems have a special capacity for incurring technical debt, because they have all of the maintenance problems of traditional code plus an additional set of ML-specific issues.” Machine Learning creates more tech debt 7 Hidden Technical Debt in Machine Learning Systems - 2015
- 9. 9 Strong abstractions OOP, FP Hexagonal architecture Tooling Version Control Testing frameworks Monitoring Logging CI Deployments CD A/B tests VS ML practices
- 10. Strong abstractions OOP, FP Hexagonal architecture Tooling Version Control Testing frameworks Monitoring Logging CI Deployments CD A/B tests 10 + specific ML issues Python drift concept etc. VS ?
- 11. “ML as software engineering” 11
- 12. 12
- 13. 13 “Working code is a side effect” Alberto Brandolini
- 15. Collaboration Data dependencies Training and Serving Skews Bad and good patterns Diving into issues 15
- 17. Collaboration difficulties How to track experiments ? What algorithm has been trained ? When ? On what data ? With which set of parameters ? How can we compare them ? ● Avoid repeating same experiments ● Reuse code 17
- 18. Working with others Collaboration difficulties 18 Model Websites Front-end developer Backend Back-end developer Mobile apps iOS/Android developer ML scientist
- 19. Working with others Teams of engineers and researchers have different goals Data scientists and researchers seek quality, at the cost of simplicity Dev and ops like seek simplicity, at the cost of performance Collaboration difficulties 19
- 21. Unstable Data dependencies A model can consume signals from: ● An algorithm (possibly ML) that updates over time ● An application that may receive updates Data Dependencies 21 Model Business appAlgorithm CACE: Change Anything Changes Everything
- 22. Underutilized Data dependencies Keep the features to a strict minimum reduces the multicollinearity risk. ● Legacy Features can be forgotten ● Features can be added as bundles ● ǫ-Features are complex and bring little value Underutilized dependencies can be detected via exhaustive leave-one-feature-out evaluations. Data Dependencies 22
- 23. According to the TensorFlow doc, 4 skews: ● Schema ● Feature ● Distribution ● Scoring and Serving 23 Training and Serving Skews
- 24. Training and Serving Skews Schema skew Example: after a migration/the deprecation of a field. This may be undetected and cause you model to silently fail. 24 { "surfaceAppartement": 70, } { "surfaceAppartement": null, "appartmentSurface": 70 }
- 25. Training and Serving Skews 25 features = { # is this milliseconds or # seconds ? 'video_duration': 10 } prediction = clf.predict(features) Feature skew When different teams implement the machine learning models and the documentation is nonexistent or insufficient. This is even more important when serving on multiple platforms with different teams.
- 26. Distribution skew When the distribution of feature values is different between training and serving. When data scientists train their algorithms with a faulty sampling mechanism. When a different corpus is used for training to overcome lack of initial data. Training and Serving Skews 26
- 27. Training and Serving Skews 27 Model User Data Outcome Scoring and Serving skew A Direct Feedback loop can help you measure the results of your algorithm and generate training data for later. But! Models may influence the data they will be trained on later. Exploration / Exploitation trade off
- 28. Training and Serving Skews 28 Scoring and Serving skew Hardcoded thresholds can lead to errors because they are model specific. If model is automatically trained with new batches of data, the previous threshold is invalidated But thresholds can be learnt. def keep_above_threshold(prediction, t): for p in predictions: if p.prob > t: yield p clf = load('model1.joblib') predictions = clf.predict(features) return keep_above_threshold(predictions, 0.6) clf = load('model2.joblib') predictions = clf.predict(features) return keep_above_threshold(predictions, 0.6)
- 29. Bad and good patterns 29
- 30. Bad and good patterns Bad: Glue code Wikipedia: “... executable code that serves solely to «adapt» different parts of code that would otherwise be incompatible”. Problems Maintenance is hard Complexity is high 30
- 31. Bad: Pipeline jungle When thinking about data collection after creating the algorithms. It is often hard to gather data in real time for the execution of the algorithm. ● Many collection steps ● SQL Join everywhere ● Intermediate results This is fragile → Rethink the pipeline as a whole Bad and good patterns 31
- 32. Bad: Dead experimental codepath Shortcuts taken by developers to implement features faster. Avoid conditional branching in production because of ● Cyclomatic complexity ● Things break unexpectedly Bad and good patterns 32 if customer_id == "1": # but customer 1 does not exist anymore return clf1.predict(data) else: return clf2.predict(data)
- 33. Bad and good patterns 33 Client API 1 API 2 Heuristic Model 2 Model 3 API 3 Schema 1 Schema 2 Schema 3 Bad: No Abstraction Client handle different APIs for the same problem. ● More code client side ● Duplication of code in APIs ● Failover and AB tests are managed on the client
- 34. Good: Abstraction An interface between the client and the models Advantages: ● Start with simple heuristics to get data early ● Add data on the API schema progressively ● Ship a new model safely with rollbacks ● Write less code on the client side Bad and good patterns 34 Heuristic Client API Model 2 Model 3 Schema Transformations
- 35. Platforms
- 36. The basics We have some basics tools like classic software Versioning ● Git LFS ● DVC Tracking platforms ● Weights and biases wandb.com ● Datmo github.com/datmo/datmo ● MLFlow Tracking 36
- 37. TensorFlow Extended End-to-end platform for deploying production ML pipelines Since 2018 37
- 38. TensorFlow Extended End-to-end platform for deploying production ML pipelines Since 2018 38
- 39. TensorFlow Extended 39 Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ❌ ✅ (schema, feature, distribution) ✅ (only tf.Serving and Apache Beam) ✅ (API, browser, on device) ❌ (possible via TensorFlow 2.0)
- 40. MLFlow Machine Learning Platform for ML Lifecycle management v1: summer 2019 40
- 41. ✅ MLFlow 41 Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ✅ ❌ ✅ (via save_model) ❌ (only REST API)
- 42. Since 2016 “Seamlessly build, deploy, and operate machine learning solutions at Uber’s scale” eng.uber.com/michelangelo Michelangelo (closed source) 42
- 43. Michelangelo (closed source) 43 ✅ Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ❌ ✅ with a compiled DSL ✅ NA
- 44. FB Learner Flow (closed source) A platform capable of easily reusing algorithms in different products, scaling to run thousands of simultaneous custom experiments, and managing experiments with ease. engineering.fb.com/core-data/introducing-fblearner-flow-f acebook-s-ai-backbone 44
- 45. FB Learner Flow (closed source) 45 ✅ Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ✅ NA NA NA
- 46. A Proxy for Machine Learning Serve your algorithms in production ● Skew detection ● Feedback loop ● AB Testing Still under active development Hyperplan 46
- 47. Hyperplan 47 Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ❌ ✅ (schema, feature, distribution) ❌ ✅ ✅
- 48. 48 To sum up Tracking and Reproducibility Skew Detection Consistent Pipeline (Training & Serving) Multiple Deployment Targets Multi Framework ❌ ✅ ❌ ✅ ✅ ✅ NA NA NA ✅ ✅ ❌ ✅ NA ✅ ✅ ❌ ✅ ❌ ✅ ❌ ✅ ❌ ✅ ❌
- 49. 49 To sum up TensorFlow Extended and MLFlow are great for big orgs TFX has a proven record for solving production issues, but only with TensorFlow MLFlow is still young and targets many problems, pick what you need Uber and Facebook ? Designed specifically for their org, huge platforms Hyperplan is made for small and medium orgs Benefits of monitoring at a small cost
- 50. References Machine Learning: The High Interest Credit Card Of Technical Debt, 2014 Hidden Technical Debt in Machine Learning Systems, (Google) 2015 Machine Learning in Production: Developing and Optimizing Data Science Workflows and Applications, First Edition (O’Reilly) The State of Machine Learning Adoption in the Enterprise (O’Reilly) TensorFlow Docs https://www.tensorflow.org/tfx MLFlow Docs https://www.mlflow.org/docs/latest/index.html 50
- 51. Merci ! @asauray - hyperplan.io