Quant university MRM and machine learning
1 like413 views
The document discusses the management and validation of AI and machine learning models in financial services, highlighting challenges, lifecycle considerations, and the importance of trusted AI validation. It outlines various machine learning techniques, common algorithms, and the role of risk management in AI adoption. Additionally, it emphasizes the need for rigorous model risk frameworks and the integration of AI within regulatory guidelines and best practices.
Quant university MRM and machine learning
- 1. 1 Managing AI / Machine Learning Models in Financial Services Presentation for Quant University Summer School Stuart Kozola August 26, 2020
- 2. 2 Outline Challenges in AI {model} Validation From Machine Learning to AI Criteria for Trusted AI {model} Validation AI in Model Risk Management Lifecycle
- 3. 3 From Machine Learning to Artificial Intelligence Rise of the Internet→ Big Data Source: http://www.aboutdm.com/2013/04/history-of-machine-learning.html History of Deep Learning: https://arxiv.org/pdf/1702.07800.pdf) Deep Neural Networks
- 4. 4 Artificial Intelligence Machine Learning AI, Machine Learning and Deep Learning Timeline 1950s Today1980s ApplicationBreadth Bioinformatics RecommenderSystems Spam Detection Fraud Detection WeatherForecasting Algorithmic Trading SentimentAnalysis Medical Diagnosis Health Monitoring ComputerBoard Games Machine Translation Knowledge Representation Perception Reasoning Interactive Programs Expert Systems Deep Learning Automated Driving SpeechRecognition RoboticsObjectRecognition Trading Reinforcement Learning
- 5. 5 Enabling Technology: Deep Learning and GPUs Source: ILSVRC Top-5 Error on ImageNet Annual Image Recognition Challenge
- 6. 6 Machine Learning Categorization Machine Learning Supervised Learning Classification Regression Unsupervised Learning Clustering Group and interpret data based only on input data Develop predictive model based on both input and output data Typeof Learning CategoriesofAlgorithms Semi-supervised, Reinforcement,… Learning Numeric output Categorical output Categorical output
- 7. 7 Clustering Algorithms Machine Learning Supervised Learning Classification Regression Unsupervised Learning Clustering Group and interpret data based only on input data Typeof Learning CategoriesofAlgorithms Semi-supervised, Reinforcement, … Learning k-Means, Fuzzy C-Means Hierarchical (Deep) Neural Networks Gaussian Mixture Hidden Markov Model Algorithms
- 8. 8 Classification Algorithms Machine Learning Supervised Learning Classification Regression Unsupervised Learning Clustering Develop predictive model based on both input and output data Typeof Learning CategoriesofAlgorithms Semi-supervised, Reinforcement, … Learning Algorithms Nearest Neighbor Discriminant Analysis Naive Bayes Support Vector Machines Non-linear Regression (GLM, Logistic) Linear Regression Decision Trees Ensemble Methods (Deep) Neural Networks
- 9. 9 Regression Algorithms Machine Learning Supervised Learning Classification Regression Unsupervised Learning Clustering Develop predictive model based on both input and output data Typeof Learning CategoriesofAlgorithms Semi-supervised, Reinforcement, … Learning Algorithms Nearest Neighbor Discriminant Analysis Naive Bayes Support Vector Machines Non-linear Regression (GLM, Logistic) Linear Regression Decision Trees Ensemble Methods (Deep) Neural Networks
- 10. 10 Common Approaches in Use Today
- 11. 11 FEATURE EXTRACTION What is Machine Learning? Machine learning uses data and produces a model to perform a task Cat Dog Bird Car Car Dog Cat Bird … MODEL
- 12. 12 Cat Dog Bird Car Learned Features + Model … Car Dog Cat Bird What is Deep Learning ? Deep learning is a type of machine learning that learns tasks directly from data
- 13. 13 Comparison of modeling approaches Learned Features + Model … Car Dog Cat Bird FEATURE EXTRACTION Car Dog Cat Bird … MODEL Deep Learning Machine Learning
- 14. 14 Reinforcement Learning By representing policies using deep neural networks, we can solve problems for complex, non-linear systems (continuous or discrete) by directly using data that traditional approaches cannot use easily Reinforcement Learning Deep Reinforcement Learning
- 15. 15 Challenges with AI {model} Validation Input Network Output
- 16. 16 Analysts Agree AI Project Continue to Grow
- 17. 17 Risk Management’s role is more important than ever Source: The State of AI in Risk Management, Chartis Research 2019 Adoption of AI models in retail banking Source: The value in digitally transforming credit riskmanagement, McKinsey & Company 2016
- 18. 18 And the new reality is more … Models Complex Models Coupled Processes Disruption Climate changeE S G Covid-19 Normal Fraudulent Source: Interpreting machine learning models, Tow ards Data Science: Medium 2019
- 19. 19 But the current environment is … complicated Source: https://xkcd.com/1987/Source: HSBC MATLAB Expo Presentation
- 20. 20 Criteria for Trusted AI {model} Validation Trusted AI Safety Trusted Fairness Robustness Interpretable Explainable
- 21. 21 Has the AI modelor system beendevelopedwith safety as a key component? Does the AI modelor system produce consistent and reliable results? Questions to answer during AI Validation Can you explain the workings of an AI model or system in human understandable terms? Can you observerand trace cause and effectin AI modelor system and explain the rationale of the decision? Does the AI modelor system make decisions or recommendations that are fair and free of bias? Is the AI modelor system immune from spoofing or other commonattacks? Does it provide privacy protectionand is it secure? Trusted AI Safety Trusted Fairness Robustness Interpretable Explainable
- 22. 22 Explainability vs Interpretability Explainability = “why is this happening?” e.g., this is a picture of a “Schnauzer” because of the eyebrows and moustache. Intrepretability = discern the mechanics without necessarily knowing why e.g., if it’s more that 434 days since 2011 and the temperature is more that 11.6 degrees, then 7000 bicycles will be rented https://christophm.github.io/interpretable-ml-book/tree.htmlVisualization discoverypresentation
- 23. 23 Adversarial Examples Even rotating an image can cause it to be misclassified
- 24. 24 Principles of fairness require being able to explain why the model is making decisions 2. Use of personal attributesas input factors for AIDA-driven decisionsis justified. 13. Data subjects are provided,upon request, clear explanations on what data is used to make AIDA-driven decisionsabout the data subject and how the dataaffects the decision. 4. AIDA-drivendecisions are regularly reviewed so that models behave as designed and intended. “AIDA” refers to artificial intelligence or data analytics, which aredefined as technologies that assistor replace human decision-making. https://www.mas.gov.sg/~/media/MAS/News%20and%20Publications/Monographs%20and%20Inf ormation%20Papers/FEAT%20Principles%20Final.pdf 8. Firms using AIDA are accountable for both internally developedand externally sourced AIDA models.
- 25. 25 Expanding Model Validation within Model Risk Frameworks
- 26. 26 Explaining/Interpreting Machine Learning Methods Model-specific Logistic Regression Decision Tree xNN Model-agnostic Global Predictor Importance Partial Dependency Local LIME SHAPley Values
- 27. 27 XNN - Separation of “model” and “explanation” This image from https://www.darpa.mil/attachments/XAIProgramUpdate.pdfand the same idea expressed in https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime
- 28. 28 Visual Interpretation of Features Across Layers https://blogs.mathworks.com/deep-learning/2019/01/18/neural-network-feature-visualization/
- 29. 29 Visual Interpretation of output: Siamese Network For Identifying Unique groups Network
- 30. 30 GAN Network that Creates Synthetic Data Network
- 31. 31 AI / Machine Learning in Model Risk Management
- 32. 32 - Explore, develop, back-test, and document models and methodologies - Improve transparency and reproducibility of model development process - Create reusable model templates - Auto-generate model documentation Model Development Environment (MDE) Achieve reduced time and costto marketand compliance for yourtraditionalrisk and AI models.Perform end-to-endmodellingin line with globalregulatory guidance (SR11-7,EU TRIM,OSFIE-23, UK SS 3/18, etc.) and industry bestpractices - Perform independent model reviews - Perform interactive what-if and sensitivity analysis on model parameters - Comment and flag various aspects for response and resolution Model Review Environment (MRE) MDE MRE MTVE MIR MEE MMD - Configure performance thresholds and alerts for breaches and generate reports - Summarize model execution results using a customizable web dashboard - Analyze the model usage to determine candidate models for retirement Model Monitoring Dashboard (MMD) - Deploy models in production environment without recoding - Integrate with existing technology infrastructures - Host production models and scale to end users in a secure controlled environment “on-prem” or “cloud” Model Execution Environment (MEE) Centralized access to models, lineage, audit trail, risk scoring, and model risk reporting Model Inventory & Repository (MIR) - Automatically run unit tests and generate test reports - Perform preproduction testing and validation for approved models - Compare tests of preproduction model with a production model Model Test & Validation Environment (MTVE) MATLAB Model Risk Management (MRM) Solution
- 33. 33 Managing Model Complexity for Deep Networks Library of pre-trained models (enables transfer learning) Design of new or customize existing networks
- 36. 36 Living Documentation for Review and Challenge of Models
- 38. 38 Fully Integrated and Automatable Workflow Files Databases Market Data Access and Explore Data 1 PreprocessData Working with Messy Data Data Reduction/ Transformation Feature Extraction 2 Develop& Validate Predictive Models Model Creation e.g. Machine Learning Model Validation Parameter Optimization 3 Deploy into Production Desktop Apps Enterprise Scale Systems Enterprise Systems 4 Real-time Reporting 3rd party dashboards Web apps 5
- 39. 39 Cloud Model Development CI / CD MATLABPlatform Domain specific toolboxes Azure DevOps MATLAB Parallel Server Training, simulation, optimization MATLAB Web App Server Risk Deep Learning Web App Dashboards Sharing, deployment, integration Data exploration, preprocessing, model development Model Inventory Model Review Model Validation and Testing Reporting Monitoring Development Review Validate Execute Monitor MATLAB Production Server MATLAB Online Apps/APIs BigData DataSources Data stores Streaming Data feeds Files Alternative Data MATLAB ModelRisk ManagementSuite has built-in APIs to seamlessly interoperatewith a wide rangeof open source and third party technologyplatformsacrossthe modeling life-cycle MATLAB is interoperable with wide range of technology platforms MDE MRE MIR MEE MMD MTVE
- 40. 40 Learn more about MathWorks, our products, and our services at mathworks.com and on social media: