Driver vs Driverless AI - Mark Landry, Competitive Data Scientist and Product Manager, H2O.ai
Download as PPTX, PDF0 likes459 views
Mark Landry, a data scientist at H2O.ai, analyzes the performance of Driverless AI in feature creation for two case studies: predicting student attempts on an online platform and identifying bank customer churn. The document compares traditional driver methods with the automated capabilities of Driverless AI, highlighting various feature importance and transformation techniques used in both approaches. Ultimately, Driverless AI demonstrated superior feature generation and accuracy compared to manual modeling.
![The Driver Approach
Exploit before Explore
• 377 columns made [quick] manual investigation harder
• Rather than iterate: {analyze > model > analyze > … },
I changed to {model > analyze > model > … }
lagged features](https://image.slidesharecdn.com/driver-vs-driverless-v1-mark-landry-171208083240/85/Driver-vs-Driverless-AI-Mark-Landry-Competitive-Data-Scientist-and-Product-Manager-H2O-ai-23-320.jpg)
Driver vs Driverless AI - Mark Landry, Competitive Data Scientist and Product Manager, H2O.ai
- 2. Data Scientist, H2O.ai GitHub: mlandry22, Email: [email protected] Mark Landry
- 3. An Analysis of Driverless AI Feature Creation Driver vs Driverless
- 4. Overview • Analyze two recent problems with an emphasis on feature generation • Overview of the problem • Discussion of my approach • Show the features Driverless created • Feature creation • Feature representation • Results
- 5. Who am I? • Data scientist @ H2O since 2015, user since 2014 • 15 top 20 finishes in Kaggle, highest rank 33 • R | H2O | data.table | GBM • The “driver”
- 6. Problem #1 How Many Attempts will a Student Make • Online question/answer platform with computer science problems • Predict the number of attempts a particular student will make on a particular problem • Data • Student: level, ranking, highest ranking • Problem: type, 3-tier level, points awarded • Training: 124,000 attempt counts • Testing: 60,000 attempt counts
- 7. Regression as Classification • Natural problem is numerical • End user prefers buckets • Volume • 1: 53% • 2 31% • 3 9% • 4 4% • 5 1% • 6 2%
- 8. The Driver Approach • Think of it like a recommender problem • Standard: Matrix factorization, collaborative filtering • GBM: use deep categorical encodings • Frequent use of target encoding interaction interaction interaction
- 9. The Driver Approach A messy chain of hierarchical target encoding and if/else statements
- 10. The Driver Approach h2o.gbm feature importance – primarily using three target encodings
- 14. The Driverless Approach Top 5 Features: divided into components • {16} {CV TE} {problem_id} {0} • {51} {CV TE} {points * problem_id} {0} • {3} {max_rating} • {32} {freq} {last_online_time_seconds * rating} • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 15. The Driverless Approach Feature #1: same base target encoding I found to be the best • {16} {CV TE} {problem_id} {0} • 16: indicator of base features – 16 is later used twice more in the top 15 • CV TE: Cross-Fold target encoding • Problem_id: feature used as basis for target encoding • 0: the target; multinomial, so the two other uses are for class 1 & 5 • {51} {CV TE} {points * problem_id} {0} • {3} {max_rating} • {32} {freq} {last_online_time_seconds * rating} • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 16. The Driverless Approach Feature #2: deeper interaction • {16} {CV TE} {problem_id} {0} • {51} {CV TE} {points * problem_id} {0} • 51: base feature ID • CV TE: out of sample target encoding result • points * problem: interaction of two different features, both related to the problem; it is subdividing the problem further • 0: again, rate of class 0 as the target • {3} {max_rating} • {32} {freq} {last_online_time_seconds * rating} • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 17. The Driverless Approach Feature #3: no transformation • {16} {CV TE} {problem_id} {0} • {51} {CV TE} {points * problem_id} {0} • {3} {max_rating} • max rating • used as is – no alternate encoding; was first natural feature in my model as well • this is the first variable of the student dimension • a 4-digit number with close to a normal distribution • {32} {freq} {last_online_time_seconds * rating} • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 18. The Driverless Approach Feature #4: frequency encoding • {16} {CV TE} {problem_id} {0} • {51} {CV TE} {points * problem_id} {0} • {3} {max_rating} • {32} {freq} {last_online_time_seconds * rating} • Counting the occurrences of two fields • Last online & rating are both numerics in the student dimension • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 19. The Driverless Approach Feature 5: four-way interaction w/ target encoding • {16} {CV TE} {problem_id} {0} • {51} {CV TE} {points * problem_id} {0} • {3} {max_rating} • {32} {freq} {last_online_time_seconds * rating} • {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0} • Target encoding of class 0 • Finding the rate for each value of the result of a four way interaction
- 20. The Driverless Approach Top 5 Features: did I try? • YES {16} {CV TE} {problem_id} {0} • NO {51} {CV TE} {points * problem_id} {0} • YES {3} {max_rating} • NO {32} {freq} {last_online_time_seconds * rating} • NO {61} {NumToCatTE} {rating * max_rating * points * last_online_time_seconds} {0}
- 22. Problem #2 Bank Customer Churn • Identify customers likely to churn balances in the next quarter by 50% • Data • 300,00 training rows; 200,000 testing rows • 377 columns • Customer: age, gender, demographics • Reported assets, liabilities • Monthly balance history
- 23. The Driver Approach Exploit before Explore • 377 columns made [quick] manual investigation harder • Rather than iterate: {analyze > model > analyze > … }, I changed to {model > analyze > model > … } lagged features
- 24. The Driver Approach After lagging, try differences and ratios • Lagging features present the balance features at several time steps. • But often, the interesting part is not the raw balance itself, but whether it is growing or shrinking • Decision trees have a hard time “seeing” this so it is wise to engineer mathematical features: + - * /
- 25. The Driver Approach After lagging, try differences and ratios • I used the leading monthly feature from the model and created new features representing month-over-month differences and a binary indicator • One field, one specific length (1 month), two calculations
- 28. The Driverless Approach It knows math! subtraction subtraction subtraction
- 29. The Driverless Approach Top 10 Features: divided into categories • (3) Subtraction: #1, #6, #8 • (1) Truncated SVD components: #2 • (2) Cluster Distances: #3, #9 • (1) Target encoding: #4 • (3) Direct features: #5, #7, #10
- 30. The Driverless Approach Lagged balances also used in clusters & SVD • Distance to cluster #1 after segmenting columns into 6 clusters • BAL_prev6 • D_prev1 • D_prev2 • I_AQB_PrevQ1 • Component #1 of truncated SVD of • D_prev1 • D_prev2 • EOP_prev1_1
- 31. The Driverless Approach Final Analysis • On first iteration, Driverless AI had surpassed my manual modeling • Features were well beyond what I would have ever attempted • Accuracy was stable: Driverless AI self-reported scores within 1% of competition submission
- 32. The Driverless Approach Competition Results Kaggle Grandmaster Kaggle Grandmaster Also Driverless AI 100% Driverless AI