Kaggle Days Porto 2019 - 1st place presentation by team DevScope
Download as PPTX, PDF0 likes836 views
The document summarizes Team DevScope's approach for the Kaggle competition KDM Porto 2019, which involved forecasting hourly agent headcounts at a call center. Their approach included blending regression, time series, and human knowledge models; addressing non-stationarity and a small test set; and extensive post-processing and validation to improve forecasts, especially around holidays. While their blend of approaches achieved first place, they discuss opportunities to better use additional features and automated time series tools in future forecasts.
Kaggle Days Porto 2019 - 1st place presentation by team DevScope
- 4. • Non-Stationary Time Series • (obvious trends/change points) • Small Test Set (744 rows, 31 days, 24 hours) • 23 months, but only Dec 2017 available • (hard for models to model proper seasonality for Dec 2018 prediction) • Hard to have an internal validation strategy • We used Nov 2018 as validation set, closest to the target month Dec 2018 • Besides that, we also trusted on submission visualization and PLB score • Time splitted validation for bayesian optimization • Time! (lack of!)
- 5. • Blend of very different approaches • (regression, time series, human knowledge) • Post/processing, human in the loop forecast • Christmas holidays, model obvious errors • Obsessive & Constant EDA of Train & Submissions • LightGBM, Prophet, Bayesian Search • Ceiling raw predictions (to be confirmed) • “The time series with the real values will be constructed using a simple agent headcount average rounded to the next integer” • Probe the leaderboard (as feedback is important) • Yes, It’s a competition!
- 6. Model Public Score Private Score Blend of Top 3 below +ceiling 8.53526 8.69781 1st Prophet Based 10.39806 9.86448 - 31 Models (LGBM regression) 10.44760 10.51587 - LGBM regression 10.23931 10.34396 - +Adjustments values for Christmas holidays/special days, based on 2017 Dec +Other Post Processing fixing specific model limitations
- 7. • •
- 11. • • • • •
- 12. • • • • •
- 13. • • • • •
- 14. • • • •
- 15. • Post/processing & fixing obvious errors • (model limitations, not always time to inspect root causes) • Negative predictions -> 0 (as agent head count cannot be negative)
- 16. • •
- 18. • Use other features (we mostly used target only) • Explore feature engineering with additional features • Understand the predictive power of other features/metrics • Explore Azure AutoML Forecast • promising results but couldn't make it on time to submit • Built-in time split cross validation, holidays & model search
- 19. • Used Total Calls as a feature to predict agents. • But it did not work. Score private: 15.41 and public: 16.37 • Did not work on local validation • Model with 4 variables (On a Call Time, Total Handle Time, Total Calls, and Agent Headcount) • One reason for not improving the score considering additional variables, can be that, additional variables are calculated based on Agent Headcount of that hour. What didn’t work (so far...)
- 22. • • • • • •
- 23. • Non-Gaussian forecasting using fable • On the Automation of Time Series Forecasting Models: Technical and Organizational Considerations. • Forecasting Stock Prices using Exponential Smoothing • An overview of time series forecasting models
- 25. © 2019 DevScope. All rights reserved. Rua Passos Manuel Nº 223 – 4º Andar 4000-385 Porto T. +351 223 751 350/51 F. +351 223 751 352 Av. Sidónio Pais, Nº 2 – 3º Andar 1050-214 5 Lisboa [email protected] www.devscope.net