Downloaded 37 times
Uploaded byPerfecto by Perforce
Smarter Automation with Machine Learning & AI-Based Reporting
The document discusses the challenges and trends in continuous testing, highlighting the struggles with automation at scale and the need for efficient test analysis, particularly with the aid of AI. It points out that slow triaging and unstable tests hinder continuous testing efforts, emphasizing the importance of identifying real failures and improving test environments. Furthermore, it provides insights into Machine Learning (ML) and AI applications for enhancing test automation and analysis, aiming to increase success rates and reduce error classification time.
More Related Content
Similar to Smarter Automation with Machine Learning & AI-Based Reporting
Smarter Automation with Machine Learning & AI-Based Reporting
- 1. © 2015, PerfectoMobile Ltd. All Rights Reserved.
- 2. About Us • YoramMizrachi • CO-Founder & CTO, Perfecto • Tzvika Shahaf • Director of Product management, Perfecto Co-Authors of Continuous Testing for DevOps Professional Working with fortune 500 enterprises and leading tech companies worldwide helping them optimize their testing strategy and achieve Continuous Testing
- 3. Agenda • Market Trendsoverview: • Continuous Testing • AI in Testing • Continuous Testing – The feedback loop cycle • Analysis Crunch • Slow triaging • Past vs. Future of test analysis • Why test Fails • Test Analysis with AI
- 4. Market Trends –Automation Testing Continuous Testing has not delivered. Everyone talks about it… But reality show that Automation-at-Scale is hard to achieve Lot of data Unstructured Rings a bell….
- 5. Reporting Creation Execution Lab Tools setup Authoring& maintenance Debugging Execution Scheduling Validations Orchestration Test environment Desktop web Artifacts collection Mobile devicesMobile Emulators Triaging dashboards Analytics Market Trends – AI in testing
- 6. Monthly Release DailyCycle Number of Tests Cases 500 10,000 Time for Single Test Review 2 minutes Mean of Review Manual review ? Time to Review @ 5% Failure Rate 50 minutes (25 x 2) ~16 hours (500 x 2) Time to Review @ 25% Failure ~4 hours (125 x 2) ~10 days (2500 x 2) Real Bugs Found Out of Failures 1 50 Available Review Window 3 days 1 hour Continuous Testing – Test Result Analysis Crunch
- 7. Slow Triaging blockscontinuous testing o If not consumed in minutes – Automation is practically not part of the Pipeline. o Inability to fix / comment-out unstable tests in CI immediately – leads to failure indifference. Can you tell which of the failures here is real and which isn’t?.... **** You can keep executing everything everyday. But it has no meaning!
- 8. Reasons for “notstable” 80% of issues have a pattern52% success rate 10% of devices, causing 80% of lab issues Lab 25% Orchestr ation 25% Scripts & FW 50% FAILURE REASON Objects Codding Time Other Scripts & FW issues Device in use No Device Orchestration issues Networking Stability Lock Other Lab issues
- 9. Smart insights iskey for Test Automation success 10% of devices, causing 80% of lab issues 0% 10% 20% 30% 40% Issues Lab 25% Orchestr ation 25% Scripts & FW 50% FAILURE REASON Objects Codding Time Other Scripts & FW issues Device in use No Device Orchestration issues Networking Stability Lock Other Lab issues What’s wrong With my Scripts What’s wrong With my Lab What’s wrong With my Executions
- 10. PAST vs. TheFUTURE 11/16/2018 10© 2015, Perfecto Mobile Ltd. All Rights Reserved. VS. Traditional Test Analysis ML/AI Based Test Analysis
- 11. Classification & Trainingof AI Modals • Supervised classification– collection of examples with the current answers • Recognize same item/person based on tagged photo albums • Training dataset – List of organized input-output data (Future and label) 11/16/2018 11 © 2015, Perfecto Mobile Ltd. All Rights Reserved. Cat Horse Car Glass Google Photo AI
- 12. Training AI thePhoto Lingo 11/16/2018 12© 2015, Perfecto Mobile Ltd. All Rights Reserved. CarCat Train Glass Dog Sea Sun Table Tree
- 13. Training AI theTesting Lingo 11/16/2018 13© 2015, Perfecto Mobile Ltd. All Rights Reserved. PopupApp not installed Elementnotfound App didn’t launch Capability not supported App Crashed Keyboard Device in Use Trust
- 14. Training the TestingAI Engine – Image Recognition • Stabilize Execution By Eliminating Noise • Pop ups • Home screen • Keyboard open • Enhanced Reporting and Analysis 11/16/2018 14© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 15. Training AI theTesting Lingo 11/16/2018 15© 2015, Perfecto Mobile Ltd. All Rights Reserved. Popup App not installed Elementnotfound App didn’t launch Capability not supported App Crashed Keyboard Device in Use Trust
- 16. Image Recognition –Popup Handling
- 17. Perfecto’s DigitalZoom –Error Analysis & Managemnt Blocked Tests - Isolate “System Errors” and eliminate False Negatives Identify TOP FAILURE reasons
- 18. Perfecto’s DigitalZoom –Error Analysis & Mngmnt. • “Green is green, so why bother?” • Focus on functional failure and reduce the triaging time
- 19. Error Classification andTest Productivity 11/16/2018 19© 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 20. Reporting Creation Execution Lab Tools setup Authoring& maintenance Debugging Execution Scheduling Validations Orchestration Test environment Desktop web Artifacts collection Mobile devicesMobile Emulators Triaging dashboards Analytics Additional usage of AI in testing