Experitest-Infosys Co-Webinar on Mobile Continuous Integration
2 likes1,493 views
The document discusses the importance of continuous integration (CI) for mobile app development, highlighting how it differs from traditional CI and detailing a recommended flow for integration. It presents a case study of a major bank that successfully implemented CI, resulting in significant test coverage and improved development speed. Additionally, it emphasizes the critical need for performance testing in mobile apps to prevent negative user experiences that can lead to loss of customers.
Experitest-Infosys Co-Webinar on Mobile Continuous Integration
- 1. Infosys & ExperitestCo-webinar: Total Quality in Mobile Apps -Continuous Integration & Performance Testing Nov 2014
- 2. Part I How To Use Continuous Integration for Mobile Speaker: Guy Arieli, ExperitestCTO
- 3. Agenda Why is it important? How is it different from regular Continuous Integration? What is the recommended flow for Continuous Integration in mobile? Case study: major bank Demo: mobile continuous integration Jenkins Continuous Integration is especially important for mobile 1 2 3 4 5
- 4. What Is Continuous Integration? Software development practice where members of a team integrate their work frequently Significantly reduces integration problems and allows a team to develop cohesive software rapidly Each person integrates at least daily –leading to multiple integrations per day CI significantly reduces integration problems and allows a team to develop cohesive software rapidly YOUR APP Developers Source Code Repository CI Server
- 5. Why is CI especially important for mobile apps? Mobile app development is faster End-user expectation for fast response time Frequent external environment changes New OS versions New devices High adoption of Agile methodology in R&D Fast app release Many (hundreds) devices to be tested Several mobile OS to be tested Huge device matrix Higher ROI because of mega matrix
- 6. How is mobile CI different? Parallel execution Complex management of Continuous Integration Smart device allocation Dual application mode (instrumented, non-instrumented)
- 7. Continuous Integration For Mobile Developers Source Code Repository CI Server YOUR APP ExecuterAgent Execute Test On Devices x Test Results
- 8. Recommended process for mobile Continuous Integration Build the application Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution View and analyze reports
- 9. Build the Application For example: If it’s an Android build, you will get a URL for downloading this APK file Build the Application Source Code Repository CI Server YOUR APP New Build of Your App Trigger the entire execution
- 10. Recommended process for mobile Continuous Integration Build the application Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution View and analyze reports
- 11. Prepare the Application YOUR APP New Build of Your App YOUR APP YOUR APP Instrumented more control and more capabilities over the application Non-instrumented (optional step) Application under test may have two modes: Tips for Implementation Unattended, on-the-fly instrumentation Test automation tool should enable: 1 2 Perform the instrumentation on the same machine that is being used for automation 3 Windows machine to perform all instrumentation (inc.on-the-fly) (also in the case of an iOS app)
- 12. Dynamic, smart reservation of the Devices to be used @os=‘android’ and @versionnumber> 4.2 and @manufacture=‘htc’ Dynamic, smart reservation Execute your regression of tests on as many devices as you can Cloud of Devices Smart Device Reservation
- 13. Recommended process for mobile Continuous Integration Build the application Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution View and analyze reports
- 14. Deploy Application and Device Preparation **These operations should be available and should be able to run entirely unattended. Device reboot Application clear data Uninstallation of applications Installation of applications Launch of applications Have full control of the device and of the application installation Working with the Settings
- 15. Recommended process for mobile Continuous Integration Build the application Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution View and analyze reports
- 16. Tests Execution CI Server ExecuterAgent Execute the Devices Remote Mobile device Lab T1 T2 T2 T2 … Tn Actions for making Test execution efficient: Capability to run the tests in parallel on different devices Decupling between the CI server and the execution machine Queue of tests that are waiting for devices the be available
- 17. Recommended process for mobile Continuous Integration Build the application Prepare the application for automation Reserve the devices to be used Deploy application and device preparation Tests Execution View and analyze reports
- 18. Recommended process for mobile Continuous Integration
- 19. Demo How to use Jenkins for mobile Continuous integration
- 20. Case Study
- 21. Headquartered in APAC Spans over 50 countries across Europe, America, Africa &Asia ~200,000 employees Mobile applications in tens of languages Ranked in top 10 largest banks worldwide Revenue more than $30 Bn. Customer Profile: Multinational banking and financial services company Major global financial services provider engaged in personal banking, credit cards, corporate and investment banking and wealth and investment management.
- 22. The bank’s Personal and Business accounts management apps Wallet apps Trading apps Mortgages apps BankCard Many more… The Need Testing end users' apps Testing the Internal bank Employees’ Apps Loan qualification form Mortgage requirements form Etc. Applications used by The bank employees providing services to customers such as:
- 23. The Requirements On-premise devices, all within the bank VPN Cross-platform testing on all OS: Android, iOS, WindowsPhone8 & BlackBerry No need for access to application source code for testing (on-the-fly instrumentation) Support for multi-field form filling including scrolling in lists, invisible objects, etc. Ability to test on devices located in one of the bank facility by testers located in other locations </>
- 24. Implementation Years working with the tool: 3 Over 100SeeTestAutomationstations Number of Automation engineers working on the project: 200 4geographical locations around the world (US, Europe, Africa and China) Number of tests before: 0 Number of tests today: ~20,000 Number of executed tests: ~200,000 (each test can run on more than one device) Onsite deployment with integration into the existing test environments QTP, Junit, Eclipse QC and Jenkins Parallel execution of tests
- 25. Coverage Total coverage: ~90%. Number of executions of the full cycle: Once a day Number of apps tested: +20, (different application in different regions) Coverage of both customer’s apps and internal systems apps Full app test coverage for all OS-Cross platform testing of Android iOS WindowsPhone8 BlackBerry Full end-to-end coverage of test scenario Support for multiple languages (including English, Spanish, French)
- 26. What if we did it manually? A single QA engineer: 56 man years to finish the cycle (lifetime). For one day we would need 20,000 trained manual QA engineers. Another Advantage The R&D engineers can work faster and take more risks According to the bank R&D managers the development speed is ROI 3-4 times faster!
- 28. The Tools Used •Automation tool •Same test runs on different devices •Plugs into all testing & CI environments: •UFT/Selenium/ Microsoft Visual Studio/ RFT/Java/C#/TestComplete//Perl/Python/Ruby •QC/TFS/Jenkins/TeamCity/RQM •Manual testing tool •Mini-recorder for automating short, repetitive, manual scenarios •Report with screenshots of each step •Device Lab located onsite (in the customer VPN) •Devices can be reserved and accessed by any remote user •Used in conjunction with SeeTestAutomation/Manual for testing of remote devices SeeTestAutomation SeeTestManual Free Tool! SeeTestCloud Manual to Automation Script Convertor
- 29. Thank you
- 30. Infosys Point of View (POV) on Mobile Application Performance Testing 17 December, 2014
- 31. Performance is key, for a Mobile user Poor App Store ratings … are often linked with poor app performance •Frequent crashes and glitches •Slow loading times •High battery drain •Unreliability •Inefficient start and exits … Forrester: “Mobile apps live and die by their ratings in an App Store… When the rating suffers, customer adoption suffers.” Analysis reveal that majority of 1-star reviews are related to performance issues, proving that Quality is key for an end-user. Apigee: “44% of users delete a mobile app immediately if it didn’t perform as expected; Performance is reported to be especially important for banking (74%) apps IMS, Goldsmiths: "Users experience a lot of negative emotions and frustrations when trying to complete some digital tasks and apps are slow to load and respond" realBusiness: “Consumers are unforgiving towards slow and poor –performing apps.” 52% 0 76% 48% 15% 9% CRITICAL, 1-STAR REVIEWS PERFORMANCE RELATED, CRITICAL, 1-STAR REVIEWS % of Critical, 1-Star Reviews For Top iOS Apps Performance Related Other Crash, Bug Lagging, Performance, Hot, Battery Drain Slow, Long Load, Load Fails Source: appurify
- 32. If left unaddressed, the impacts are significant Lost transactions not only result in lost revenue for banks, but have a detrimental impact on customer loyalty and brand reputation –it's a lose-lose situation. 38 percent try another app when faced with a problem, 34 percent stop using the app, and 19 percent complain to friends and family Majority users delete or uninstall mobile app because of performance problems. About 18% delete the app immediately if it froze for just 5 sec. 38% delete the app if it freezes for longer than 30 seconds 30% of users would spend more money with an organization that had a good mobile app 29 percent would pay more for a product or service if the organization’s app performed better than its competitors Nearly one third of smartphone and tablet owners would change banks if a mobile app wasn’t up to par
- 33. Addressing Customer Reviews on Mobile apps.. 33 •One in six customers who switch banks indicate a poor mobile banking experience prompted the switch •Between 51% and 76% of mobile users use their mobile apps to check balances, transfer money, pay bills, and look up branch locations and hours. Hence reduced performance leads to greater & immediate customer dissatisfaction Problem Cause Solution Application is buggy (doesn’t open,crashes, freezes often). Sloppy code, varying internet connection, poor memory management. Thorough testing of the app under multiple conditions and variables to ensure that an app is devoid of issues Infosys’ PerformanceTesting Solution ClientApplication Performance Server Side Performance Network Impact on Performance
- 34. Device Impact Testing / Client Application Performance 34 Key Measurement Parameters: •Battery Consumption •Device CPU and Memory constraints •Device I/O –Assess speed for read and write to devices Tools: -XcodeInstruments (memory allocation, leaks, CPU usage …) -Android ADB (battery consumption, CPU usage / processing speed) -Android DDMS ( memory usage, device I/O, track running thread info ..) -Quadrant (only for Android) -Neoload
- 35. Server Side Performance / Load Testing 35 Key Measurement Parameters: •Response Time •Error Rate •Throughput •Server CPU utilization •Server Memory utilization Tools: -LoadRunner -Neoload -CloudTest Mobile Performance Tool Recommendation: •Ability to support multiple devices •Ability to support native applications as well as browser-based applications •Ability to record through the device using agents, or through emulators, as well as directly from a browser to reduce complexity •Ease of use and fast turnaround time
- 36. Network Impact Testing 36 Network Emulation Options: Network Coverage -Occasional disconnection of the network. Packet Loss & Duplication –Packet loss & duplication for an application session by dropping packets based on preset values Delay and Jitter -Delay of data originating from app . Multi-Network testing (Roaming) –Switching from one type of network to another type of network e.g. (3G, 4G, LTE) Key Measurement Parameters: •Response Time •Exception Handling •UI Issues •Consistency Tools: -Infosys Network Impact Testing Solution (iNITS) -Shunra -Neoload Mobile Performance Tool Recommendation: •Ability to dynamically emulate various network conditions •Ability to generate and or emulate load from various locations, leveraging the cloud •Ability to test regardless of the protocol the application uses
- 37. Some of the Performance Testing Best Practices for Mobile Apps Test early: Do quick check on mobile web application performance, using page analysis tools like ‘PageSpeedInsights’, Yslowetc. For native apps, developers can perform regular checks with Instruments / DDMS tools Verify & validate the performance on a real device Perform both modular and system performance tests Correlate server-side metrics with client-side response times. It helps nailing down the root cause May look at reusing the functional scripts for load Testing (e.g. SeeTestLoad-LoadRunner) Look for consistency in test results, during multiple test iterations Ensure complete performance test coverage, executing Load, Endurance, Stress and Single-user test cycles 37
Editor's Notes
- #39: To reproduce this slide simply create a new slide, right click and select layout and apply the Notes&Disclaimer layout.