Introduction to Test Automation
59 likes44,845 views
The document provides a comprehensive overview of test automation, detailing various scripting approaches such as record and playback, linear scripting, modular scripting, data-driven testing, and keyword-driven testing. It discusses the benefits and challenges of each method while highlighting the importance of testability, collaboration in automation, and the use of version control and continuous integration practices. The document emphasizes that while automation is crucial, manual testing, particularly exploratory testing, remains necessary.
Introduction to Test Automation
- 1. An introduction to Test Automation © Copyright Pekka Klärck <[email protected]> Creative Commons Attribution 3.0 License
- 2. Scope ● Dynamic vs. static ● Functional vs. non-functional ● Acceptance vs. unit (vs. module vs. integration) ● Frameworks vs. drivers ● Running tests vs. generating tests ● Full scale automation vs. helping manual testing ● Test execution vs. test management
- 3. Different scripting approaches ● Record and playback ● Linear scripting ● Modular scripting ● Data-driven testing ● Keyword-driven testing
- 4. Record and playback ● Capture interaction with system and replay it ● Popular approach among commercial tools
- 5. Record and playback: Example Selenium IDE
- 6. Record and playback: Benefits ● Very easy and fast to create initially ● No programming skills needed
- 7. Record and playback: Problems ● Does not test anything unless checkpoints added ● Very fragile ● Often single change in UI can break all tests ● Hard to maintain ● Plenty of separate test scripts ● No modularity or reuse ● System must be ready before automation can start ● Does not support acceptance test driven development (ATDD)
- 8. Record and playback: Summary ● Seldom a good approach in general ● Never a good basis for large scale automation
- 9. Linear scripting ● Non-structured scripts interact directly with the system under test (SUT) ● Can use any programming language ● Also produced by capture and replay tools
- 10. Linear scripting: Example Selenium RC Python API
- 11. Linear scripting: Benefits ● Fast to create initially ● Flexible ● Can use common scripting languages ● No license costs
- 12. Linear scripting: Problems ● Creating tests requires programming skills ● Very fragile ● One change in the system may break all scripts ● Hard to maintain ● Plenty of test scripts ● No modularity or reuse
- 13. Linear scripting: Summary ● Adequate for simple tasks ● Never a good basis for large scale automation
- 14. Modular scripting ● Driver scripts “drive” test execution ● Interaction with the SUT done by functions in a test library
- 15. Modular scripting: Example ← Test library ↓ Driver script
- 16. Modular scripting: Benefits ● Reuse of code ● Creating new tests gets faster ● Maintainability ● Changes require fixes in smaller areas ● Driver scripts are simple ● Even novice programmers can understand and edit ● Creating new ones is not hard either
- 17. Modular scripting: Problems ● Building test library requires initial effort ● Takes time ● Requires programming skills ● Test data embedded into scripts ● Requires some understanding of programming ● New tests require new driver scripts
- 18. Modular scripting: Summary ● Good for simple tasks ● Works also in larger usage if everyone who needs to understand tests can program ● Not good for non-programmers
- 19. Data-driven testing ● Test data taken out of test scripts ● Customarily presented in tabular format ● One driver script can execute multiple similar tests ● New driver script still needed for different kinds of tests
- 21. Data-driven testing: Benefits ● Test libraries provide modularity ● Same benefits as with modular scripting ● Creating and editing existing tests is very easy ● No programming skills needed ● Maintenance responsibilities can be divided ● Testers are responsible for the test data ● Programmers are responsible for automation code
- 22. Data-driven testing: Problems ● Test cases are similar ● For example '1 + 2 = 3' and '1 * 2 = 2' ● New kinds of tests need new driver script ● For example '1 * 2 + 3 = 6' ● Creating driver scripts requires programming skills ● Initial effort creating parsers and other reusable components can be big
- 23. Data-driven testing: Summary ● Good solution even for larger scale use ● New kinds of tests requiring programming is a minus ● May be an overkill for simple needs
- 24. Keyword-driven testing ● Not only test data but also directives (keywords) telling how to use the data taken out of the test scripts ● Keywords and the test data associated with them drive test execution
- 25. Keyword-driven testing: Example Robot Framework syntax from SeleniumLibrary demo: http://bit.ly/rf-web-test-demo
- 26. Keyword-driven testing: Benefits ● All same benefits as with data-driven testing ● Non-programmers can create and edit tests ● Separation of test data and code ● Tests can be constructed freely from keywords ● Non-programmers can create also new kinds of tests ● With suitable keywords also data-driven tests possible ● All tests can be executed by one framework ● No need to create and maintain separate driver scripts
- 27. Keyword-driven testing: Problems ● Initial effort can be really big ● But there are open source solutions available!
- 28. Keyword-driven testing: Summary ● Very good solution for large scale use ● Use existing solutions if you can ● May be an overkill for simple needs
- 29. Interacting with the SUT ● Testability ● Testing through GUI ● Testing below GUI ● Other interfaces
- 30. Testability ● The hardest part of automation is interacting with the system under test ● Especially hard with GUIs ● Programming APIs are easiest ● Important to make the system easy to test ● Some common guidelines ● Add identifiers to GUI widgets ● Textual outputs should be easy to parse ● Consider providing automation interfaces
- 31. Testing through GUI ● Same interface as normal users use ● Can be technically challenging or impossible ● Not all GUI technologies have good tools available ● Often fragile tests ● Often relative slow to execute ● Good approach to use when feasible
- 32. Testing below GUI ● Automating through business layer often easy ● Tests typically run very fast ● But you still need to test the GUI ● Test the GUI is wired correctly to the business logic ● GUIs always have some functionality of their own ● Pragmatic hybrid solution: ● Test overall functionality below the GUI ● Some end-to-end tests through the GUI—not necessarily even automated
- 33. Other interfaces ● Not all systems have a GUI ● Many systems have multiple interfaces ● Programming APIs, databases, server interfaces, command line, … ● Automation framework which can utilize different drivers works well in these situations ● Non-GUI interfaces generally easy to automate ● Most of them targeted for machines ● Test library is just another client
- 34. When to automate and by whom? ● After development by separate team ● During development collaboratively
- 35. Automation after development ● Often by different team ● In worst case on a different floor, building, or continent ● Communication problems ● Typical in waterfall-ish projects ● Slow feedback loop ● Testability problems can be show stoppers ● Often hard to get testability hooks added afterwards ● May need to resort to complicated and fragile solutions
- 36. Collaborative automation ● Automation considered an integral part of development ● Collaboration between testers and programmers ● Typical in Agile projects ● In acceptance test driven development (ATDD) automation started before implementation ● Testability normally not a problem ● Programmers can create testability hooks ● Testability and available tooling can be taken into account even with technology decisions
- 37. Supporting practices ● Version control ● Continuous integration
- 38. Version control ● Test data and code should be stored the same way as production code ● Recommended to store tests with the production code ● Easy to get an old version of the software with related tests ● Lot of great open source alternatives available ● Subversion, Git, Mercurial, … ● No excuse not to use
- 39. Continuous integration ● Key to full scale automation ● Tests are run automatically when ● New tests are added ● Code is changed ● Can also have scheduled test runs ● Useful if running all tests takes time ● Great open source solutions available ● Jenkins/Hudson, Cruise Control, BuildBot, … ● Custom scripts and cron jobs can be retired
- 40. Available tools ● Commercial ● Open source ● Freeware
- 41. Commercial tools ● Good ones tend to be expensive ● But not all expensive are good ● Even cheap licenses can prevent full team collaboration ● Often hard to integrate with ● Other automation tools (especially from other vendors) ● Version control and continuous integration ● Hard or impossible to customize ● Risk of product or company discontinuation
- 42. Open source tools ● Large variety ● Some are great—others not so ● Normally easy to integrate with other tools ● Free, as in beer, is good ● Everyone can use freely ● Free, as in speech, is good ● Can be customize freely ● Can never really die
- 43. Freeware tools ● Getting rare nowadays ● Most free tools are also open source ● No license costs ● Tend to be easier to integrate with other tools than commercial ● Hard or impossible to customize ● Risk of discontinuation
- 44. Generic skills to learn ● Scripting languages ● Python, Ruby, Perl, Javascript, ... ● Regular expressions ● A must when parsing textual outputs ● XPath and CSS selectors ● A must when doing web testing ● SQL ● A must with databases ● Using version control
- 45. Is manual testing still needed? ● YES!! ● Avoid scripted manual testing ● Automate it instead ● Concentrate on exploratory testing ● Machines are great for running regression tests ● Humans are great for finding new defects
- 46. Questions? Thanks! © Copyright Pekka Klärck <[email protected]> Creative Commons Attribution 3.0 License