Stephen K. Allott - People Challenges for Test Teams - SoftTest Ireland
1 like438 views
This document discusses common challenges faced by test teams based on the author's experience working with various clients. It identifies people-related issues as a key challenge area, including motivation, skills gaps, perception of testers, and management pressures. The document also notes technology issues like increasing complexity, frequent releases and changes, offshore teams, and lack of documentation. It provides an analysis of test process maturity across clients and concludes that testers must adapt to changing business needs, gain technical skills, get involved earlier, and prove their value through metrics.
Stephen K. Allott - People Challenges for Test Teams - SoftTest Ireland
- 1. People Challenges for Test Teams Stephen K. Allott SoftTest Ireland Copyright 2008 ElectroMind Ltd 1
- 2. SoftTest Ireland Copyright 2008 ElectroMind Ltd 2
- 3. SoftTest Ireland Copyright 2008 ElectroMind Ltd 3
- 4. SoftTest Ireland Copyright 2008 ElectroMind Ltd 4
- 5. Introductions My background Aims of this workshop Explain some common testing challenges based on recent experience Ask for your experiences Agree a list of challenges we all face as testers Try to come up with some quick wins & longer term approaches to solutions SoftTest Ireland Copyright 2008 ElectroMind Ltd 5
- 6. My recent clients Investment bank Airline web site Travel company web site Mobile phone operator Retail web site Travel company back office Recruitment web site Includes assignments done in association with TestHouse (London) and Neueda Technologies (Antrim & Dublin) SoftTest Ireland Copyright 2008 ElectroMind Ltd 6
- 7. Their technologies Microsoft Dot (.) NET 1/2/3 Microsoft VSTS / TFS Oracle MS SQL Server Windows XP Citrix Metaframe Quality Centre / QTP SoftTest Ireland Copyright 2008 ElectroMind Ltd 7
- 8. Number of testers Typically 12 to 14 in the test team A couple of clients had only 1, some have none! SoftTest Ireland Copyright 2008 ElectroMind Ltd 8
- 9. Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Key Area Controlled Efficient Optimising Test strategy A B C D Life-cycle model A B Moment of involvement A B C D Estimating and planning A B Test specification techniques A B Static test techniques A B Metrics A B C D Test automation A B C Test environment A B C Office environment A Commitment and motivation A B C Test functions and training A B C Scope of methodology A B C Communication A B C Reporting A B C D Defect management A B C Testware management A B C D Test process management A B C Evaluation A B Low-level testing A B C Composite TPI Test Maturity Matrix from several clients SoftTest Ireland Copyright 2008 ElectroMind Ltd 9
- 10. Summary of TPI results No documented test process No formal test policy, strategy or plan Many tests, no formal design No metrics No proper test environment Some tested in “live” No documented test basis Not involved early enough SoftTest Ireland Copyright 2008 ElectroMind Ltd 10
- 11. Some common observations Constantly changing business priorities Increasing (frantic) demand for new functionality Weekly, sometimes daily release cycles Attempt to use automation to solve all the problems No numbers to explain testing’s added value Small important teams yet with little influence SoftTest Ireland Copyright 2008 ElectroMind Ltd 11
- 12. People challenges – behaviour Motivation A few are resigned to how it is, no hope? Some expect spoon feeding Some adopt a negative attitude Many are willing to make improvements, if there was time Turnover – very high offshore Culture Some managers get too involved in detail Co-dependency – testers must push back It hasn’t broken yet so . . . SoftTest Ireland Copyright 2008 ElectroMind Ltd 12
- 13. People challenges – skills Testing Skills IT Skills Processes Programming Standards Databases Techniques Infrastructure Tools Web Domain Knowledge Soft Skills Industry specific Communication Application knowledge Negotiation E-Business Managing technical teams SoftTest Ireland Copyright 2008 ElectroMind Ltd 13
- 14. People challenges – perception In general testers are not valued Testers are often blamed Project delays Poor quality Managing expectations with RBT Lack of awareness of what the test team does SoftTest Ireland Copyright 2008 ElectroMind Ltd 14
- 15. People challenges – managers Demand for certified staff Demand for skills & knowledge Pressure to cut costs and use offshore resources Developing good test leads Role of testers in an agile project Selling the value of testing to the organisation SoftTest Ireland Copyright 2008 ElectroMind Ltd 15
- 16. Themes Facilitated discussion bearing in mind the following themes: 1. Relentless business pressure for new functionality 2. Constant and last minute changes to requirements 3. Introduction of new development / testing methodologies e.g. agile 4. Very frequent releases, often weekly, sometimes daily! 5. Increasing application complexity over time 6. Knowledge gap (e.g. what lies beyond the browser?) 7. Steep learning curve (business as well as technical) 8. Use of contract and offshore resources (loss of in-house expertise) 9. Little or no documentation against which to test 10. Lack of attention to all testing requirements (e.g. middleware, back end database, non-functional quality characteristics) SoftTest Ireland Copyright 2008 ElectroMind Ltd 16
- 17. Discussion – how do we respond? Business Pressures Technology Offshore New functionality Complex Cheaper Frequent change Rapid change Plentiful Maturity Interfaces Expertise Behaviour Skills Managers Motivation Analysis Costs Attitude IT / Technical Time to market Culture Business Agile Testing Soft skills SoftTest Ireland Copyright 2008 ElectroMind Ltd 17
- 18. Conclusions In my opinion testers of the future must . . . Measure / appreciate application complexity Be able to accept and manage last minute changes Be flexible but push back with estimates Increase their technical knowledge Learn more about testing, especially techniques Apply the tried and tested techniques Capture metrics to help prove the business case Get involved in the projects much earlier Recruit / train / certify / develop / coach = Retain Share your views with the group SoftTest Ireland Copyright 2008 ElectroMind Ltd 18
- 19. Final Thoughts / Questions? SoftTest Ireland Copyright 2008 ElectroMind Ltd 19
- 20. People Challenges for Test Teams [email protected] +44(0) 773 476 1363 www.electromind.com SoftTest Ireland Copyright 2008 ElectroMind Ltd 20
- 21. People Challenges for Test Teams Appendix – Bonus Slides SoftTest Ireland Copyright 2008 ElectroMind Ltd 21
- 22. A typical project in trouble CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3 300 279 300 263 250 254 250 229 213 200 200 Changes to CRs CR count 150 150 116 124 100 100 65 58 55 50 40 50 27 14 0 0 0 0 June July August September October Months CRs Churn Bugs SoftTest Ireland Copyright 2008 ElectroMind Ltd 22
- 23. Bug snapshot by priority Sev 1 Sev 2 Sev 3 Sev 4 Total Pri 1 57 81 14 152 Pri 2 19 52 12 1 84 Pri 3 6 14 10 30 Pri 4 2 1 3 None 68 288 114 10 480 Total 146 428 154 21 749 SoftTest Ireland Copyright 2008 ElectroMind Ltd 23
- 24. Bug snapshot by resolution Sev 1 Sev 2 Sev 3 Sev 4 Total By design 10 37 29 4 80 Duplicate 6 33 7 1 47 External 2 16 4 0 22 Fixed 86 254 95 12 447 Not repro 29 56 4 0 89 Postponed 12 17 6 4 39 Will not fix 1 15 9 0 25 Total 146 428 154 21 749 SoftTest Ireland Copyright 2008 ElectroMind Ltd 24
- 25. Impact of the improvements CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3 300 279 300 263 250 254 250 229 213 Changes to CRs 200 200 CR count 150 150 116 124 100 86 100 65 66 90 58 55 61 50 40 50 27 30 14 17 18 15 15 10 10 5 5 13 12 9 0 0 0 5 1 0 r ry ne ry r ly ch ril r er t be be be s Ju Ap ua a gu ob Ju ar m nu m em br M Au ct ve ce Ja Fe O pt No De Se Months CRs Churn Bugs SoftTest Ireland Copyright 2008 ElectroMind Ltd 25
- 26. Corporate Sales and Sales and Marketing Systems 1. Consulting Practice Mgt. Marketing Systems 2. Vendor Account Mgt. Events Systems 3. Event Mgt. 1. Vendor 4. Leads and Referral 2. User Reg. Databases 3. User Shop 1. Profile 2. Common 3. Event OLTP System Interfaces 4. Transaction OLTP 1. Event Orders 5. Contact History OLTP 2. Billing & Financials 6. Domain Reference 3. Credit Card (real-time) 4. Event Communication / FAX 5. Direct email service World-wide Marketing DB 6. Direct mail Postal interface 1. Profile OLAP 7. Leads & Referrals 2. Event OLAP 8. OLTP Staging Interface 3. Analytical Reporting 9. OLAP Staging Interface 4. Contract History OLAP 10. SAP 5. Domain 11. Licensing 12. Call Center Source: Reality Test Inc.
- 27. Basic complexity calculation Characteristic Value Factor Total Windows 37 8 296 Objects 23 1 23 Tables 47 1 47 Apps Impacted 3 10 30 Factors are weighted from a testing perspective SoftTest Ireland Copyright 2008 ElectroMind Ltd 27
- 28. Comparison across applications Application UI Back Env Test Middle Total End Code Tier Sales Mgr. 1360 2250 152 0 0 3762 Call Center 960 824 90 129 0 2003 Web Sales 389 183 109 0 2 681 Middleware 0 146 23 27 232 428 component SoftTest Ireland Copyright 2008 ElectroMind Ltd 28
- 29. Complexity comparison 4000 Complexity Value 3000 2000 1000 0 er er r e ss TR ta es te as ag ag el la en al C B D G an an S C SS e g eb M M dg l in al P W nt es ok le C ou w al Lo no S cc K A Application Name SoftTest Ireland Copyright 2008 ElectroMind Ltd 29
- 30. Actual staffing comparison 20 Number of Testers 15 Estimated Actual 10 5 0 er er r e ss TR ta es te as ag ag el la en al C B D G an an S C SS e eb g M M dg l in al P W nt es ok C le ou w al Lo no S cc K A Applications SoftTest Ireland Copyright 2008 ElectroMind Ltd 30
- 31. Organisation evaluation Organisation within the company Process definition and execution Teamwork Ability to do the job SoftTest Ireland Copyright 2008 ElectroMind Ltd 31
- 32. Organisation within the company Section Item being evaluated Max Score 1.1 Reporting structure 50 1.2 Professionalism 170 1.3 Organisational churn 135 1.4a Staffing – permanence 30 1.4b Staffing – stability 35 1.4c Staffing – morale 50 1.4d Staffing – number of testers 40 1.4e Staffing – organisation type 40 1.5 Budget control 30 SoftTest Ireland Copyright 2008 ElectroMind Ltd 32
- 33. Process definition and execution Section Item being evaluated Max Score 2.1 Software development life-cycle 60 2.2 Design reviews and documentation 70 2.3 Early participation in development 50 2.4 Ability to influence schedule deadlines 60 2.5 Development standards 60 2.6 Defined testing approach & procedures 190 2.7 Metrics reporting 60 2.8 Churn 90 2.9 Patch releases 40 2.10 Post-project reviews 60 SoftTest Ireland Copyright 2008 ElectroMind Ltd 33
- 34. Teamwork Section Item being evaluated Max Score 3.1 Support from production support 60 3.2 Support from management 60 3.3 Correct level of quality 90 3.4 Resource allocation 80 SoftTest Ireland Copyright 2008 ElectroMind Ltd 34
- 35. Ability to do the job Section Item being evaluated Max Score 4.1 Adequate equipment / lab 200 4.2 Automated testing 70 4.3 Programming ability 70 4.4 Non-testing activities 50 SoftTest Ireland Copyright 2008 ElectroMind Ltd 35
- 36. Total evaluation score 1300 Total Evaluation Score 1200 Evaluation 1100 1000 900 Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01 Date SoftTest Ireland Copyright 2008 ElectroMind Ltd 36
- 37. Organisational Evaluation 500 Organization 400 Process Teaming Evaluation Ability 300 200 100 Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01 Date SoftTest Ireland Copyright 2008 ElectroMind Ltd 37
- 38. Components of a centre of excellence l Tes Project 1 p oo t l ab e o urc s Re Tool Experts Project 2 Test Managers SIG Project 3 e Knowledge Bas C on Project 4 re su ent lta DC nt R& s SoftTest Ireland Copyright 2008 ElectroMind Ltd 38
- 39. Benefits of a centre of excellence Independent testing Variety of work Breadth & depth Identity, pride Fewer product recalls Morale, professionalism Fewer production failures Extra revenue stream Central reference point Consistent approach Knowledgeable staff Multi-methodologies Expertise across org. Earlier involvement Structured career path SoftTest Ireland Copyright 2008 ElectroMind Ltd 39