TOOL SUPPORT FOR TESTING (K2)

ADVANCED SOFTWARE ENGINEERING

WEEK – 13

D.M.K.D Dissanayake
MSc-IT(Cyber security), PGDIT, BSc(Hons),HR(Dip)
TOOL SUPPORT FOR TESTING (K2)

We will cover:
• Types of Test Tools (K2)
• Effective Use of Tools: Potential Benefits and Risks
• Introducing a Tool into an Organization (K1)
TYPES OF TEST TOOLS
PURPOSE OF TOOL SUPPORT FOR TESTING
TOOL SUPPORT FOR TESTING
TOOL SUPPORT FOR TESTING

 Test tools can be used for one or more activities that support testing. These include:
1. Tools that are directly used in testing such as test execution tools, test data generation tools and
result comparison tools
2. Tools that help in managing the testing process such as those used to manage tests, test results,
data, requirements, incidents, defects, etc., and for reporting and monitoring test execution
3. Tools that are used in reconnaissance, or, in simple terms: exploration(e.g., tools that monitor file
activity for an application)
4. Any tool that aids in testing (a spreadsheet is also a test tool in this meaning)
TOOL SUPPORT FOR TESTING

 Tool support for testing can have one or more of the following purposes depending on the
context:
• Improve the efficiency of test activities by automating repetitive tasks or supporting manual test
activities like test planning, test design, test reporting and monitoring
• Automate activities that require significant resources when done manually (e.g., static testing)
• Automate activities that cannot be executed manually (e.g., large scale performance testing of
client-server applications)
• Increase reliability of testing (e.g., by automating large data comparisons or simulating behaviour)
TOOL SUPPORT FOR TESTING
TEST TOOL CLASSIFICATION (K2)
TEST TOOL CLASSIFICATION (K2)
TEST TOOL CLASSIFICATION (K2)

• There are a number of tools that support different aspects of testing.

• Tools can be classified based on several criteria such as purpose, commercial / free /
open-source / shareware, technology used and so forth.
• Tools are classified in this syllabus according to the testing activities that they support.
• Some tools clearly support one activity; (Mantis, Bugzilla, Redmine)
• others may support more than one activity, but are classified under the activity with
which they are most closely associated.
• Tools from a single provider, especially those that have been designed to work together,
may be bundled into one package.
TEST TOOL CLASSIFICATION

• Some types of test tools can be intrusive, which means that they can affect the actual
outcome of the test.
• For example, the actual timing may be different due to the extra instructions that are
executed by the tool, or you may get a different measure of code coverage.
• The consequence of intrusive tools is called the probe effect.
• Some tools offer support more appropriate for developers (e.g., tools that are used
during component and component integration testing). Such tools are marked with
"(D)" in the list below.
TEST TOOL CLASSIFICATION
TOOL SUPPORT FOR MANAGEMENT OF TESTING AND TESTS
TOOL SUPPORT FOR MANAGEMENT OF TESTING AND TESTS

 Management tools apply to all test activities over the entire software life cycle.
 They include:
 Test Management Tools
 Requirements Management Tools
 Incident Management Tools (Defect Tracking Tools)
 Configuration Management Tools
TEST MANAGEMENT TOOLS

• These tools provide interfaces for executing tests, tracking defects and managing
requirements, along with support for quantitative analysis and reporting of the test
objects.
• They also support tracing the test objects to requirement specifications and
• They may have an independent version control capability or an interface to an external
configuration management tool.
REQUIREMENTS MANAGEMENT TOOLS

 These tools store requirement statements,

• store the attributes for the requirements (including priority),
• provide unique identifiers and support tracing the requirements to individual tests.
• These tools may also help with identifying inconsistent or missing requirements.
INCIDENT MANAGEMENT TOOLS (DEFECT TRACKING TOOLS)

• These tools store and manage incident reports, i.e., defects, failures, change requests or
perceived problems and anomalies,
• they help in managing the life cycle of incidents, optionally with support for statistical
analysis.
CONFIGURATION MANAGEMENT TOOLS

 Although not strictly test tools, these are necessary for storage and version management
of testware and related software especially when configuring more than one
hardware/software environment in terms of operating system versions, compilers,
browsers, etc.
TOOL SUPPORT FOR STATIC TESTING
TOOL SUPPORT FOR STATIC TESTING

 Static testing tools include:

 Review Tools
 Static Analysis Tools (D)
 Modelling Tools (D)

 Static testing tools provide a cost effective way of finding more defects at an earlier
stage in the development process.
REVIEW TOOLS

• These tools assist with review processes, checklists, review guidelines and are used to
store and communicate review comments and report on defects and effort.
• They can be of further help by providing aid for online reviews for large or geographically
dispersed teams.
STATIC ANALYSIS TOOLS (D)

 These tools help developers and testers find defects prior to dynamic testing by providing
support for:
 enforcing coding standards (including secure coding),
 analysis of structures and dependencies.

 They can also help in planning or risk analysis by providing metrics for the code (e.g.,
complexity).
MODELLING TOOLS (D)

• These tools are used to validate software models (e.g., physical data model (PDM) for a
relational database), by enumerating inconsistencies and finding defects.
• These tools can often aid in generating some test cases based on the model.
TOOL SUPPORT FOR TEST SPECIFICATION
TOOL SUPPORT FOR TEST SPECIFICATION

 Test Design Tools

These tools are used to generate test inputs or executable tests and/or test oracles from
• requirements,
• graphical user interfaces,
• design models (state, data or object)
• code.
 Test Data Preparation Tools
Test data preparation tools manipulate databases, files or data transmissions to set up
test data to be used during the execution of tests to ensure security through data
anonymity.
TOOL SUPPORT FOR TEST EXECUTION AND LOGGING (K1)
TOOL SUPPORT FOR TEST EXECUTION AND LOGGING (K1)

 Tool Support for Test Execution and Logging include:

 Test Execution Tools
 Test Comparators
 Coverage Measurement Tools (D)
 Security Testing Tools

(D) Indicates that the tools offer support more appropriate for developers
 TEST EXECUTION TOOLS

• These tools enable tests to be executed automatically, or semi-automatically, using stored

inputs and expected outcomes, through the use of a scripting language and usually
provide a test log for each test run.
• They can also be used to record tests, and usually support scripting languages or GUI-
based configuration for parameterization of data and other customization in the tests.
TEST HARNESS/UNIT TEST FRAMEWORK TOOLS (D)

• A unit test harness or framework facilitates the testing of components or parts of a system
by simulating the environment in which that test object will run, through the provision of
mock objects as stubs or drivers.
• the tools offer support more appropriate for developers
TEST COMPARATORS

• Test comparators determine differences between files, databases or test results.

• Test execution tools typically include dynamic comparators,
• post-execution comparison may be done by a separate comparison tool.
• A test comparator may use a test oracle, especially if it is automated.
COVERAGE MEASUREMENT TOOLS (D)

 These tools, through intrusive or non-intrusive means, measure the percentage of specific
types of code structures that have been exercised (e.g., statements, branches or
decisions, and module or function calls) by a set of tests.
 E. g.
 Statements, branches or decisions
 Modules or function calls

 the tools offer support more appropriate for developers

SECURITY TESTING TOOLS

 These tools are used to evaluate the security characteristics of software.

 This includes evaluating the ability of the software to protect data
• confidentiality,
• integrity,
• authentication,
• authorization,
• availability,
• non-repudiation.

 Security tools are mostly focused on a particular technology, platform, and purpose.
TOOL SUPPORT FOR PERFORMANCE AND MONITORING (K1)
TOOL SUPPORT FOR PERFORMANCE AND MONITORING (K1)

 Performance and Monitoring Support Tools include:

 Dynamic Analysis Tools (D)
 Performance Testing/Load Testing/Stress Testing Tools
 Monitoring Tools

(D) Indicates that the tools offer support more appropriate for developers
DYNAMIC ANALYSIS TOOLS (D)

 Dynamic analysis tools find defects that are evident only when software is executing, for
example:
 time dependencies
 memory leaks.
 They are typically used in component and component integration testing, and when testing
middleware.
 These tools offer support more appropriate for developers
PERFORMANCE TESTING/LOAD TESTING/STRESS TESTING TOOLS

• Performance testing tools monitor and report on how a system behaves under a variety of
simulated usage conditions in terms of number of concurrent users, their ramp-up pattern,
frequency and relative percentage of transactions.
• The simulation of load is achieved by means of creating virtual users carrying out a
selected set of transactions, spread across various test machines commonly known as load
generators.
MONITORING TOOLS

• Monitoring tools continuously analyze, verify and report on usage of specific system
resources, and give warnings of possible service problems.
TOOL SUPPORT FOR SPECIFIC TESTING NEEDS
TOOL SUPPORT FOR SPECIFIC TESTING NEEDS

Data Quality Assessment

 Data is at the centre of some projects such as data conversion/migration projects and applications
like data warehouses and its attributes can vary in terms of criticality and volume.
 In such contexts, tools need to be employed for data quality assessment to review and verify the
data conversion and migration rules to ensure that the processed data is correct, complete and
complies with a pre-defined context-specific standard.
 Other testing tools exist for usability testing.
EFFECTIVE USE OF TOOLS: POTENTIAL BENEFITS AND RISKS (K2)

We will cover:
 Potential Benefits and Risks of Tool Support for Testing (for all tools) (K2)
 Special Considerations for Some Types of Tool (K1)
POTENTIAL BENEFITS AND RISKS OF TOOL SUPPORT FOR TESTING
(FOR ALL TOOLS)

• Simply purchasing or leasing a tool does not guarantee success with that tool.
• Each type of tool may require additional effort to achieve real and lasting benefits.
• There are potential benefits and opportunities with the use of tools in testing, but there
are also risks.
POTENTIAL BENEFITS AND RISKS OF TOOL SUPPORT FOR TESTING
(FOR ALL TOOLS)

 Potential benefits of using tools include:

Repetitive work is reduced
 (e.g., running regression tests, re-entering the same test data, and checking against coding standards)

Greater consistency and repeatability

 (e.g., tests executed by a tool in the same order with the same frequency, and tests derived from
requirements)
Objective assessment
 (e.g., static measures, coverage)

Ease of access to information about tests or testing

 (e.g., statistics and graphs about test progress, incident rates and performance)
POTENTIAL BENEFITS AND RISKS OF TOOL SUPPORT FOR TESTING
(FOR ALL TOOLS)

Risks of using tools include:

• Unrealistic expectations for the tool (including functionality and ease of use)
• Underestimating the time, cost and effort for the initial introduction of a tool (including training
and external expertise)
• Underestimating the time and effort needed to achieve significant and continuing benefits from
the tool (including the need for changes in the testing process and continuous improvement of the
way the tool is used)
• Underestimating the effort required to maintain the test assets generated by the tool
RISKS OF USING TOOLS INCLUDE:

 Over-reliance on the tool (replacement for test design or use of automated testing where manual
testing would be better)
 Neglecting version control of test assets within the tool
 Neglecting relationships and interoperability issues between critical tools, such as requirements
management tools, version control tools, incident management tools, defect tracking tools and tools
from multiple vendors
 Risk of tool vendor going out of business, retiring the tool, or selling the tool to a different vendor
 Poor response from vendor for support, upgrades, and defect fixes
 Risk of suspension of open-source / free tool project
 Unforeseen, such as the inability to support a new platform
SPECIAL CONSIDERATIONS FOR SOME TYPES OF TOOL

 Test Execution Tools

 Test execution tools execute test objects using automated test scripts.
 This type of tool often requires significant effort in order to achieve significant benefits.
 Capturing tests by recording the actions of a manual tester seems attractive, but this approach
does not scale to large numbers of automated test scripts.
 A captured script is a linear representation with specific data and actions as part of each script.
This type of script may be unstable when unexpected events occur.
SPECIAL CONSIDERATIONS FOR SOME TYPES OF TOOL

 A data-driven testing approach

 the test inputs (the data) are separated, usually into a spreadsheet,
 a more generic test script is used that can read the input data and execute the same test script
with different data.
 Testers who are not familiar with the scripting language can then create the test data for these
predefined scripts.
 There are other techniques employed in data-driven techniques, where instead of hard-coded
data combinations placed in a spreadsheet, data is generated using algorithms based on
configurable parameters at run time and supplied to the application.
 For example, a tool may use an algorithm, which generates a random user ID, and for
repeatability in pattern, a seed is employed for controlling randomness.
SPECIAL CONSIDERATIONS FOR SOME TYPES OF TOOL

 a keyword-driven testing approach,

 the spreadsheet contains keywords describing the actions to be taken (also called action words),
and test data. Testers (even if they are not familiar with the scripting language) can then define
tests using the keywords, which can be tailored to the application being tested.
 Technical expertise in the scripting language is needed for all approaches (either by testers or by
specialists in test automation).
 Regardless of the scripting technique used, the expected results for each test need to be
stored for later comparison.
STATIC ANALYSIS TOOLS

• Static analysis tools applied to source code can enforce coding standards,
• but if applied to existing code may generate a large quantity of messages.
• Warning messages do not stop the code from being translated into an executable
program, but ideally should be addressed so that maintenance of the code is easier in
the future.
• A gradual implementation of the analysis tool with initial filters to exclude some
messages is an effective approach.
TEST MANAGEMENT TOOL

• Test management tools need to interface with other tools or spreadsheets in order to
produce useful information in a format that fits the needs of the organization.
MAIN CONSIDERATIONS OF INTRODUCING A TOOL

• Assessment of organizational maturity, strengths and weaknesses and identification of

opportunities for an improved test process supported by tools
• Identification of internal requirements for coaching and mentoring in the use of the tool
• Evaluation of training needs considering the current test team's test automation skills
• Estimation of a cost-benefit ratio based on a concrete business case