Why performance testing?
Do want to use an Movie Booking web-site to book a ticket
and wait for 15 minutes to know that there are no seats available?
Why performance testing?
How many of us like to click on a button and see the result –
“Error : Page not found”?
Bad performance is bad for business
Lack of proper performance testing results in loss of revenue,
loss of credibility, loss of customers
Why Applications become slow?
Multiple users using the application at the same point of time
Load on the application increases
Applications become slow
What is Performance Testing Check
Performance of
application
Application Under
Creating the test
load on the
application (Lets assume Gmail
application)
What are Vusers Can we have
Image real users
there were
1000 users
These
were real
users
Virtual Users
OR
Vusers
Types of users
User Base (100000) registered
Application Users / Online users (100) (90 +10)
Concurrent users (10)
Key Facts
Number of concurrent users is not same as the number of
application users
Performance testing tools and protocols
8
What is
Architecture?
What is architecture?
An architecture is a
combination of software and system components and connections
10
Importance of architecture
Helps in
Designing smaller and more manageable pieces of application
Understanding dependencies between components
Enables interaction
Between different parties (stakeholders) who intend to develop the computer-
based system
Between different systems and system components
11
What is a server
Software (Computer Program) or Hardware which
Understands your request
Process your request
Send the response back
Application Server (APP Server)
An application server is a server program
in a computer in a distributed network that
provides the business logic
for an application program
Web Server
An web server is full implementation of HTTP Protocol
A Browser is partial implementation of Http Protocol
Application layers
Presentation layer
Accepts user inputs, displays data and any results computation on the user
interface
Application processing layer
Provides application specific functionality
e.g., in a banking system, banking functions such as open account, close
account, view account details etc.
Data management layer
It manages the system database
15
Client/Server - 2 Tier architecture
Improves multi-user updating.
These days again this model is picking up because of mobile apps
There are 2 types of Models
16
3-Tier Architecture
Presentation
Layer
Business logic tier between presentation logic and data tier
Business Logic
Layer
Data Access
Layer
Data
PC Database
(Thin Client) Server
Application
Server
17
Three tier architecture
Advantages
Separate processor might be present for each application layer
This architecture is simpler to
manage than fat-client model and also gives better performance
than a thin-client model
Can be scaled easily
more servers can be added as demands increase
18
N-Tier (Multi-Tier Architecture)
Business Logic
Layer
Presentation Layer Data Access Layer
Enterprise Data
Database
PC (Thin Client) Web Server Application Server
Server
Specialized web servers were introduced which can generate presentation content
which is transferred to the browser on the client tier, which takes care of rendering
the user interfaces
19
What is performance testing?
PT is testing an application for speed, stability and scalability in “Production
like Environment” under virtual user load to meet Non-Functional
requirements. (NFR’s)
Speed: How fast the application responds to your request. In other words,
Response time should be less
Stability: Performance should not degrade even after using the
Application for very long time.
Scalability: Application’s ability to handle increasing user without
performance degradation
Purpose of Performance Testing
To identify Performance Bottlenecks
Functionality of the application should not be changed under real
world conditions
Key Notes:
Performance Tester are required to identify Performance Bottlenecks but not Defects
(Bugs)
Important Terms in PT
PT: Performance testing
NFR : Non-Functional testing.
Response Time - Time taken by the system to respond to a specific
transaction request
Think Time –Time taken for selecting new transaction after receiving
the response for previous transaction.
Throughput - Transactions per second, Bytes per second
90 percentile response time
When we execute a Performance Test, same transaction will be executed
multiple times.
Since, the same transaction is executed multiple times, multiple response
times are captured
Refer
Client willtobe
inserted spreadsheet
provided formin
with Max, 90% ,RTAvg,
calculations.
and 90% response times.
LoadRunner automatically calculate the 90% response times
Performance Testing VS Performance
Engineering
Performance testing is a subset of performance engineering
Performance Testing Performance Engineering
Performance Testing usually deals PE not only deals with identifying the
with identifying issues in the issues but also eliminating the same.
Application Under Test
Types of Performance Testing
Load Testing
This is a mandatory test which will be done prior to any test.
This test usually simulate current user load on the AUT.
Usually Peak load for the application is considered for this test.
This test ensures that the AUT is stable and handle expected peak
load once the application is deployed.
Scenario Load Test – MERCURY SAMPLE PROJECT
Name
Scenario Load Test – Duration 1 hour.
Type
Scenario To simulate the peak Load and to monitor the performance of
Objective the MERCURY SAMPLE PROJECT online system
Steps The online load will be maintained at steady state for 1 hour
with only critical transactions
Entry All the Monitors are in place
Criteria Test Data is set-up
Shakedown completed successfully
Exit Criteria Response times meet the SLA
Test completion report is agreed upon by stakeholders
Stress Testing
Stress Test is conducted by increasing the user load gradually until the application
breaks.
Objective of this test is to obtain the breaking point or saturation point.
Usually once the breaking point is attained there is a noticeable increase in errors
and also there is a big increase in response time.
This test is performed to check if application can accept spikes.
Spike Testing
Spike Testing is considered to be subset of Stress Testing.
It is done by increasing the user load beyond anticipated load for short periods of
time.
Scenario Stress Test
Name
Scenario Type Stress Test – Duration N/A
Scenario To objective is verify that the application can handle the projected growth
Objective and to discover the breaking point of MERCURY SAMPLE PROJECT
Online
Steps 1. Ramp up to 150% of peak load volume and thereafter continuously
increase load until breaking point is found
Break point – When the error rate is more than 20% or response times
are really high
Entry Criteria All the Monitors are in place
Test Data is set-up
Peak Load test completed successfully
Exit Criteria Test completion report is agreed upon by stakeholders
Endurance Testing
It is also called as Soak Test.
This test is performed for long periods of time (8 hrs, 16 hrs, 1 day, 3
day) with expected user load.
Purpose of this test is to identify performance bottlenecks like
Memory Leaks, connection leaks etc.
Scenario Soak Test – MERCURY SAMPLE PROJECT
Name
Scenario Type Endurance – Duration 8 hour.
Scenario To discover memory issues and bottlenecks that might occur under daily
Objective usage of the application
Steps Steady state Test is maintained for 8 hour with half the Peak Load.
Entry Criteria All the Monitors are in place
Test Data is set-up
Peak Load test completed successfully
Exit Criteria Test completion report is agreed upon by stakeholders
Scalability Testing
Test is performed using the user load considering the growth of the
application under test down the years.
This test is performed to check the capability to scale up or scale out
in terms of User Load.
Volume Testing
It is a load test except that huge data populated in the database.
The data populated in the DB is expected down the years.
Purpose of this test is to check if there is any change in the response
time with increase DB volumes.
Performance Testing Lifecycle Stages
Non-functional Requirement Gathering and
Analysis
Test Requirement Test Strategy and
Planning
Test Plan Test Design and
Development
Iterations
Test Scripts Test Execution
Bottleneck
Analysis
Test Results Test Result
Analysis
Final Test Reports
Test Reports
Workload Load Modeling
Work load
Includes the number of virtual users and the volume of transactions per user
Estimation for number of users required for a given volume or vice-versa
Collect information on
various combination of business transactions
For each business transaction
information collected is probable number of users performing same transactions simultaneously
during normal operation
peak time operations.
Refer to below work load modeling example
Little’s LAW
N = X * (Rt)
N = No. of Concurrent Users
X = Throughput (TPS)
Rt = Response time (Secs)
N = X * (Rt + Zt) (For system with think time)
Zt = Think Time
Industry standard for Zt is 10 secs
Rt + Zt = Script Execution Time
Little’s LAW
For Work load Modeling, Pacing is required to be calculated. So the formula
would be
N = X * (Rt + Zt + Pacing)
Rt + Zt = SET
NFR Gathering
Infrastructure Details
As part of NFR gathering, PT team is supposed to capture the configuration details of
both production environment and performance testing environment
Data collected
Operating System of each server– OS version
Hardware configuration of each server – Number of CPU’s, speed, memory etc..
Details of server clusters.
Support software on each server – Web server, application server, database
Example
Apache Tomcat Server 8
1.6GHz processors with 32MB Cache, 16GB Memory, 100GB Hard Disks, HP-UX and
Java ES pre-installed
Transaction Details
As part of NFR gathering, PT team is supposed to captures list of
performance critical transactions.
What are Performance Critical Transactions:
Transactions which are executed Frequently.
Transactions which are critical for Business.
Transactions that are suspected to have high resource requirements
Transaction Details (Contd..)
Example
For a banking application, following transactions are critical
Account Summary Details
Checking Transaction History
Balance Transfer
Login
Logout
Following transactions are not critical
Change Password
Change Username
Change Theme
Order Checkbook
Open a new account
Scalability Related data
Few years down the line, there is chance that user load on the application
might increase because of the business growth.
Because of this increased load, the performance of the application might get
affected.
So, performance Testing team is supposed to capture the future user volume
growth.
Example
There is a 100% increase in user load annually.
Workload Related data
Application usage patterns and volumes for each performance Critical Transaction is
captured as part of this model.
This data is identified by
Interviewing Clients
Analyzing existing logs
Example
Data related to response times and other
metrics
As part of NFR Collection, Performance Testing Team is suppose to capture
Response Time (RT) related data.
After the Performance test execution, the test results are compared with the
SLA to determine if the application meets performance expectations
Examples -
When application is subjected to 1000 concurrent user load, “Login”
transaction should not take more then 2 seconds to complete
When application is subjected to 1000 concurrent user load, CPU
utilization of the WebApp Server should never cross 60%.
DB Data
Database volumes affect round times for operations that access the database
To simulate realistic load on the application the DB tables should be loaded
with sufficient number of records
The data retention model is used to capture the key DB tables and estimated
number of records in each of these tables
All this data will help in simulating realistic database volumes during test
execution
45
