Performance Monitoring at Spreadshirt

Performance Monitoring
at Spreadshirt

Performance has an
impact on conversion rate ...

http://de.slideshare.net/devonauerswald/walmart-pagespeedslide

… and other business metrics
● bounce rate
● cart size
● revenue
● time on site
● page views
http://www.soasta.com/whitepapers/time-is-money-the-business-value-of-web-performance/
Time is Money, The Business Value of Web Performance, Tammy Everts
● user satisfaction
● user retention
● organic search traffic
● brand perception
● productivity

How can performance be measured?

Core Metrics Overview
time to first byte

time to first byte start render time

above the fold time

above the fold time page load time

Metric Details
time to first byte start render time page load time fully loaded
above the fold time

Metric Details
backend frontend

Metric Details
DNS lookup
Init TCP connection
Server processing
Content download

Metric Details
Number of resources and transfered bytes +
order of resources

Which performance tools can be useful in
software development lifecycle?

Software Development Lifecycle
Lifecycle image taken from SOASTA tool overview

Data Analysis &
Optimization Tools
Synthetic Performance Monitoring &
Real User Monitoring (RUM) &
Application Performance Monitoring (APM)
Load Testing &
Optimization Tools
Performance Tools

Performance Tools at Spreadshirt
Grafana/Graphite

Performance Tools at Spreadshirt
Data Analysis
Synthetic Performance
Monitoring
Real User Monitoring
(RUM)
Grafana/Graphite
Application Performance
Monitoring (APM)

Why do we have an
Application Performance Monitoring tool
and why isn’t it enough?

Machine
Application Performance Monitoring (APM)
Service
Monitoring
Server
Monitoring Data
Send Metrics
Write Monitoring Data Read Monitoring Data
and Aggregations
Browser
Developer
Request
Send Metrics
Service
Agent
Machine
Agent
Monitoring
Website/
Application

Pros
● Good overview on performance of monitored machines and services
● Machine level information on cpu, memory, disk usage
● Service level information like JVM heap usage, number of threads, etc.
● Resource level information on response times, throughput, error rates
● Analysis tools allow to quickly find performance problems
● End-to-end drill down on a per request basis allows to quickly find root cause of
problems

Cons
● Complex to set up in a microservice environment as each service and machine needs
to be monitored with an agent
● No information on how pages load in real browser environments
● No information on how my DNS provider or CDN works
● No information on how 3rd party libraries behave in browser

Application Performance Monitoring
provides view on internal service and
machine performance
synthetic and real user monitoring
and why do we need both?

What is the difference between
Synthetic and Real User Monitoring

Synthetic Performance Monitoring
Spreadshirt
Website
Monitoring
Server
Monitoring
Website
Monitoring Data
Poll
Page
Write Monitoring Data Read Monitoring Data
and Aggregations
Browser
Developer
Request

Pros
● Provides information about page load times in real browsers
● Controlled test environment
● Defined test locations
● Defined frequency
● Traffic independent
● Allows to check availability, speed, errors
● Allows to monitor 3rd party libraries, DNS provider and CDN

Cons
● Only information about configured pages
● No information about speed, errors on different customer environments like
○ browser
○ device
○ country
○ operating system
○ ...
● No information about traffic distribution
● No connection to business metrics (like conversion rate)

Real User Monitoring (RUM)
Spreadshirt
Website
Browser
Monitoring
Website
Monitoring Data
Request
Page
Write Monitoring Data
Read Monitoring Data
and Aggregations
Browser
Developer
Request
Monitoring
Javascript
Monitoring
Server
Send
Beacon

Pros
● Runs on end users browsers and collects information about speed and errors on
different
○ browsers
○ devices
○ countries
○ operating systems
○ …
● Allows to track real user traffic in real time
● Allows to track business metrics (conversion rate) in connection to performance
metrics
● Allows to track 3rd party libraries, DNS provider and CDN

Cons
● Provides data on pages with traffic only
● Uncontrolled environment
● Monitoring Javascript needs to be added to each page
● Single page apps (SPA) need to be monitored individually

Synthetic and Real User Monitoring
complement each other
and provide outside view on performance
synthetic and real user monitoring

Which features do the available
monitoring tools provide and how do
they work?

Application Performance Monitoring
light with Grafana/Graphite

Grafana/Graphite Features
● Available on https://grafana.sprd.net resp. https://graph.server.lej.sprd.net
● Light version of Application Performance Monitoring
● Allows to track all kinds of metrics (not only performance related ones)
● Allows to create custom dashboards
● Allows to visualize metrics in different ways (time lines, numbers, tables, ...)
● Allows to run more complex analysis with the available Graphite functions (like top
10 slowest resources, ...)
● Allows to define and visualize thresholds
● Does not allow to define alerts yet

Grafana Dashboard
Favorite dashboards Available dashboards
Search dashboard
Create new dashboard

Grafana Shopping Overview
Throughput metric as number
Response time metric with
exceeded threshold
Linked to dashboard
with details (on click)
Add panel

Grafana UMP Details
Response time metric p99 as timeline
Response time metric mean as timeline
Stacked throughput as timeline
Edit panel

Grafana UMP Details Configuration
Response time p99 metric
Response time mean metric

Synthetic Performance Monitoring
with Rigor

Rigor Features
● Available on https://monitoring.rigor.com
● Allows to configure uptime, real browser and even selenium-style checks
● Allows to measure from different locations with different frequencies
● Allows to define thresholds on measured metrics
● Provides dashboards to visualize performance data
● Provides http archive on measurement and visualization as waterfall diagram
● Collects all kinds of performance metrics
● Allows to create and distribute performance reports
● Provides tags to be used in dashboards and for conducting actions
● Allows to alert via e-mail, phone, sms, chat, web hook
● Provides status page
● Provides API

Rigor Dashboard
Real browser checks
Access check

Rigor Real Browser Checks
Add check
Configure checkAccess check

Rigor Real Browser Configuration
Configure URL
Configure frequency
Configure alert

Rigor Real Browser Check Configuration
Configure locations
Configure thresholds

Rigor Real Browser Check Dashboard
Switch view
Access measurement
Adjust timeDisplay events
System event

Rigor Real Browser Check Page Performance Dashboard
Select metric
Access measurement

Rigor Real Browser Check Domain Performance Dashboard
Select metric
Access measurement

Rigor Real Browser Check Waterfall Diagram
Open details

Rigor Executive Performance Dashboard
Access check

Rigor Operations Performance Dashboard
Access check

Rigor Custom Tag Performance Dashboard
Select tag

Rigor Custom Tag Performance Dashboard
Access check

Rigor Status Page - https://monitoring.rigor.com/status/218
Current load time
History

Rigor Custom Report Configuration
Select checks
Select date range
Select metric

Rigor Custom Report Visualization
Access check

Rigor E-Mail Alert
Access check
Current state

Rigor E-Mail Report
Access check

Rigor Rocket.Chat Integration
Access check

Real User Monitoring
with SOASTA mPulse

SOASTA mPulse Features
● Available on https://mpulse.soasta.com/concerto/Central
● Allows to configure applications per domain (e.g. for spreadshirt.de)
● Allows to configure page groups, metrics, timers and dimensions
● Collects performance metrics available to browsers
● Displays real-time performance data
● Provides dashboards to visualize performance data for different roles
● Allows to create custom dashboards
● Allows to filter data by all kinds of dimensions, e.g. browser, device, country
● Provides means to visualize complex correlations
● Allows to integrate data from other sources, like Rigor
● Allows to create and distribute reports
● Allows to alert via e-mail, chat, web hook …

SOASTA Applications
Create application
Access application

SOASTA Application Configuration
Access page groups

SOASTA Page Group Configuration
Configure page groups
Test page

SOASTA Metrics Configuration
Configure metric

SOASTA Timers Configuration
Configure timer

SOASTA Dimensions Configuration
Configure dimension

SOASTA Javascript Plugin
Add Boomerang Javascript
Add custom Javascript
code for handling SPA

SOASTA Javascript Beacon
Loaded Boomerang Javascript
Sent beacon

SOASTA Dashboards
Pre-built dashboards
Custom dashboards
Add dashboard
Access dashboard

SOASTA Summary Dashboard
Filters for drill down
Page load vs page views
Key metrics and timers
Filter statistics
Page load by country

SOASTA Summary Dashboard Configuration
Show available widgets
Show dashboard configuration
Drag & drop
to add to dashboard
Select to configure widget
Configure widget
Configure default filter

SOASTA Showcase Dashboard
Live beacons
Live page load and back-end time
Live statistics

SOASTA Custom Revenue Dashboard
Business relevant timers and metrics
Revenue vs. bounce rate
Page views vs. page load
Orders

SOASTA Custom Page Load Times Dashboard
Page load mean
Page load percentiles

SOASTA Custom Shopping Team Dashboard
Backend time vs. frontend time vs. page views
Page load time by page type
Load times from Rigor (external data)

SOASTA What-If Dashboard
Move sliders to see impact of
improvements on revenue
Sessions with average load time vs.
conversion rate per average load time

SOASTA Waterfall Dashboard
Found beacons for selected filters
Resource timings for selected beacon
Access to beacon data

SOASTA Waterfall Dashboard Performance Data

SOASTA Waterfall Dashboard Request Data

SOASTA Waterfall Dashboard Page Construction Data

SOASTA Reports
Add report
Custom reports

SOASTA Report Configuration
Configure schedule
Configure recipient
Drag & drop dashboard

SOASTA Report E-Mail
Configured dashboard
as image or link

SOASTA Alerts
Add Alert
Custom Alerts

SOASTA Alert Configuration
Configure data event
Configure recipient
Configure message
Drag & drop elements

SOASTA Alert E-Mail
Link to dashboard that
provides more details

SOASTA Rocket.Chat Integration
Link to dashboard that
provides more details
Alert from SOASTA

What if there are more complex
questions to answer?

Questions like ...
● On which page groups do I have performance problems?
● What are the most used path through my application regarding the defined
conversion goal?
● Which page groups are relevant regarding my defined conversion goal (e.g. order)?
● On which environments do I have performance problems (devices, browsers,
countries, operating systems)?
● Which page resources on my site are slow?
● Which page groups should I optimize first?

Data Analysis
with SOASTA Data Science Workbench
(DSWB)

Data Science Workbench (DSWB)
Soasta
DSWB
Importer
Monitoring
Data
Browser
Analyst
Request
AWS
Redshift
Cluster
Page Data
Resource
Data
Soasta
DSWB
(Julia)Workbench
Request
Data
Import

Data Science Workbench Features
● Available on https://spreadshirt.soasta-dswb.com
● Soasta DSWB is an environment that allows to analyze and visualize
performance data
● Runs on AWS Redshift Cluster with Julia as programming language
● Packages for Julia available to access databases, work with data, etc.
● Soasta DSWB provides additional packages for data analysis, drawing and
charting

SOASTA DSWB Overview
Select running notebook

SOASTA DSWB Notebook
Cell with default connection configuration
for spreadshirt.de data
Execute selected cell
(or Shift + Enter)
Tables with page and resource data
for spreadshirt.de
Selected time frame
Documentation

SOASTA DSWB Page Load Co-occurrences
Histogram that shows page views vs.
page load for selected time frame

SOASTA DSWB Page Load Co-occurrences
Visualization of factors that
influence load times

SOASTA DSWB Page Load Treemap
Treemap that visualizes load times in relationship
to page views for selected filter hierarchy

SOASTA DSWB User Path Sunburst Chart
Visualization of all user click paths
towards checkout success page
(means measured conversion goal)

SOASTA DSWB Conversion Impact Chart
Visualization of load time relevance of page
group in order to reach conversion goal
(means reaching checkout success page)

SOASTA DSWB Conversion vs Load Time Chart
Visualization of average session load time for converted and unconverted
sessions (means sessions that reached checkout success page or not) vs
conversion rate for average session load time

SOASTA DSWB Data Access
Visualization of plain beacon data for spreadshirt.de based
on an SQL executed against AWS Redshift Cluster

So, which tool should I use
for what again?

Conclusion
● Use Grafana/Graphite (and Icinga) to get information about internal services like
response times, throughput and error rates
● Use tools like JProfiler to debug performance problems
● Use Rigor to get outside view on selected pages regarding page response times and
availability
● Use SOASTA mPulse to get extended view on page load time behavior in real user
environments
● Use SOASTA DSWB as planning tool to answer more complex questions regarding
performance and plan the next steps

Performance Monitoring at Spreadshirt

More Related Content

Viewers also liked

Similar to Performance Monitoring at Spreadshirt

Recently uploaded

Performance Monitoring at Spreadshirt