Download to read offline
Uploaded byCliff Crocker
Walmart Web Performance Circa 2013
The document outlines a presentation on real user monitoring (RUM) at Walmart, discussing the importance of site performance for conversion rates and providing insights on optimizing page load times. Key findings reveal that slower page responses correlate with higher bounce rates and lower conversion, while setting performance targets led to measurable improvements. The analysis emphasizes that every second saved in load time can significantly enhance conversion rates and revenue.
More Related Content
![Creating Modern Metadata Systems [FutureStack16 NYC]](https://cdn.slidesharecdn.com/ss_thumbnails/djfinalfuturestack16presentation2-160815211822-thumbnail.jpg?width=640&height=640&fit=bounds)
![[2C6]Everyplay_Big_Data](https://cdn.slidesharecdn.com/ss_thumbnails/2c6everyplaybigdata-140930020825-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Walmart Web Performance Circa 2013
Walmart Web Performance Circa 2013
- 1.
- 2. Presentation Schedule • Cliff- 10-20m on RUM • Aaron – 10-20m on Monitoring in real-time • Balaji – 10-20m on Correlating RUM and Business Analytics • Final Q&A
- 3. Cliff Crocker - Performance -Reliability - Platform & Site Analytics Twitter: @cliffcrocker
- 4. shhhh….. • We arenot the fastest retail site on the internet today. IE 8 – Web Page Test Comp Index – Item Page Performance 11s – Fully Loaded time 4
- 5. What gets measured,gets done… • Before we start to optimize – see where our pain is and prepare to measure for success. • You never go on a diet without first stepping on the scale… 5 Source: Flickr
- 6. Synthetic Monitoring Pros – Technologyis great – Real Browsers (IE, Chrome, FF) – Built in Alerting – Charting/Trending – Waterfall analysis – Screen shots & headers – Object level detail Cons – False Positives – Impossible/impractical to measure everything – Fixed number of browser/OS combinations – Simulated bandwidth constraints at best – Too few data points for statistical relevance 6
- 7. ~2.2% of GlobalInternet usage - Thanksgiving this year 7
- 8. • If onlythose users could tell us about their site experience… 8 Source: Flickr
- 9. Enter boomerang.js -https://github.com/yahoo/boomerang boomerang.js with NavTiming (thanks Buddy & Phil) • Doc Complete, Page Response, Page Processing, TTFB, DNS, Connect & more • Cookie data (for parent domain) • Location (geolookup on IP) • Referrer • User Agent • Anything else you want to stuff into the beacon + 9 Source: Flickr
- 10.
- 11. Start with aurl group (i.e. ‘Item Page’) Find offenders that have the most impact
- 12. Identify usage patternsfor browsers (Chrome and Safari users more active on weekends) 12
- 13.
- 14. What we foundscared us… Home Page Performance – Jan 2012 14
- 15. Set some goalsand SLAs • Focus on ‘Page Processing Time’ First – see Golden Rule – 80% of time spent here (more like 90% for Walmart) • Look at backend & network response time SLAs later • Use 95th Percentile • Set Achievable SLAs • Revisit Monthly • Celebrate Wins! 15
- 16. Case Study: ItemPage • Problem: – Page takes ~24s for slowest 5% of users • Too many elements • Slow third party modules • Several other pagespeed ‘no-nos’ • Goal: Meet SLA for February – 20s (95th percentile) • Approach: – Scrum team dedicated to perf optimization for 1 sprint – Team pools resources and ideas - focuses on biggest bang 16
- 17. Success! 8s off the95th %tile 17
- 18.
- 19. RUM in NearReal Time
- 20. BEACON SERVER BEACON SERVERCLIENTCLIENT INTERNETINTERNET MAGIC SMOKE Howboomerang.js Works 1. client downloads JavaScript payload 2. client sends RUM payload to beacon 3. beacon server responds with HTTP 204 WHO CARES!WHO CARES! 20
- 21. Aaron Kulick - Performance -Founder: SF & SV Web Performance Group http://www.sfwebperf.org Twitter: @GoFastWeb WHO AM I?
- 22. Configuring boomerang.js isEASY Getting Actionable Data Is HARD 22
- 23. Where’s the data? Theinitial incarnation of RUM @ WMT leveraged Akamai at the edge. • data reliability issues • data availability issues • data “freshness” issues It works… for certain values of work Source: Flickr Source: Flickr 23
- 24. HUBBLEHUBBLE Enter Hubble JETTYJETTY DURABLE QUEUEDURABLEQUEUE APACHE FLUME NGAPACHE FLUME NG HDFSHDFS UDPUDP HADOOPHADOOP 24 REAL TIMEREAL TIME
- 25. The Bridge BetweenThen and Now • Etsy/StatsD - https://github.com/etsy/statsd – Timers – Counters – Simple Aggregation • Min, Max, Mean, Median, 95th Percentile • Whisper, Carbon & Graphite - https://launchpad.net/graphite – Scalable Realtime Graphing – RRD-like Storage Requirements + Federated Option – Supports Irregular Updates
- 26. The Now Bits HUBBLEHUBBLE UDPUDP StatsDStatsD AGGREGATION Graphite + Whisper Graphite + Whisper UDP/TCPUDP/TCP STORAGE + DISPLAY •Fixed some calculation problems in Etsy/StatsD • Added median calculation for timers • Incorporated multiple flush interval patches
- 27. Pretty Pictures RAW DATAFORMAT uswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0, 10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0, 11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0, 9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0, 9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0, 11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0, 10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0, 8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0, 14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0 uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0, 2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5, 2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5, 2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0, 2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0, 3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0, 3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5, 2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0 27
- 28. The Work Tomorrow… TheGood •Metric Throughput •Commodity Storage •Commodity Infrastructure The Bad •More Metric Throughput •Calculation Complexity •Web Sockets (pretty) •Metric Fan-out 28
- 29.
- 30. Is Page Performancea Factor of Site Conversion? And how big is it? February, 2012 v s
- 31. 31 Walmart.com - FunFacts • Reach –Millions of Shoppers/week. –Billions of page requests/year - Spikes up to 1500% –Billions of internal product search volumes/year • Scale –Millions of active product SKUs + Market Place –Millions of pages indexed in search engines • Complexity –1/4th of page contents served by partners, affiliates and Marketplace –Multiple departments, 10+ checkout paths Page Performance & Site Conversion – Feb 2012
- 32. 32 So, how doyou monitor?... Page Performance & Site Conversion – Feb 2012
- 33. 33 Few Industry Benchmarks… •Factoid 1: Large eCommerce site extensively A/B tested page performance and published a study showing 100 millisecond delay = 1% drop in revenue • Factoid 2: Search Engines A/B tested performance and found that a 500 millisecond delay caused a 20% drop in traffic. • Factoid 3: In an experiment across multiple retailers, a 1 second delay caused a 7% decline in conversion Page Performance & Site Conversion – Feb 2012
- 34. 34 So, how bigis it for Walmart.com? Page Performance & Site Conversion – Feb 2012
- 35. 35 Agenda • Phase 1– Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
- 36. 36 Agenda • Phase 1– Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 – Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
- 37. 37 Impact of siteperformance on overall site conversion rate…. Baseline – 1 in 2 site visits had response time > 4 seconds * Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds * Overall average site load time is lower for the converted population (3.22 Seconds) than the non- converted population (6.03 Seconds) Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 38. 38 @ Page level…. Pageload time is lower for Buyers compared to Non-Buyers * The Page load time is highest for certain pages - 6.38 secs when there was a conversion and 8.06 where there was no conversion. Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 39. 39 @ Department level…. Departmentload time is lower for Buyers compared to Non-Buyers * Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 40. 40 What about bounce? PageBounce Rate Vs Response Time * Key pages have high bounce rates which correlates with high T_Page as well * Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing pages. Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 41. 41 Bounce rates @department level…. Department Bounce Rate Vs Response Time * High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly higher Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 42. 42 Agenda • Phase 1– Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
- 43. 43 Phase 2 –Targets for Page Performance…… Conversion & Bounce Rate Impacts Drives Prioritization Note: Load Time here is the time taken from head of the page to page ready (T_Page) Page Performance & Site Conversion – Feb 2012
- 44. 44 Agenda • Phase 1– Baseline Measurement - Impact of Site Performance on Conversion, Bounce rates & Revenue • Phase 2 - Targets for Page Performance • Phase 3 – Optimization Results • Key Highlights & Takeaways Page Performance & Site Conversion – Feb 2012
- 45. 45 Phase 3 –Success Story…. First Win….and yes conversion had positive improvements… Page Performance & Site Conversion – Feb 2012
- 46. 46 • Page speedmatters for site conversion! • Monitor real user performance in a “Big Data” way!! • Every 1s improvement = Up to 2% increase in CVR • 100ms improvement = Up to 1% incremental revenue • SEO benefits for entry pages and reduce bounces • Test & Learn - Target segments and run A/B Tests focused on improving page performance Key Highlights Page Performance & Site Conversion – Feb 2012
- 47. We’re Hiring forEverything!!!! 47 Source: Flickr http://www.walmartlabs.com/open-positions/ https://walmartstores.com/careers/apply/?ba=eCom @cliffcrocker or @GoFastWeb