Download as PDF, PPTX
![WEBSITE CONVERSION OPTIMIZATION:
OPTIMIZELY [JONI SHOWS]
22
Stockmann.com is not performing.
Let’s go to:
https://www.optimizely.com/ab-
testing/
Let’s do these changes:
1. remove second menu
2. move ”back to school” up
3. change logo to black and white](https://image.slidesharecdn.com/digitalanalyticslecture10-150818190211-lva1-app6891/75/Digital-analytics-Optimization-Lecture-10-23-2048.jpg)
Uploaded byJoni Salminen
Digital analytics: Optimization (Lecture 10)
The document outlines the importance of digital analytics and optimization in information technology, particularly at Aalto University. It covers various optimization techniques, such as A/B testing and conversion optimization, emphasizing the need to focus on user experience and data-driven decisions. Additionally, it discusses strategies for identifying and implementing effective changes to improve business performance through systematic testing and analysis.
More Related Content
Similar to Digital analytics: Optimization (Lecture 10)
Digital analytics: Optimization (Lecture 10)
- 1. Information Technology Program AaltoUniversity, 2015 Dr. Joni Salminen [email protected], tel. +358 44 06 36 468 DIGITAL ANALYTICS 1
- 2. Approaching R • DownloadR: (google ’download r’) • Download R-Studio: (google ’download r studio’) • Install both. 1
- 3. Just one questionfrom my friend… 2 What’s the most important application of analytics?
- 4. OPTIMIZATION 3 You could have alsoanswered ”decision-making”, but this is how Joni wants it today.
- 5. What is optimization? •A constant process of improving an object or set of object in order to better the values of key metrics tied to business performance. • i.e., making changes to make more money. 4 It’s always about the green$, baby.
- 6. The many facesof optimization • Google → search-engine optimization • Facebook → EdgeRank optimization (newsfeed optimization) • Website → landing page optimization, conversion optimization • etc. • In each, content is a common denominator. Systematic testing for finding compatibility between content and audience. • (Content = product, message, blog article, social media post) 5
- 7. Whenever there issomething to measure, it can be optimized. • rank on a website • popularity of a social media post • people’s purchase behavior 6 If you can’t measure it…
- 8. In many situations,you seemingly optimize for an algorithm. However, almost always the algorithm follows human decision- making. • e.g., most popular search results rank higher in Google, most popular posts in Facebook • therefore, eventually you always want to optimize for user experience (no tricks, remember UFO = user- focused optimization) 7 UFO… That’s clever!
- 9. Conversion optimization • Measurestaken to improve the likelihood of users taking a desired action after the click. • In practice, we modify websites: test different value propositions, visual elements, placements, layouts, features, or offerings. • Note: – Users are active agents in making concious decisions when browsing a website. – Conversion optimization is seen to take place mostly after the click (however, when does this not apply?) 8
- 10. Conversion differs fromchannel to channel Display Search engine Facebook Visitors 1000 1000 1000 Cost (€) 1000 1000 1000 CVR (%) 2 5 2 9 Yo! Check this out: • let’s buy 3,000 visitors for an ecommerce site from three sources, 1,000 from each • CPC = 1 € • CVR varies • calculate CPA – which channel has the lowest?
- 11. Conversion differs fromchannel to channel Display Search engine Facebook Visitors 1000 1000 1000 Cost (€) 1000 1000 1000 CVR (%) 2 5 2 CPA (€) 50 20 50 10 Yo! Check this out: • let’s buy 3,000 visitors for an ecommerce site from three sources, 1,000 from each • CPC = 1 € • CVR varies • calculate CPA – which channel has the lowest? Why do channels convert differently?
- 12.
- 13. You think landingpage A is better, Joni says B is better. What do we do? 12 control (A) test (B) +3,6 % Let’s test and see who was right. The structure of the test: Only change call to action, what do we learn?
- 14. A/B testing inone picture (Lillevälja, 2013) 13 One test version is made, and it is run in parallel with the original version (control). The website traffic will be divided randomly between the variations.
- 15. Some considerations inA/B testing • keep clear separation between manipulated and non- manipulated variables: the more variables manipulated, the more difficult it is to show a cause- effect relationship (preferably, test one variable (or design) at a time) • make sure you have enough data for statistical significance • have a hypothesis & theory (what do you think will be the result? why?) 14
- 16. Hypothesis formulation (Lillevälja,2013) 15 “Problem: Less than one percent of visitors sign up for our newsletter. Hypothesis: “Visitors don’t see the value in signing up for our newsletter. Adding three bullet points about the benefits will increase signup rates.” In this case, we would try placing a good summary of benefits the newsletter member would get from joining the newsletter. Even if the original version works better in your A/B test, you learned something about your visitors. You clearly defined why you did the test and can draw conclusions based on the outcome.”
- 17. A very commonerror in optimization • Toni: ”Oh, look Oliver, the ad version 1 has 50 clicks and the version 2 only 20 clicks! Let’s stop version 2, or what do you reckon?” • Oliver: ”You’re right buddy, let’s stop that sucker from wasting our impression. Joni would be so proud of us…” • Toni: ”Hooray!” 16
- 18. The reality… Version 1Version 2 Clicks 50 20 Impressions 300 100 17 What are the guys missing?
- 19. The reality… Version 1Version 2 Clicks 50 20 Impressions 300 100 CTR 17% 20% 18 What are the guys still missing?
- 20. A/B TESTING: ONLINEADVERTISING • Open the Excel file, and calculate if there is statistical significance. 19
- 21. The reality… Version 1Version 2 Clicks 50 20 Impressions 300 100 CTR 17% 20% Significant NO 20 The method applies to ANY performance-based marketing: • Google AdWords • Facebook Ads • display advertising • etc.
- 22. A/B testing andconfidence interval (Nanigans, 2014) 21 few clicks, large chance of error
- 23. WEBSITE CONVERSION OPTIMIZATION: OPTIMIZELY[JONI SHOWS] 22 Stockmann.com is not performing. Let’s go to: https://www.optimizely.com/ab- testing/ Let’s do these changes: 1. remove second menu 2. move ”back to school” up 3. change logo to black and white
- 24. How it worksin the background 1. JavaScript replaces content dynamically 2. matches the variations with selected audience 3. checks if they perform the desired action 4. calculates statistical significance and reports the winning design 23 Got it!
- 25. Where to applyA/B testing do-it-yourself? • long sales pages • home page (hide/remove elements, rearrange, change fonts, pictures, CTAs…) • as you can see, many are minor changes. Yet, e.g. lack of wordings and guiding can have a proportionally big impact. 24
- 26. For anything morecomplex than that, you need IT people, a management process, and a healthy culture of experimentation. Which is why 99% of companies don’t do split testing. 25 It’s just too much effort – data is for nerds, while golf is for managers.
- 27. Sequential vs. splittest • Sequential: first test A, then B (easier to execute) • Split: simultaneously A and B (usually harder to execute) • In both, traffic is split randomly between A and B. • Why is split usually considered better? 26
- 28. But, sequential testingcan be okay as well, if it’s not mission-critical stuff • pick a point of change • implement change (CMS) • keep change for a period (e.g., 2 weeks) • measure change before-after (equal periods) • if improvement, leave it • if worse, go back 27 Mission-critical means somebody’s life depends on it.
- 29.
- 30. Optimization strategy alternatives 1.Large disruption 2. Chained micro-gains …there’s also a mid-way of first getting large disruption (close to global maximum) and then decreasing the magnitude of tests (fine-tuning to global maximum) 29
- 31. Optimization strategy 1. Capturelow-hanging fruits 2. Go detailed ”as campaigns mature and settle into a steady state, the amount of “low-hanging fruit” gradually decreases. This necessitates a shift of focus from speed to accuracy, which means that it becomes increasingly important to make solid, data-driven decisions.” (Yamaguchi, 2013) 30
- 32. Consider two things! a.SIGNIFICANCE of change b. MAGNITUDE of change • not significant = not reliable • no big difference = not that important 31
- 33. Like any otherinvestment decision, the potential gains to run a test need to exceed its costs (time, money, effort) If small expected gain and cumbersome implementation, don’t do. If vice versa, do. Something in the middle, consider case by case. 32
- 34. Optimization process 1. Geta corporate buy-in: secure resources, access, set the right expectations 2. Identify low hanging fruits. (How? By spotting anomalies, such as bottle necks.) 3. Choose a very simple KPI you want to optimize for 4. Choose other metrics that support that KPI 5. Then operationalize, i.e. make a plan on how to improve each metric 6. Don’t forget – it’s okay to make errors; you’re optimizing many many times, so individual fuck-ups regress to mean in the long run. 33
- 35. Optimization via funnelstage analysis a. If top of the funnel is the bottleck for conversion, then optimize for awareness metrics (e.g., reach, frequency, CPM) b. If bottom of the funnel is the bottleneck for conversion, then optimize for direct sales metrics (conversion-%, avg. basket) • Ideally, you want to take both into account, but practice has shown it’s best to focus on one thing at a time. Either first build good environment for conversion and then start to drive traffic, or then start to drive traffic and iterate quickly as you get live data. • (remember, in a given point in time, only select people need your product.) 34
- 36. How to identifypoints of improvement? • look at industry best practices: what others have tested • identify loopholes and bottle necks with your analytics: – pages getting the most traffic – landing pages with most value per visit – biggest exit pages – pages with biggest bounce 35
- 37. What can youtest? • website layout • value propositions (e.g. price vs. selection) • copy text (message formulation, use of punctuation, CTAs) • political slogans • target audiences • visuals (images: placement, size, content) • forms (length, number of fields) • product contents & pricing 36 Oh boy, sky is the limit.
- 38. As you cansee, conversion optimization is not (only) some geeky stuff like changing button colors. It is driven from the premise that ”people are not stupid”, and therefore involves all the magic tricks marketers have on their sleeves (e.g., pricing psychology, bundling, scarcity, social effects). 37 Every day, I admire marketers more and more.
- 39. PIE: potential, importance,ease (Goward, 2013) 38 Brainstorm. Give value for every item on PIE dimensions. Calculate the average of each item. Prioritize based on averages.
- 40.
- 41. The anatomy ofa landing page 40 social media indicators testimonials / customer feedback certificates / referrals call-to-action pictures copy text
- 42. Adhering to conventions 41 convention= element and position to which a customer is accustomed (i.e., a ruling practice) shopping basket top-right logo top-left categories left side products in the middle campaign banner above products search in header contact info & help in footer Do not deviate from conversions without a good reason. (Confuses users.)
- 43. Full of socialproof (Amazon, 2012) 42 reviews indicate product quality (”fake it till you make it”) purchase frequency tells of the choices by other customers targeted recommendation is based on other users’ behavior Remember, people are social animals (Aristotle, ~300 BC)
- 44. Trust seals &brand solicits (reference proof) 43 reference customers, brand spill over effects ”If big boys trust it, I guess so can I”
- 45. Authentic pictures “Marketers oftenuse stock images that imply nothing about the value of the offer, settling for ‘pretty’ images that make no clear connection to the offer’s core value. Remember, images that say nothing are worth nothing. The force of an image increases with its authenticity. Images can bring a realism that reduces the ‘virtual distance’ between an offer’s value and the recipient’s perception of that value. Therefore, marketers must attempt to find images that help the visitor see and experience the core value of the product.” (Marketing Experiments 2010) 44 Always show real people, not some stock photos!
- 46. ”Marketing bullshit” –avoid this! 45 • Be specific. ”industry leader”, ”best data” – BS! • murky qualitative claims • information value zero • Watch the tone. ”Your hunt is over!” Do I know you? • The customer should be given choice in being contacted.
- 47. Concretizing value propositions ”Tooffer an example, commonly used PPC terms such as ‘biggest’ mean nothing out of context. Instead of wasting ad space with unsubstantiated generalities, choose to tell the user, ‘106,000+ new users in 2010.’ Don’t tell the user that something offers ‘fastest downloads’ when it is more effective to say, ‘Download time is X seconds.’ Such superlatives offer the user no information that would encourage a clickthrough.” (Marketing Experiments 2011) 46
- 48. It’s like allresearch. You have some data, and then you generalize. But yeah… All this is just ”best practice”, something that other people tell it’s the best. It may appeal to your marketer’s intuition, but there are books written about how intuition can fail. (So, let the data tell what’s right in your case.) 47
- 49. An example ofmarketer’s intuition 48 1,15%
- 50.
- 51. Survivor bias todo and case studies of A/B testing • “An A/B test that worked for another company isn’t always repeatable. Don’t blindly copy tests from success stories expecting similar results. Instead try to understand why it worked for them, and what lessons you can draw from it.” (Kogan, 2014) • usually, only the most successful tests are published (cf. lottery) • these are a small minority • therefore, the majority will not get reported at all! • the result: over-optimistism over A/B testing (survivor bias). 50
- 52. What’s the purposeof testing? To understand why. “Marketers should not assume that a popular page design will be effective for every situation. The problem with ‘best practices’ or ‘best designs’ is that they rarely work across the board. It is more important to move beyond understanding the ‘what’ of page layout, and use testing to attain the deeper understanding of ‘why’.” (Marketing Experiments 2010) 51
- 53. Discovering, through areliable test, which alternative yields the best response, enables you to generalize that alternative beyond the test. For example, test five different copy texts which are ”the nominees” for new slogan in Facebook, and based on the results you can say which one resonates best with the audience (before launching it in mass marketing). 52
- 54. Testing with ”focusgroup” mentality (small budget, limited reach), and then expanding to the mass marketing. (Nothing new here, it was already done by Ogilvy and Hopkins starting from 1920s, but now it’s hell of a lot easier, cheaper and faster. And yet most copywriters don’t do it!) 53
- 55. Testing does notautomatically lead into improvements… 54 Why?
- 56. This is why: •most tests fail to show improvement (survivor bias) • small changes → small improvements (garbage in, garbage out) • postponing or cancelling the proposed changes (HiPPO) 55 There are always complications.
- 57. Smaller differences takelonger to clear. Instead, you would like to make many tests in a given time period, not only one. CVR (%) Change (%) Required sample size per variant Days to clear Test 1 3 5 72,300 72 Test 2 3 10 18,500 18 Test 3 3 30 2,250 2 56 (Johns, 2015)
- 58. ”Goodbye Google” (aka41 shades of blue) • “Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.” (Bowman, 2009) • Is this a case of local maximum problem? What do you think? 57
- 59. How to survivethe local maximum problem? 58
- 60. Problem with A/Btesting: HiPPO • ”Listen, I know what works. Let’s do like this and that’ll be the end of it.” • ”…okay.” 59 I wish I didn’t always cave in!
- 61. Some theoretical considerations 60 THEORY??Dude, who cares! (Just kidding.)
- 62. Compounding benefits ofoptimization (Marketing Experiments, 2005) • Base level in the example: – monthly sales = 100,000 $ – costs = 85,000 $, monthly profit 15,000 $ • The company implements nine improvements in nine months (one per month). 61
- 63. Compounding benefits ofoptimization (Marketing Experiments, 2005) 62 The benefit table Improvements (1 per month) Gain Profit per month Change in profit (0. Base line) N/A $15,000 0% 1. Improving PPC ad copy 5% (CTR) $19,500 30% 2. Decreasing CPC 5% (CTR) $19,999 3% 3. Optimizing landing page 5% (CVR) $30,249 51% 4. Optimizing order form 5% (CVR) $35,761 18% 5. Streamlining website copy 5% (CVR) $41,549 16% 6. Adding upselling and cross-selling options 5% (Sales) $47,627 15% 7. Lowering price 5% (Sales) $49,988 5% 8. Changing shopping cart flow 5% (Sales) $56,487 13% 9. Adding trust indicators 5% (Sales) $63,312 12%
- 64. Compounding benefits ofoptimization: example (Marketing Experiments, 2005) • Results: – Extra profit = 48,312 $ – Compounding benefit = 322 % – (Simple benefit = 163 %) • In other words, there is an ”interest to interest” effect in optimization. 63
- 65. Implications for amarketer • Make systematically small improvements • Focus on both before and after the click 64 I know it’s boring at first, but seeing the fruits of your labor motivates!
- 66. Scaling of optimization:example • Three companies A. 1000 visitors per day B. 10,000 visitors per day C. 100,000 visitors per day • Other parameters – Conversion prior to optimization = 1 % – Conversion after the optimization = 2 % – Fixed monthly cost of optimization = 2000 € – Avg. basket = 50 € • Let’s optimize. What is the ROI for each company? 65
- 67. Scaling of optimizationefforts: results 66 0 20000 40000 60000 80000 100000 120000 A B C myynnin muutos myynti kiinteä kust per kk ROI: A = -50 % B = +400 % C = +4900 %
- 68. Multiplication argument (Nielsen,2008) a) “In a multiplication, if you want to increase the outcome by a certain percentage, you can increase any of the factors by that percentage. It doesn’t matter which factor is increased — the result will be the same. b) Thus, to double a site’s business, you can double the number of unique visitors. However, this would be very expensive, requiring that you more than double the advertising budget (assuming you’re already advertising under the most-promising keywords, and thus need to buy traffic from less promising or more expensive sources). c) Alternatively, you can double the conversion rate and achieve the same business improvement. (…) In most cases, it’s far cheaper to use 15% of your development budget than to more than double your advertising budget.” 67
- 69. Example: halving theadvertising costs • When you want to double sales, either double the advertising budget (if possible) or the conversion rate. • As the number of paid visitors increases, the more feasible conversion optimization becomes as an investment 68 Case A: Low conversion Case B: High conversion Ad spend 100,000 50,000 Visitors (CPC = 0.25) 400,000 200,000 Conversion 1 % 2 % Sales quantity 4,000 4,000
- 70. Return to eCommercemetrics (Fellman, 2015) • Visitors • Conversion rate • Average basket • Margin • Example (monthly sales): • 100,000 x 0.02 x 100 € x 0.40 = 80,000 € 69 Visitors Conversion rate Average basket Margin Dissect each, and consider how you can optimize. (Remember: this was the breakdown of metrics that were deemed important for ecommerce.)
- 71. Metrics for newsletters:tying the chosen metrics to the optimization process 70 Subscribers x delivery rate x open rate x click rate x conversion rate x margin = profit Ways to improve Lead generation tactics Bundling, high-markup items Conversion optimization Quality (mitigation of spam complaints) Headline optimization
- 72. Why is conversionalways low? (Regardless of optimization) • P(cv) = P(sees) x P(understands) x P(has need) x P(has money) • Let’s advertise to the whole of Finland! • Assume an optimistic 10% at each step • 5,000,000 x 0.1 x 0.1 x 0.1 x 0.1 = 500 • 500 customers, yey! • …but wait, the campaign costed us 50,000 €. Let’s see: 50,000/500 = 100 €. (Will I keep my job or not?) 71
- 73. Another way tolook at it… • At time point t, a sub-set m of total market M has the need for product x. In a world like this, most products are not finding a match in the market at the time of advertising. However, the long-term effect should be positive to justify the cost. • (Which takes us back to the question of choosing the lookback window.) 72
- 74. So, what canwe do? All else being equal, a marketer can increase conversion by improving targeting (driving more qualified traffic, e.g., focusing only on the end of the purchasing path) or by driving more overall traffic in the beginning of the purchasing path. 73 But will the quality of the traffic remain constant?
- 75. Potential solution: microconversions (Google, 2015) 74
- 76. Conclusions (1/2) • Conversionoptimization is feasible when C < avB x (S1 – S0), i.e. • the fixed cost of optimization (C) is smaller than the average basket (avB) times the change in sales quantity (S1 – S0), or when additional sales cover the costs. • When there’s a small number of visitors, the cost of optimization, whether in-house or agency, can easily exceed the benefits. However, what would be the exception? 75
- 77. ”every product hasan amazing dropoff of usage from when people first encounter it” (Chen, 2015) 76
- 78. Conclusions (2/2) a. Generally,the largest gains from conversion optimization emerge from removing bottle necks (which can be found by analysing users’ behavior) b. The more traffic and sales, the more lucrative it is for a company to invest in conversion optimization (because of the scaling effect) c. When chained, a large number of small improvements may generate a positive ROI (because of compounding benefits). 77
- 79.