SKIM at Sawtooth Software Conference 2013: ACBC Revisited
3 likes1,649 views
ACBC outperformed CBC in predicting TV preferences in holdout tasks. Researchers tested variations of ACBC and found that performance was robust across variations. Dynamic CBC performed better than standard CBC. Increasing the number of price cutpoints in piecewise estimation significantly improved ACBC performance. However, models overestimated brand preferences and underestimated the effect of price compared to actual choices. Future research aims to address discrepancies between predictions and choices.
SKIM at Sawtooth Software Conference 2013: ACBC Revisited
- 1. expect great answers ACBC Revisited 2013 Sawtooth Software Conference October 2013 | Marco Hoogerbrugge, Jeroen Hardon, Chris Fotenos
- 3. Respondents first complete a build your own (BYO) exercise to reveal their ideal product 3
- 4. Next they are shown a screening section where they can accept or reject several possible purchase options 4
- 5. Within the screening section, respondents are asked whether certain levels of attributes are ‘unacceptable’ or ‘must-have’ for them 5
- 6. Given some insight into the respondents’ preferences, they finally see a series of choice tasks more aligned with their consideration set 6
- 7. Reduced design spaces for each respondent • In highly complex categories with many possible concepts ACBC helps to collect data on the combinations that are most relevant to each respondent. Orthogonal designs are created with the emphasis on the relevant sphere for each respondent at the cost of D-efficiency of the design at individual level. Design Space Respondent 1 Attribute B • Design Space Respondent 2 Design Space Respondent 3 Attribute A 7
- 8. Aimed at getting more relevant choice data in big and complex markets • Big markets say, you have a simulator with 50 products (e.g. 7 brands x 7 tiers) • In regular CBC tasks a respondent chooses between concepts of different brands and different tiers • In ACBC, respondents choose between brands within a tier, or between tiers within a brand, or even more refined • Complex markets • Multi-attribute products (not just one SKU attribute) • Optional: price dependent on all or most underlying product features 8
- 9. To assess and search for ways to improve ACBC, we tested several variations of it along with other methodologies in the US TV market. 9
- 10. Our thoughts behind also including CBC legs • What are possible success factors and potential weaknesses of ACBC? And can we apply them also within CBC? • Reduced design space per respondent • Increased utility balance in every choice task • If ACBC is (perhaps) too extreme in applying these properties, CBC + one of these properties might become more successful. • Reduce design space Dynamic CBC (see later) • Increased utility balance CBC with between concept prohibitions 10
- 11. Our thoughts behind improving ACBC Price Quality Actual products in the market simulator 11
- 12. Modern televisions contain many features which creates a wide price spectrum and more complex purchase decision for shoppers Brand Screen Size Screen Type LED, LCD, LED-LCD, Plasma Resolution 720p, 1080p Wi-Fi Capability Yes/No 3D Capability Yes/No HDMI Inputs 0 – 3 HDMI Connections Price 12 22”, 32”, 42”, 52”, 57”, 62”, 67”, 72” $120 - $3,500
- 13. To benchmark and improve ACBC, we tested it against 7 other methodologies, including slight variations of ACBC itself Methodology Description A CBC Standard CBC B ACBC Standard ACBC C ACBC ACBC with price included in the unacceptable questions D Dynamic CBC SKIM-developed version of CBC where the design is adapted based on previous answers E ACBC ACBC with a more narrow tested price range of 90% to 120% F ACBC ACBC with 4 attributes varied from the BYO choice in the generation of concepts G CBC Price balanced CBC H ACBC Each leg 300 resps, total 2400 resps ACBC without the screener section, giving every respondent 12 total choice tasks In all ACBC legs, brand was not included in the BYO task. All legs showed respondents a summed total price for each concept (according to ACBC logic) 13
- 14. Dynamic CBC shows respondents a full set of CBC-like choice tasks with an ACBC-like design that is adapted based on their responses Like standard CBC, Dynamic CBC starts with a near-D-efficient design space for all respondents. • Respondents see a full set of choice tasks. • After 1/3 and 2/3 of the choice tasks, the design space is narrowed down based on the respondents’ choices in order to collect more data on the concepts that are most relevant to them. Design Space: Tasks 1-3 Design Space: Tasks 4-6 Attribute B • Design Space: Tasks 7-9 Attribute A 14
- 15. Following the ACBC/CBC exercise, respondents were shown a series of three holdout choice tasks • • 15 The holdout tasks presented respondents with concepts that were similar in their features and total prices. With a large number of products in every holdout task, this is much more like a real simulator than a usual holdout task of three or four concepts!
- 16. Conjoint models can seldom predict one clear ‘winner’ in these holdout tasks • Example of a share of preference distribution (model prediction) for a respondent: concept1 1.48% 0.03% concept12 0.88% concept3 0.65% concept13 29.78% concept4 0.33% concept14 0.95% concept5 4.68% concept15 27.31% concept6 0.18% concept16 12.99% concept7 4.67% concept17 4.67% concept8 16 concept11 concept2 • 0.64% 2.40% concept18 0.17% concept9 1.75% concept19 4.28% This same problem may well occur with actual simulators with many products! concept10 0.07% concept20 2.11%
- 17. Success metric: mean SoP So we thought it better to look at the means of the predicted share of preference of the concepts that the respondent chose in the three holdout tasks. 17
- 18. First results: respondents enjoy taking a (well designed) conjoint experience 18
- 19. Respondents liked taking the ACBC and CBC exercises with not many finding the tasks too difficult to complete “Thank you so much for letting me know about different types of TVs. Now I have an idea what is the best TV I should buy within 6 months.” (CBC leg) Standard CBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO Price balanced CBC ACBC without a screening section 0% I liked filling in this questionnaire 19 20% 40% 60% 80% 100% It was difficult for me to fill in this questionnaire
- 20. Respondents liked taking the ACBC and CBC exercises with not many finding the tasks too difficult to complete “I liked the format used in this survey. It was not confusing and questions were based on answers that I had previously given rather than random repetition.” (ACBC leg) Standard CBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO Price balanced CBC ACBC without a screening section 0% I liked filling in this questionnaire 20 20% 40% 60% 80% 100% It was difficult for me to fill in this questionnaire
- 21. Main results 21
- 22. All ACBC legs outperformed all CBC legs though there is not a significant difference between different ACBC variations ACBC without a screening section ACBC w/ price range of 90%120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 10% 20% Mean SoP of Selected Holdout Task Concept 22 30%
- 23. Intermezzo All legs showed respondents a summed total price for each concept (according to ACBC logic) • • 23 Summed price = (summed value of attribute levels) * random number • Price becomes a continuous variable Consequently we applied piecewise utility estimation for price in all legs • Number of cutpoints to be defined before running HB. ACBC allows up to 12 cutpoints.
- 26. Adding more cut points in the piecewise estimation (more granularity in price utilities) significantly increases the performance, especially of the ACBC legs! ACBC without a screening section ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 10% 20% Mean SoP of Selected Holdout Task Concept Original Analysis with 12 cut points Analysis with 25 cutpoints 26 30%
- 27. We were thrilled about some of these findings but disappointed about others • + • Omitting the screening section in ACBC isn’t bad • A narrower price range in ACBC isn’t bad • More cutpoints in piecewise estimation improves ACBC • All models beat random choice by a factor 3-4 • +/• Dynamic CBC performs significantly better than standard CBC 27
- 28. We were disappointed about some of these findings but thrilled about others • • None of the alternative ACBC legs outperformed current ACBC » The flipside of the coin is that ACBC is actually very robust! • Price balanced CBC didn’t outperform standard CBC • The absolute value of the mean SoP is actually quite low (~20%) » This may well be the case in actual simulators with many products too! 28
- 29. Mean Squared Error in a holdout task MSE = ∑ (Aggregate actual share – Aggregate mean predicted share) 2 concepts Or √ MSE = √∑ ( (Aggregate actual share – Aggregate mean predicted share) 2 ) concepts 29
- 30. MSE as a success metric confirms previous conclusions ACBC without a screening section ACBC w/ price range of 90%120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC Price balanced CBC CBC Standard CBC 0% 1% 2% 3% 4% 5% 6% √(MSE), average of three holdout tasks 30
- 31. One concept in one holdout task causes a large amount of the high MSE scores • Actual holdout share (aggregate level): 43% • Predicted holdout share: • Standard CBC leg is worst (predicts 8%) • CBC with price balance predicts 17% • ACBC modules are better don’t come very close (avg prediction 27%) 31
- 32. Results with and without this one concept ACBC without a screening section ACBC w/ price range of 90%-120% ACBC w/ 4 atts varied from BYO ACBC Standard ACBC ACBC w/ price in unacceptables Dynamic CBC CBC Price balanced CBC Standard CBC 0% 1% 2% Excluding one concept 32 3% 4% 5% 6% √(MSE), average of three holdout tasks Original Analysis
- 33. A step aside Cluster analysis on SoP predictions leads to a conclusion but not to a solution 33
- 34. Cluster analysis across the legs leads to two layers in respondent grouping All respondents Respondents who are predicted to buy a ‘low end’ TV (1/3 of sample) Respondents who are predicted to buy a ‘mid/high end’ TV (2/3 of sample) Different respondent groups for whom Different respondent groups for whom a particular brand is predicted a particular brand is predicted 34
- 35. Example: the high end Vizio cluster Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% LG Panasonic Samsung Sony Vizio Actuals of holdout task1 Actuals of holdout task2 Actuals of holdout task3 0% 35 20% 40% 60% 80% 100%
- 36. The actual shares in the holdout tasks are much lower for Vizio Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% 30.1% Actuals of holdout task1 35.2% Actuals of holdout task2 25.0% Actuals of holdout task3 0% 36 20% 40% 60% 80% 100% LG Panasonic Samsung Sony Vizio
- 37. Price plays a much bigger role in the actual choices than predicted Mean predicted SoP for holdout task1 52.7% Mean predicted SoP for holdout task2 51.9% Mean predicted SoP for holdout task3 50.6% 30.1% Actuals of holdout task1 35.2% Actuals of holdout task2 LG Panasonic Samsung Sony Vizio 25.0% Actuals of holdout task3 0% 20% 40% 60% 80% 100% Panasonic has cheapest product within the highest tier of holdout 1, Samsung 2nd cheapest. Samsung has cheapest product within the lowest tier of holdout 3. 37
- 38. This same phenomenon applies to all clusters • • 38 Brand is apparently estimated as too important, at the cost of price. This is perfectly in line with the most extreme mismatch in a concept that we saw before in the MSE calculation: the share of the cheapest product was greatly underestimated in this concept.
- 39. Another analytical approach (regression) leads to the same conclusions Dependents: actual holdout answers Aggregate multinominal logistic regression Explanatory: SoP predictions for all holdout tasks This aggregate regression leads to: - Positive coefficients for ‘own’ SoP of a concept, as expected - Also positive coefficients for SoPs of concepts with the same TV of competing brands in order to partially undo the brand emphasis in the own SoPs 39
- 40. Conclusions 40
- 41. Conclusions • 41 ACBC again outperforms CBC (in a complex market).
- 42. Conclusions • ACBC again outperforms CBC (in a complex market). • We’ve seen no significant differences within the tested ACBC variations • This indicates that the performance of ACBC is robust. • It seems like ACBC can be made more efficient (omitting screener) with the same performance. 42
- 43. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • 43 ACBC again outperforms CBC (in a complex market). Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC.
- 44. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC. • 44 ACBC again outperforms CBC (in a complex market). Performance of ACBC improves quite a lot by increasing the number of price slopes >11.
- 45. Conclusions • • We’ve seen no significant differences within the tested ACBC variations • Dynamic CBC performs much better than standard CBC; further refinement may lead to results close to or better than ACBC. • Performance of ACBC improves quite a bit by increasing the cutpoints for piecewise uitility estimation >11. • 45 ACBC again outperforms CBC (in a complex market). Respondents like a (well designed) conjoint survey better than we had thought, including standard CBC.
- 46. Directions for future research • Finding solutions for the discrepancies that we found between actual holdout answers and conjoint predictions • Developing more variants of Dynamic CBC • Double checking on existing data with holdout tasks • Increase number of cutpoints in piecewise estimation • Cluster analysis approach with predicted SoP 46
- 47. contact us or follow us online! Marco Hoogerbrugge Research Director [email protected] Jeroen Hardon Senior Research Consultant [email protected] Chris Fotenos Project Manager [email protected]