Battle of the Scales: Examining Respondent Scale Usage across 10 countries

Battle of the Scales: Examining Respondent Scale Usage across 10 countries

• 1.
  Battle of theScales Examining Respondent Scale Usage across 10 countries By Melanie Courtright, Kartik Pashupati, and Annie Pettit
• 2.
  Background 2
• 3.
  Battle of theScales Background Scales are a cornerstone of market research They’re how we determine that: • 49% of people like Coca-Cola and 42% of people like Pepsi • Men like watching sports more than women do • How citizens feel about the government • Canadians like Shania Twain more than Brits… or do they?? 3
• 4.
  Battle of theScales Background 4
• 5.
  Battle of theScales So Many Client Questions…  Is there a difference in the reliability of attitudinal scales when using 4-point, 5- point, 7-point, and 10-point scales?  Does excluding a neutral point impact the answers?  Does labeling each point vs. only the end points produce different results?  Can we replicate and extend the results of previous research on the impact of cultural factors on response styles?  Do scales with greater variance (e.g., 7-point and 10-point scales) reduce Extreme Response Style (ERS) compared to lesser variance (e.g., 4-point and 5-point scales)?  Conversely, do scales with greater variance (e.g., 7-point and 10-point scales) produce a greater incidence of Medium Response Style (MRS) compared to lesser variance (e.g., 4-point and 5-point scales)? 5
• 6.
  Battle of theScales Research Plan 6
• 7.
  Battle of theScales Research Plan  7-minute attitudinal survey  Globally relevant topics  Mix of positive and negative wording  Scales with published measures of reliability  Include behavioral statements that should correlate to attitudinal questions  Simultaneously field across ten countries  Age and gender quota sampling by country  Same sample source throughout  Research Now’s proprietary Valued Opinions Panel (VOP)  First study to  Simultaneously compare the effect of multiple response options (4-, 5-, 7- and 10-point scales)  Using a large, census-balanced, multi-country sample 7
• 8.
  Battle of theScales Research Plan  Fieldwork conducted December 2012 and January 2013  All five scale options tested in each of ten countries Sample 4 5 point 5 point 7 10 TOTAL Sizes point LABELED UNLABELED point point Brazil 494 255 253 486 488 1976 China 496 256 251 501 499 2003 France 497 252 249 497 499 1994 Germany 496 249 250 499 499 1993 India 489 243 245 494 490 1961 Japan 497 251 249 497 498 1992 Mexico 496 249 250 493 497 1985 Russia 497 248 249 495 493 1982 UK 506 250 249 499 501 2005 US 498 251 249 497 500 1995 TOTAL 4966 2504 2494 4958 964 19,886 8
• 9.
  Battle of theScales Analysis Plan Three indices:  Extreme Response Style Index (ERSI) – Respondents who answered either extreme of scale were assigned a score of 1. Otherwise, they were assigned zero.  Acquiescence Response Style Index (ARSI) – Respondents who strongly agreed with an item were assigned a score of 1. Otherwise, they were assigned zero.  Medium Response Style Index (MRSI) – On scales with an odd number of options (i.e., 5 point, 7 point), respondents who answered exactly in the middle were assigned a medium response score of 1. Otherwise, they were assigned zero. Possible summary values ranged from 0.0 to 1. 9
• 10.
  Battle of theScales Use of Scale Results Hypothesis 1: There will be no meaningful differences in ERS, MRS or ARS indices between male and female respondents. Male Female ERS .43 .43 ARS .32 .33 MRS .28 .28 Result 1 Confirmed: Men and women do not differ in their response patterns. 10
• 11.
  Battle of theScales Use of Scale Results Hypothesis 2: There will be significant but not systematic differences in ERS, ARS and MRS Indices across the different age groups. .50 .46 .44 .44 .45 .43 .43 .41 .41 .41 .41  ERS (Extreme) .40 .35 .33 .33 .32 .32 .32 .31 .30 .28 .29 .29 .29 .29 .30 .28 .28 .28 .27 .27 .25  ARS (Acquiescent) .25 ▲MRS (Medium) .20 15-17 18-24 25-34 35-44 45-54 55-64 65-74 75-84 85+ Results 2 partially confirmed: There were significant differences but the differences were systematic. ERS and MRS gradually increased with age until 55-64, and then declined. The pattern for ARS (yea-saying) was reversed. 11
• 12.
  Battle of theScales Use of Scale Results Hypothesis 3: There will be significant differences in ERS, ARS, and MRS across the different countries. .60 .53 .53 .48  MRS (Acquiescent) .50 .45 .41 .41 .40 .40 .37 ▌ERS (Extreme) .40 .31 .29 .28 .28 .29 .29 .27 .26 .26 .27 .30 .33 .20 .10 .00 Results 3 confirmed: Respondents from Brazil and Mexico have the highest tendency to give extreme responses. Respondents from Japan have a significantly lower ERS Index, and significantly higher MRS (and ARS). 12
• 13.
  Battle of theScales Use of Scale Results Hypothesis 4: Individualism (Hofstede) will correlate positively with ERS and negatively with MRS. Individualism will equate to stronger, and therefore more extreme, opinions. .55 .49 .50  ERS (Extreme) .45 .41 .39 .38 .40 .35 .31  ARS (Acquiescent) .29 .28 .30 .27 ▲MRS (Medium) .25 .20 Low Medium High Individualism Results 4 not confirmed: Differences could not be attributed to individualism. Analysis actually showed a negative correlation with ERS. 13
• 14.
  Battle of theScales Use of Scale Results Hypothesis 5: Respondents in countries that are higher in masculinity (Hofstede) would exhibit higher ERS. .50 .46 .45 .41 .42  ERS (Extreme) .40 .34 .35 .32 .31 .30  ARS (Acquiescent) .30 .28 .28 ▲MRS (Medium) .25 Low Medium High Masculinity Results 5 not confirmed: While masculinity did affect the differences, the results were not in the expected direction. Countries with lower masculinity demonstrated higher ERS indices. 14
• 15.
  Battle of theScales Use of Scale Results 15
• 16.
  Battle of theScales Number of Scale Points Results Hypothesis 6: The number of scale points and scale labeling will affect ERS, MRS, and ARS. .50 .45 .45 .45 .42 .41 .41 .40  ERS (Extreme) .34 .33 .35 .33 .32 .33 .32 .31 .31 .30 .27  ARS (Acquiescent) .25 .20 ▲MRS (Medium) .20 .15 4 point 5 point 5 point 7 point 10 point labeled unlabeled Results 6 confirmed: 7- and 10-point scales saw fewer medium responses. ERS, MRS, and ARS were all lower for the 5-point labeled scale versus the unlabeled scale. 16
• 17.
  Battle of theScales Number of Scale Points Results Hypothesis 7: The number of scale points and scale labeling will have an impact on scale reliability. Cronbach's alpha # of # of 5 point Total Reverse 4 5 point unlabel 7 10 Scale Items Items point labeled ed point point Health Environment Sensitivity 8 1 0.8 0.8 0.8 0.8 0.8 Personal Health Responsibility 8 2 0.7 0.6 0.6 0.6 0.7 Motorcycle Helmet Mandate 2 1 0.7 0.7 0.7 0.7 0.7 Attitude toward helping others 4 0 0.9 0.85 0.9 0.9 0.9 (AHO) Material Values Scale (MVS 9) 9 2 0.8 0.75 0.8 0.8 0.8 Attitude toward Advertising in 7 4 0.7 0.7 0.7 0.7 0.7 General (AAG) Online privacy concern 2 0 0.5 0.5 0.5 0.6 0.55 Lie acceptability scale 8 4 0.8 0.8 0.8 0.8 0.8 Results 7 Not confirmed: There is no significant variation in the reliability of scales by number nor labeling of scale points. Note: For ease of reading, alphas were rounded. See the paper for precise values. 17
• 18.
  Battle of theScales Summary Findings  ERS, MRS, and ARS do not differ by gender, but do differ by age  Response styles vary by country – India, Mexico, Russia and Brazil are similar – The US and UK are similar, as are France and Germany. – Respondents from Japan are unique in terms of lower extreme and higher medium response styles  Reasons for country differences are not yet isolated – Need more research and – Need scale norms that are available on a multi-country basis.  Varying the number of response options – Does affect MRS – Does NOT impact scale reliability or ERS  Scale labeling did not impact scale reliability, but did impact ERS, MRS, and ARS 18
• 19.
  Battle of theScales Closing Thoughts READ THE PAPER! 19
• 20.
  Battle of theScales Closing Thoughts 20
• 21.
  Battle of theScales Closing Thoughts What about mobile? If number of options and labeling do impact results and screen space is a luxury on mobile devices… 21
• 22.
  Battle of theScales Closing Thoughts What about social media? Hideous Disgusting Abhor If number of Crap options and Yuck Dumb labeling do Huh impact results, Dunno Whatevs what is the Good complementary Nice Cool impact on textual Awesome data… Wicked Bomb 22
• 23.
  Battle of theScales More to Come! 23