Carma internet research module scale development

Carma internet research module scale development

• 1.
  Scale Development forthe Web CARMA Internet Research Module Jeff Stanton
• 2.
  Scale Development forthe Web • …is just like scale development for paper and pencil instrument, except… • Diminishing response rates make shorter scales (3-5 items) more critical • Having catchy, interesting, easy-to-read item content also encourages persistence with your study • Reduced overall instrument length affects choices of the generality/breadth of measures
• 3.
  Scale Development Steps • Scale/concept development and definition (literature & researcher) • Item generation (subject matter experts) • Item review (subject matter experts) • Pilot test psychometrics (item variance, internal consistency) • Cull items based on statistical and judgmental criteria (researcher; subject matter experts) • Secondary pilot test with initial evidence of nomothetic network (researcher; subject matter experts) • Preliminary analysis of validation evidence (researcher; subject matter experts) • Validation with experimental evidence or multi-trait, multi-method matrix (researcher; subject matter experts) • Publication of psychometric and validity evidence (researcher)
• 4.
  Scale/concept development anddefinition (literature & researcher) • The development of any scale should begin with a literature review of related concepts or constructs • Based on ideas in the literature the researcher should develop a definition of the new construct to be measured • The new construct should be defined positively (what it is) and negatively (what it isn’t) • The rationale for creating the new construct and measure should be fleshed out at this time
• 5.
  Item generation (subjectmatter experts) • Armed with the construct definition, a panel of experts (faculty, students, industry experts, practitioners, etc.) can generate an initial pool of items • The pool should contain 5-10 times as many items as one expects to include in the final measure • One can use a range of brainstorming techniques to generate item ideas • Web surveys can be useful for collecting item ideas! • The response format should be considered at this time as well; depending on the construct, a Likert, frequency, intensity, pair-choice, checklist, semantic differential or other scale format may be suitable
• 6.
  Item review (subjectmatter experts) • Generally, after an initial item generation activity, one should using sorting techniques to organize the items into factors or banks • Sorting can also be used for review by new SMEs; reviewed items can be kept, held for editing or discarded • Final item pool should be presented with appropriate response format to a final set of SMEs prior to pilot testing
• 7.
  Pilot test psychometrics(item variance, internal consistency) • Without worrying too much about validity concerns at this stage, the items should be fielded for response by a group of appropriate participants • Generally, a minimum of responses per item fielded should be collected • After item data are collected, screened, and cleaned, calculate basic item statistics such as mean, variance, skewness, inter-item correlations, and internal consistency
• 8.
  Cull items basedon statistical and judgmental criteria (researcher; subject matter experts) • Use the basic statistics to delete (or hold for editing) those items that performed poorly • If there is sufficient data, some preliminary work with exploratory factor analysis can be used to assess factor purity and make decisions about whether a unitary or faceted scale is more desirable • Items with borderline statistical properties should be considered for editing by SMEs before completely discarding: use a combination of statistical and judgmental criteria to decide
• 9.
  Secondary pilot testwith initial evidence of nomothetic network (researcher; subject matter experts) • The second pilot test will generally be on a diminished set of items, but not necessarily the final set; there may be rewritten items that have not been fielded before • Field the items together with a few other related measures, some where a strong correlation is expected and some where no correlation is expected • Here the demands of statistical power are stronger because you are looking both for significant correlations with other measures and some nil correlations as well; demonstrating a null result requires more statistical power; consult Cohen’s “A Power Primer” for guidance: use regression models
• 10.
  Preliminary analysis ofvalidation evidence (researcher; subject matter experts) • This is the final adjustment step prior to an actual validity run; items can be discarded at this stage, but any rewriting should be very minimal • Depending upon the amount of data you have collected and the maturity of earlier processes, it is possible to perform confirmatory factor analysis on these data • The output of this stage should be a scale that is considered final and basically ready for publication (after the collection of another batch of validity evidence)
• 11.
  Validation with experimentalevidence or multi- trait, multi-method matrix (researcher; subject matter experts) • This is the “official” validation, whose statistical results will be reported for publication: as much care and attention to this study as any substantive study of a research topic • Experimental validation procedures have several merits; a manipulated independent variable is not subject to the common method variance critique; the choice of a manipulation must be based in theory, hopefully the same theory that was initially used to define the construct; experimental methods (when successful) help allay concerns of spurious correlations with other measures • Short of experimental evidence, another powerful strategy is the multi-trait, multi-method matrix; it is quite challenging to find measures captured by alternate methods; MTMM, when successful, is good for showing how the new measure is uniquely positioned to avoid capturing variance of unrelated measures while being related but distinctive from similar constructs
• 12.
  Publication of psychometricand validity evidence (researcher) • Not many new scale developments get this far, and there is generally a dearth of journals that will publish validation studies • Nonetheless, this is the sine qua non of validation: peer review of the techniques used to support the goodness and usability of the new scale