Introduction to Survey Methodology

Peivand Pirouzi, Ph.D.

1
Goals for this Lecture

• Introduce professor and course

• Define what we mean by the term “survey”
– Characteristics of typical surveys
– Distinction between “polls” and “surveys”
• Lay out the basic steps in conducting a
rigorous research survey
• Discuss various survey modes
– Pros and cons
– Trade-offs
• Considerations in making mode
selection
Class is All About Collecting and
Analyzing Survey Data

• In this class you’ll learn:

– How to design good survey questions and craft an
effective survey instrument
– About the trade-offs between various survey
modes
– The rudiments of sampling for surveys
– Some statistical methods useful for analyzing
survey data
What is a Survey?

• A survey is a:
– systematic method for gathering information
– from (a sample of) entities
– for the purposes of constructing quantitative
descriptors
– of the attributes of the larger population of which
the entities are members
• In many ways, a survey is just a form of data
collection…
A Special Type of Data Collection

• Typically surveys:
– Gather information by asking people questions
– Collect information by either (1) interviewers
asking questions and recording responses or (2)
respondents reading and recording their own
answers
– Collect information from a subset of the
population, a sample, rather than from all
members
Polls vs. Surveys

• No clear distinction between the two terms

– “Poll” often used for private sector opinion studies
• Use many of the same design features as
studies that would be called surveys
– “Poll” rarely used to describe government or
scientific surveys
• To me, the term poll implies either
– a commercial or less-scientific study, or
– a quick turn-around survey whose results may be
of short-term interest
Steps in Conducting a Survey (1)

• Clearly state research objective(s)

• Decide on survey mode(s)
– How will you contact potential respondents?
• “Contact mode” – web, e-mail, phone, etc.
– In what media will the survey be given?
• “Survey or response mode” – web, e-mail, phone,
etc.
– How will you follow up with non-respondents?
• “Follow-up mode” – web, e-mail, phone, etc.
• Determine fielding strategy
– How to maximize response rates?
• Design the survey questions and the survey
instrument
Steps in Conducting a Survey (2)

• Determine sample size and sampling strategy

• Obtain Institutional Review Board (IRB) or other
approval as necessary
– Are respondents promised confidentiality?
– What is the impact if their survey responses become
known?
• Pre-test, pre-test, pre-test
– Give the survey to some test subjects and get their
feedback
• What works and what doesn’t?
• Are you getting correct data/information?
– Revise and re-pre-test as necessary
• Draw sample and field the survey
Steps in Conducting a Survey (3)

• Follow up with non-respondents

• Assemble the survey data
– Clean data as necessary
• Analyze the data and report results
– Summarize the data
• Weight as necessary and appropriate to infer back to
population
• Calculate and report margin of error
– Evidence of bias?
• Unit and item non-response
• Communicate survey results to respondents if
promised
One View of the Steps
Another View of the Steps
Examples of Large-scale Surveys

• Statistics Canada:
https://www150.statcan.gc.ca/n1/en/type/
data?MM=1

• Statista
https://www.statista.com

• Canadian Survey of Consumer

Expectations:
https://www.bankofcanada.ca/publications/cana
dian-survey-of-consumer-expectations/
Other examples of surveys
National Crime Victimization Survey
(NCVS)
National Survey on Drug Use and
Health (NSDUH)
Survey of Consumers (SOC)
National Assessment of Educational
Progress (NAEP)
Behavioral Risk Factor Surveillance
System (BRFSS)
Current Employment Statistics (CES)
Program
Which Survey Mode Should I Use?

• You are going to conduct a

research survey
– Goal is to make the best
inference possible from
sample to population
within budget constraints
• To important questions:
– What is the most
appropriate method to
choose for a particular
research question?
– What is the impact of
choosing a particular
method on survey cost
and accuracy?
Traditional Survey Modes

• Mail
– Paper questionnaire sent to respondents
– Self-administered and mailed back
• Telephone
– Interviewers call respondents on telephone
– Interviewer-assisted
• In-person
– Interviewers go to respondent’s home or office
– Interviewer-assisted
Broadly speaking…

• Interviewer-assisted surveys more costly than self-

administered
– Face-to-face most expensive, then telephone, then mail,
then web
• Interviewer-assisted surveys less prone to various
errors and item nonresponse
– Particularly compared to self-administered paper survey
– More or less compared to self-administered computer-
assisted surveys depending on sophistication of program
• Self-administered less subject to sensitivity bias
– Can combine some self-administered questions into a
predominantly interviewer-assisted mode
However, timeliness is not just about…

• …the length of time a survey is in the field

– Really, it’s from instrument design through analysis
• Much of that is not affected by survey mode
– Compared to a mail survey, web-based surveys
save on two main things:
• Mailing time to send out and return surveys
• Time required to do data entry, coding, and
cleaning for paper surveys
• Yet, it can also take a lot of time to code, test,
and de-bug very complicated web (more
generally, computer-assisted) survey programs
And, cost is not just about…

• …manhours
– On large mail surveys, printing and postage costs can
be significant
• 1,500 surveys x 4 mailings at $1.50 each = $9,000!
• 1,500 $1 incentives = $1,500
– On telephone surveys must consider fixed costs of phone
and CATI equipment + variable costs
– On face-to-face interviews, travel costs can be significant (not
to mention interviewer time)
• But in terms of manhours, don’t forget: (1) instrument
design time; (2) in non-electronic modes, data entry
and coding time; (3) non-response follow-up efforts;
and (4) analysis time
And accuracy is not just about…

• … response rates and

missing items
– Total survey error:
what mode or modes
will most decrease all
types of error?
• Lots of
considerations
– Frame coverage
– Nonresponse rates
– Sensitivity bias, etc…
Cost and Frame Availability
Often Drive Mode Choice

• Survey budgets never unlimited

– Frequently cost will eliminate one or more modes
• And likely drive other survey design decisions
– I.e., face-to-face interviews too expensive
• On the other hand, surveys using area
probability frames usually have to start out
using face-to-face interviews
• Similarly, if a complete e-mail frame is
available, web survey is possible option
– But general population surveys eliminate
web…
Some Modes Logically Group (1)

• Telephone and face-to-face surveys

– Telephone often considered an alternative to face-
to-face
– Both interviewer-assisted and can have similar
frame coverage issues
– Many longitudinal surveys start out with a face-to-
face interview with telephone follow-ups
• Mail sometimes an alternative to telephone
– If both telephone and addresses available
Some Modes Logically Group (2)

• Web surveys often considered a replacement

for mail surveys
– Coverage is still an issue for web surveys
– Web superior in terms of automation and speed
– However, literature consistently finds that
nonresponse rates are higher for web surveys
Using Multiple Modes

• Reasons to use multiple modes:

– Maximize response rates
• Reduce/eliminate selection & coverage bias
• Allow respondents to use most convenient mode
– Reduce costs
• Cross-sectional surveys: start with least costly
mode and move to more expensive modes as
necessary to get response
• Longitudinal surveys: Use less expensive modes
in follow-on survey rounds
Considerations When Using
Multiple Modes

• Focus should be on designing instruments to ensure

equivalency across modes
– Rather than optimizing for a particular mode
– E.g., make the follow-on web survey resemble its
counterpart paper survey
• Ensure operation is set up to track cases across the
modes
– Must avoid duplications and uncoordinated respondent
contact
• Must design and field so that mode effects can be
disentangled from other sample characteristics
– May have to randomize survey mode over a subset
Other Data Collection Methods

• Surveys are not the only (nor necessarily the

best) way to collect data
• Other methods include
– Administrative records
– Focus groups and qualitative investigations
– Randomized experiments
• Which is “best” depends on the research
question(s) and/or the purpose for which the
data will be used
What We Have Covered

• Defined the term “survey,” including the

– Characteristics of types of surveys this class will focus on
– Distinction between “polls” and “surveys”
• Discussed the basic steps in conducting a rigorous
research survey
• Considerations when choosing a survey mode
– In various ways, choice likely to affect survey
• Timeliness
• Accuracy
• Cost
– And cost may drive mode choice possibilities