Quantitative Methods

Defining Surveys and Experiments

A survey design provides a quantitative description
of trends, attitudes, and opinions of a population,
or tests for associations among variables of a
population, by studying a sample of that
population. Survey designs help researchers
answer three types of questions: (a) descriptive
questions (e.g., What percentage of practicing
nurses support the provision of hospital abortion
services?);
(b) questions about the relationships
between variables (e.g., Is there a
positive association between
endorsement of hospital abortion
services and support for implementing
hospice care among nurses?); or in
cases where a survey design is
repeated over time in a longitudinal
study;
(c) questions about predictive
relationships between
variables over time (e.g., Does
Time 1 endorsement of support
for hospital abortion services
predict greater Time 2 burnout
in nurses?).
An experimental design systematically
manipulates one or more variables in order to
evaluate how this manipulation impacts an
outcome (or outcomes) of interest. Importantly, an
experiment isolates the effects of this
manipulation by holding all other variables
constant. When one group receives a treatment
and the other group does not (which is a
manipulated variable of interest), the experimenter
can isolate whether the treatment and not other
factors influence the outcome.
For example, a sample of nurses could be
randomly assigned to a 3-week expressive
writing program (where they write about their
deepest thoughts and feelings) or a matched
3-week control writing program (writing about
the facts of their daily morning routine) to
evaluate whether this expressive writing
manipulation reduces job burnout in the
months following the program (i.e., the writing
condition is the manipulated variable of
interest, and job burnout is the outcome of
interest).
Whether a quantitative study employs a
survey or experimental design, both
approaches share a common goal of
helping the researcher make inferences
about relationships among variables, and
how the sample results may generalize to a
broader population of interest (e.g., all
nurses in the community).
The Survey Design
Identify the purpose of survey research. The
primary purpose is to answer a question (or
questions) about variables of interest to you. A
sample purpose statement could read: “The
primary purpose of this study is to empirically
evaluate whether the number of overtime hours
worked predicts subsequent burnout symptoms in
a sample of emergency room nurses.”
Indicate why a survey method is the preferred type
of approach for this study. In this rationale, it can
be beneficial to acknowledge the advantages of
survey designs, such as the economy of the design,
rapid turnaround in data collection, and
constraints that preclude you from pursuing other
designs (e.g., “An experimental design was not
adopted to look at the relationship between
overtime hours worked and burnout symptoms
because it would be prohibitively difficult, and
potentially unethical, to randomly assign nurses to
work different amounts of overtime hours.”).
Indicate whether the survey will be
cross-sectional—with the data
collected at one point in time—or
whether it will be longitudinal—
with data collected over time.
Specify the form of data collection. Fowler (2014)
identified the following types: mail, telephone, the
Internet, personal interviews, or group administration
(see also Fink, 2016; Krueger & Casey, 2014). Using an
Internet survey and administering it online has been
discussed extensively in the literature (Nesbary, 2000;
Sue & Ritter, 2012). Regardless of the form of data
collection, provide a rationale for the procedure, using
arguments based on its strengths and weaknesses,
costs, data availability, and convenience.
The Population and Sample
The population. Identify the population in the
study. Also state the size of this population, if
size can be determined, and the means of
identifying individuals in the population.
Questions of access arise here, and the
researcher might refer to availability of
sampling frames—mail or published lists—of
potential respondents in the population.
Sampling design
Identify whether the sampling design for this population is single
stage or multistage (called clustering). Cluster sampling is ideal
when it is impossible or impractical to compile a list of the
elements composing the population (Babbie, 2015). A single-stage
sampling procedure is one in which the researcher has access to
names in the population and can sample the people (or other
elements) directly. In a multistage or clustering procedure, the
researcher first identifies clusters (groups or organizations), obtains
names of individuals within those clusters, and then samples within
them.
Type of sampling
Identify and discuss the selection process for
participants in your sample. Ideally you aim to
draw a random sample, in which each
individual in the population has an equal
probability of being selected (a systematic or
probabilistic sample). But in many cases it may
be quite difficult (or impossible) to get a
random sample of participants.
Alternatively, a systematic sample can have
precision equivalent random sampling (Fowler,
2014). In this approach, you choose a random start
on a list and select every X numbered person on the
list. The X number is based on a fraction determined
by the number of people on the list and the number
that are to be selected on the list (e.g., 1 out of every
80th person). Finally, less desirable, but often used,
is a nonprobability sample (or convenience sample),
in which respondents are chosen based on their
convenience and availability.
Stratification
Identify whether the study will involve stratification
of the population before selecting the sample. This
requires that characteristics of the population
members be known so that the population can be
stratified first before selecting the sample (Fowler,
2014). Stratification means that specific
characteristics of individuals (e.g., gender—
females and males) are represented in the sample
and the sample reflects the true proportion in the
population of individuals with certain
characteristics.
When randomly selecting people from a
population, these characteristics may or
may not be present in the sample in the
same proportions as in the population;
stratification ensures their representation.
Also identify the characteristics used in
stratifying the population (e.g., gender,
income levels, education).
Within each stratum, identify
whether the sample contains
individuals with the characteristic
in the same proportion as the
characteristic appears in the entire
population.
Sample size determination

Indicate the number of people in the

sample and the procedures used to
compute this number. Sample size
determination is at its core a tradeoff: A
larger sample will provide more accuracy
in the inferences made, but recruiting more
participants is time consuming and costly.
In survey research, investigators
sometimes choose a sample size
based on selecting a fraction of the
population (say, 10%) or selecting a
sample size that is typical based on
past studies. These approaches are
not optimal; instead sample size
determination should be based on
your analysis plans (Fowler, 2014).
Instrumentation
Name the survey instruments used to collect data. Discuss
whether you used an instrument designed for this research, a
modified instrument, or an instrument developed by someone
else. For example, if you aim to measure perceptions of stress
over the last month, you could use the 10-item Perceived
Stress Scale (PSS) (Cohen, Kamarck, & Mermelstein, 1983) as
your stress perceptions instrument in your survey design. Many
survey instruments, including the PSS, can be acquired and
used for free as long as you cite the original source of the
instrument.
But in some cases, researchers have made the
use of their instruments proprietary, requiring a
fee for use. Instruments are increasingly being
delivered through a multitude of online survey
products now available (e.g., Qualtrics, Survey
Monkey). Although these products can be
costly, they also can be quite helpful for
accelerating and improving the survey research
process.
For example, researchers can create
their own surveys quickly using custom
templates and post them on websites
or e-mail them to participants to
complete. These software programs
facilitate data collection into organized
spreadsheets for data analysis,
reducing data entry errors and
accelerating hypothesis testing.
Validity of scores using the instrument. To
use an existing instrument, describe the
established validity of scores obtained
from past use of the instrument. This
means reporting efforts by authors to
establish validity in quantitative research—
whether you can draw meaningful and
useful inferences from scores on the
instruments.
The three traditional forms of validity to
look for are (a) content validity (Do the
items measure the content they were
intended to measure?), (b) predictive or
concurrent validity (Do scores predict a
criterion measure? Do results correlate
with other results?), and (c) construct
validity (Do items measure hypothetical
constructs or concepts?).
In more recent studies, construct validity has
become the overriding objective in validity, and it
has focused on whether the scores serve a useful
purpose and have positive consequences when
they are used in practice (Humbley & Zumbo,
1996). Establishing the validity of the scores in a
survey helps researchers to identify whether an
instrument might be a good one to use in survey
research. This form of validity is different from
identifying the threats to validity in experimental
research, as discussed later.
Reliability of scores on the instrument
Also mention whether scores resulting from past
use of the instrument demonstrate acceptable
reliability. Reliability in this context refers to the
consistency or repeatability of an instrument. The
most important form of reliability for multi-item
instruments is the instrument’s internal
consistency—which is the degree to which sets of
items on an instrument behave in the same way.
This is important because your instrument
scale items should be assessing the same
underlying construct, so these items should
have suitable intercorrelations. A scale’s
internal consistency is quantified by a
Cronbach’s alpha (α) value that ranges
between 0 and 1, with optimal values ranging
between .7 and .9.
For example, the 10-item PSS has
excellent internal consistency across
many published reports, with the
original source publication reporting
internal consistency values of α =.84–
.86 in three studies (Cohen, Kamarck,
and Mermelstein, 1983).
It can also be helpful to evaluate a second
form of instrument reliability, its test-retest
reliability. This form of reliability concerns
whether the scale is reasonably stable over
time with repeated administrations. When you
modify an instrument or combine instruments
in a study, the original validity and reliability
may not hold for the new instrument, and it
becomes important to establish validity and
reliability during data analysis.
Sample items

Include sample items from the instrument

so that readers can see the actual items
used. In an appendix to the proposal,
attach sample items or the entire
instrument (or instruments) used.
Content of instrument
Indicate the major content sections in the instrument,
such as the cover letter (Dillman, 2007, provides a useful
list of items to include in cover letters), the items (e.g.,
demographics, attitudinal items, behavioral items,
factual items), and the closing instructions. Also mention
the type of scales used to measure the items on the
instrument, such as continuous scales (e.g., strongly
agree to strongly disagree) and categorical scales (e.g.,
yes/no, rank from highest to lowest importance).
Pilot testing
Discuss plans for pilot testing or field-testing the survey and
provide a rationale for these plans. This testing is important to
establish the content validity of scores on an instrument; to
provide an initial evaluation of the internal consistency of the
items; and to improve questions, format, and instructions. Pilot
testing all study materials also provides an opportunity to
assess how long the study will take (and to identify potential
concerns with participant fatigue). Indicate the number of
people who will test the instrument and the plans to
incorporate their comments into final instrument revisions.
Administering the survey
For a mailed survey, identify steps for
administering the survey and for following up to
ensure a high response rate. Salant and Dillman
(1994) suggested a four-phase administration
process (see Dillman, 2007, for a similar three-
phase process). The first mail-out is a short
advance-notice letter to all members of the
sample, and the second mail-out is the actual mail
survey, distributed about 1 week after the advance-
notice letter.
The third mail-out consists of a postcard follow-up
sent to all members of the sample 4 to 8 days after
the initial questionnaire. The fourth mail-out, sent
to all nonrespondents, consists of a personalized
cover letter with a handwritten signature, the
questionnaire, and a preaddressed return envelope
with postage. Researchers send this fourth mail-
out 3 weeks after the second mail-out. Thus, in
total, the researcher concludes the administration
period 4 weeks after its start, providing the returns
meet project objectives.
Components of an Experimental Study
Method Plan
An experimental method plan follows a
standard form: (a) participants and design, (b)
procedure, and (c) measures. These three
sequential sections generally are sufficient
(often in studies with a few measures, the
procedure and measures sections are
combined into a single procedure section).
Participants
Readers need to know about the selection,
assignment, and number of participants who will
take part in the experiment. Consider the following
suggestions when writing the method section plan
for an experiment:
Describe the procedures for recruiting participants
to be in the study, and any selection processes
used.
Often investigators aim to recruit a
study sample that shares certain
characteristics by formally stating
specific inclusion and exclusion study
criteria when designing their study
(e.g., inclusion criterion: participants
must be English language speaking;
exclusion criterion: participants must
not be children under the age of 18).
Recruitment approaches are wide-
ranging, and can include random digit
dialing of households in a community,
posting study recruitment flyers or e-
mails to targeted communities, or
newspaper advertisements. Describe
the recruitment approaches that will be
used and the study compensation
provided for participating.
One of the principal features distinguishing
an experiment from a survey study design
is the use of random assignment. Random
assignment is a technique for placing
participants into study conditions of a
manipulated variable of interest. When
individuals are randomly assigned to
groups, the procedure is called a true
experiment.
If random assignment is used, discuss how
and when the study will randomly assign
individuals to treatment groups, which in
experimental studies are referred to as
levels of an independent variable. This
means that of the pool of participants,
Individual 1 goes to Group 1, Individual 2 to
Group 2, and so forth so that there is no
systematic bias in assigning the
individuals.
This procedure eliminates the
possibility of systematic differences
among characteristics of the
participants that could affect the
outcomes so that any differences in
outcomes can be attributed to the
study’s manipulated variable (or
variables) of interest (Keppel &
Wickens, 2003).
Often experimental studies may be interested in
both randomly assigning participants to levels of a
manipulated variable of interest (e.g., a new
treatment approach for teaching fractions to
children versus the traditional approach) while also
measuring a second predictor variable of interest
that cannot utilize random assignment (e.g.,
measuring whether the treatment benefits are
larger among female compared to male children; it
is impossible to randomly assign children to be
male or female).
Designs in which a researcher has only
partial (or no) control over randomly
assigning participants to levels of a
manipulated variable of interest are
called quasi-experiments.
At the end of the participants section, it is
helpful to provide a formal experimental
design statement that specifies the
independent variables and their
corresponding levels. For example, a formal
design statement might read, “The experiment
consisted of a one-way two-groups design
comparing burnout symptoms between full-
time and part-time nurses.”
Experimental Procedures
Identify the type of experimental design to be used
in the proposed study. The types available in
experiments are pre-experimental designs, quasi-
experiments, and true experiments. With pre-
experimental designs, the researcher studies a
single group and implements an intervention
during the experiment. This design does not have a
control group to compare with the experimental
group.
In quasi-experiments, the investigator uses
control and experimental groups, but the
design may have partial or total lack of
random assignment to groups. In a true
experiment, the investigator randomly
assigns the participants to treatment
groups. A single-subject design or N of 1
design involves observing the behavior of a
single individual (or a small number of
individuals) over time.
Identify what is being compared in the experiment.
In many experiments, those of a type called
between-subject designs, the investigator
compares two or more groups (Keppel & Wickens,
2003; Rosenthal & Rosnow, 1991). For example, a
factorial design experiment, a variation on the
between group design, involves using two or more
treatment variables to examine the independent
and simultaneous effects of these treatment
variables on an outcome (Vogt & Johnson, 2015).
This widely used experimental design explores
the effects of each treatment separately and
also the effects of variables used in
combination, thereby providing a rich and
revealing multidimensional view. In other
experiments, the researcher studies only one
group in what is called a within-group design.
For example, in a repeated measures design,
participants are assigned to different
treatments at different times during the
experiment.
Another example of a within-group design
would be a study of the behavior of a single
individual over time in which the
experimenter provides and withholds a
treatment at different times in the
experiment to determine its impact.
Finally, studies that include both a
between-subjects and a within-subjects
variable are called mixed designs.
Provide a diagram or a figure to illustrate the specific
research design to be used. A standard notation system
needs to be used in this figure. As a research tip, we
recommend using the classic notation system provided by
Campbell and Stanley (1963, p. 6):
• X represents an exposure of a group to an experimental
variable or event, the effects of which are to be measured.
• O represents an observation or measurement recorded
on an instrument.
• Xs and Os in a given row are applied to the same specific
persons. Xs and Os in the same column, or placed
vertically relative to each other, are simultaneous.
• The left-to-right dimension indicates the temporal
order of procedures in the experiment (sometimes
indicated with an arrow).
• The symbol R indicates random assignment.
• Separation of parallel rows by a horizontal line
indicates that comparison groups are not equal
(or equated) by random assignment. No
horizontal line between the groups displays
random assignment of individuals to treatment
groups.
One-Shot Case Study
This design involves an exposure of a group to a
treatment followed by a measure.
Group A X_____________________O
One-Group Pretest-Posttest Design
This design includes a pretest measure
followed by a treatment and a posttest for
a single group.
Group A O1————X————O2
Static Group Comparison or Posttest-Only With
Nonequivalent Groups
Experimenters use this design after
implementing a treatment. After the treatment,
the researcher selects a comparison group and
provides a posttest to both the experimental
group(s) and the comparison group(s).
Group A X______________________O
Group B _______________________O
Alternative Treatment Posttest-Only With
Nonequivalent Groups Design
This design uses the same procedure as
the Static Group Comparison, with the
exception that the nonequivalent
comparison group received a different
treatment.
Group A X1_____________________O
Group B X2_____________________O
Nonequivalent (Pretest and Posttest) Control-
Group Design
In this design, a popular approach to quasi-
experiments, the experimental Group A and the
control Group B are selected without random
assignment. Both groups take a pretest and
posttest. Only the experimental group receives the
treatment.
Group A O————X————O
___________________________
Group B O—————————O
Single-Group Interrupted Time-Series Design
In this design, the researcher records
measures for a single group both before and
after a treatment.
Group A O—O—O—O—X—O—O—O—O
Control-Group Interrupted Time-Series Design
This design is a modification of the Single-Group
Interrupted Time-Series design in which two groups
of participants, not randomly assigned, are observed
over time. A treatment is administered to only one of
the groups (i.e., Group A).
Group A O—O—O—O—X—O—O—O—O
__________________________________
Group B O—O—O—O—O—O—O—O—O
Pretest–Posttest Control-Group Design
A traditional, classical design, this procedure
involves random assignment of participants to
two groups. Both groups are administered both
a pretest and a posttest, but the treatment is
provided only to experimental Group A.
Group A R——–––O———X———O
Group B R———O———————O
Posttest-Only Control-Group Design
This design controls for any confounding
effects of a pretest and is a popular
experimental design. The participants are
randomly assigned to groups, a treatment is
given only to the experimental group, and both
groups are measured on the posttest.
Group A R——————X—————O
Group B R————————————O
Solomon Four-Group Design
A special case of a 2 × 2 factorial design, this
procedure involves the random assignment of
participants to four groups. Pretests and
treatments are varied for the four groups. All
groups receive a posttest.
Group A R————O———X———O
Group B R————O———————O
Group C R———————X————O
Group D R——————————–—O
A-B-A Single-Subject Design
This design involves multiple observations of a
single individual. The target behavior of a single
individual is established over time and is
referred to as a baseline behavior. The baseline
behavior is assessed, the treatment provided,
and then the treatment is withdrawn.
Baseline A Treatment B Baseline A
O–O–O–O–O–X–X–X–X–X–O–O–O–O–O–O
Threats to Validity
There are several threats to validity that will raise
questions about an experimenter’s ability to conclude
that the manipulated variable(s) of interest affect an
outcome and not some other factor. Experimental
researchers need to identify potential threats to the
internal validity of their experiments and design them so
that these threats will not likely arise or are minimized.
There are two types of threats to validity: (a) internal
threats and (b) external threats.
Internal validity threats are experimental procedures,
treatments, or experiences of the participants that threaten
the researcher’s ability to draw correct inferences from the
data about the population in an experiment. There are
those involving participants (i.e., history, maturation,
regression, selection, and mortality), those related to the
use of an experimental treatment that the researcher
manipulates (i.e., diffusion, compensatory and resentful
demoralization, and compensatory rivalry), and those
involving procedures used in the experiment (i.e., testing
and instruments).
External validity threats arise when experimenters draw
incorrect inferences from the sample data to other
persons, other settings, and past or future situations.
These threats arise because of the characteristics of
individuals selected for the sample, the uniqueness of
the setting, and the timing of the experiment. For
example, threats to external validity arise when the
researcher generalizes beyond the groups in the
experiment to other racial or social groups not under
study, to settings not examined, or to past or future
situations.
References
Creswell, J. W. & Creswell, J. D. (2022). Research design:
Qualitative, quantitative, and mixed methods approaches (6th ed.).
Thousand Oaks, CA: Sage.
Fraenkel, J. R., Wallen, N. E., & Hyun, H. E. (2023). How to design
and evaluate research in education (11th ed.). New York: McGraw-
Hill.