Factorial Designs

A Simple Example
Probably the easiest way to begin understanding factorial designs is by looking at an example.
Let's imagine a design where we have an educational program where we would like to look at a
variety of program variations to see which works best. For instance, we would like to vary the
amount of time the children receive instruction with one group getting 1 hour of instruction per
week and another getting 4 hours per week. And, we'd like to vary the setting with one group
getting the instruction in-class (probably pulled off into a corner of the classroom) and the other
group being pulled-out of the classroom for instruction in another room. We could think about
having four separate groups to do this, but when we are varying the amount of time in
instruction, what setting would we use: in-class or pull-out? And, when we were studying setting,
what amount of instruction time would we use: 1 hour, 4 hours, or something else?
With factorial designs, we don't have to compromise when answering these questions. We can
have it both ways if we cross each of our two time in instruction conditions with each of our two
settings. Let's begin by doing some defining of terms. In factorial designs, a factor is a major
independent variable. In this example we have two factors: time in instruction and setting. A
level is a subdivision of a factor. In this example, time in instruction has two levels and setting
has two levels. Sometimes we depict a factorial design with a numbering notation. In this
example, we can say that we have a 2 x 2 (spoken "two-by-two) factorial design. In this notation,
the number of numbers tells you how many factors there are and the number values tell you how
many levels. If I said I had a 3 x 4 factorial design, you would know that I had 2 factors and that
one factor had 3 levels while the other had 4. Order of the numbers makes no difference and we
could just as easily term this a 4 x 3 factorial design. The number of different treatment groups
that we have in any factorial design can easily be determined by multiplying through the number
notation. For instance, in our example we have 2 x 2 = 4 groups. In our notational example, we
would need 3 x 4 = 12 groups.
We can also depict a factorial design in design notation. Because of the treatment level
combinations, it is useful to use subscripts on the treatment (X) symbol. We can see in the figure
that there are four groups, one for each combination of levels of factors. It is also immediately
apparent that the groups were randomly assigned and that this is a posttest-only design.
Now, let's look at a variety of different results we might get from this simple 2 x 2 factorial
design. Each of the following figures describes a different possible outcome. And each outcome
is shown in table form (the 2 x 2 table with the row and column averages) and in graphic form
(with each factor taking a turn on the horizontal axis). You should convince yourself that the
information in the tables agrees with the information in both of the graphs. You should also
convince yourself that the pair of graphs in each figure show the exact same information graphed
in two different ways. The lines that are shown in the graphs are technically not necessary -- they
are used as a visual aid to enable you to easily track where the averages for a single level go
across levels of another factor. Keep in mind that the values shown in the tables and graphs are

group averages on the outcome variable of interest. In this example, the outcome might be a test
of achievement in the subject being taught. We will assume that scores on this test range from 1
to 10 with higher values indicating greater achievement. You should study carefully the
outcomes in each figure in order to understand the differences between these cases.
Factorial design is an important method to determine the effects of multiple variables on a
response. Traditionally, experiments are designed to determine the effect of ONE variable upon
ONE response. R.A. Fisher showed that there are advantages by combining the study of multiple
variables in the same factorial experiment. Factorial design can reduce the number of
experiments one has to perform by studying multiple factors simultaneously. Additionally, it can
be used to find both main effects (from each independent factor) and interaction effects (when
both factors must be used to explain the outcome). However, factorial design can only give
relative values, and to achieve actual numerical values the math becomes difficult, as regressions
(which require minimizing a sum of values) need to be performed. Regardless, factorial design is
a useful method to design experiments in both laboratory and industrial settings.
Factorial design tests all possible conditions. Because factorial design can lead to a large number
of trials, which can become expensive and time-consuming, factorial design is best used for a
small number of variables with few states (1 to 3). Factorial design works well when interactions
between variables are strong and important and where every variable contributes significantly.

Scientific research involves a systematic process that focuses on being objective and gathering a multitude
of information for analysis so that the researcher can come to a conclusion. This process is used in all
research and evaluation projects, regardless of the research method (scientific method of inquiry, evaluation
research, or action research). The process focuses on testing hunches or ideas in a park and recreation
setting through a systematic process. In this process, the study is documented in such a way that another
individual can conduct the same study again. This is referred to as replicating the study. Any research done
without documenting the study so that others can review the process and results is not an investigation
using the scientific research process. The scientific research process is a multiple-step process where the
steps are interlinked with the other steps in the process. If changes are made in one step of the process, the
researcher must review all the other steps to ensure that the changes are reflected throughout the process.
Parks and recreation professionals are often involved in conducting research or evaluation projects within
the agency. These professionals need to understand the eight steps of the research process as they apply to
conducting a study. Table 2.4 lists the steps of the research process and provides an example of each step
for a sample research study.
Step 1: Identify the Problem
The first step in the process is to identify a problem or develop a research question. The research problem
may be something the agency identifies as a problem, some knowledge or information that is needed by the
agency, or the desire to identify a recreation trend nationally. In the example in table 2.4, the problem that
the agency has identified is childhood obesity, which is a local problem and concern within the community.

This serves as the focus of the study.

Step 2: Review the Literature

Now that the problem has been identified, the researcher must learn more about the topic under
investigation. To do this, the researcher must review the literature related to the research problem. This step
provides foundational knowledge about the problem area. The review of literature also educates the
researcher about what studies have been conducted in the past, how these studies were conducted, and the
conclusions in the problem area. In the obesity study, the review of literature enables the programmer to
discover horrifying statistics related to the long-term effects of childhood obesity in terms of health issues,
death rates, and projected medical costs. In addition, the programmer finds several articles and information
from the Centers for Disease Control and Prevention that describe the benefits of walking 10,000 steps a
day. The information discovered during this step helps the programmer fully understand the magnitude of
the problem, recognize the future consequences of obesity, and identify a strategy to combat obesity (i.e.,
walking).
Step 3: Clarify the Problem
Many times the initial problem identified in the first step of the process is too large or broad in scope. In
step 3 of the process, the researcher clarifies the problem and narrows the scope of the study. This can only
be done after the literature has been reviewed. The knowledge gained through the review of literature
guides the researcher in clarifying and narrowing the research project. In the example, the programmer has
identified childhood obesity as the problem and the purpose of the study. This topic is very broad and could
be studied based on genetics, family environment, diet, exercise, self-confidence, leisure activities, or
health issues. All of these areas cannot be investigated in a single study; therefore, the problem and purpose
of the study must be more clearly defined. The programmer has decided that the purpose of the study is to
determine if walking 10,000 steps a day for three days a week will improve the individuals health. This
purpose is more narrowly focused and researchable than the original problem.
Step 4: Clearly Define Terms and Concepts
Terms and concepts are words or phrases used in the purpose statement of the study or the description of
the study. These items need to be specifically defined as they apply to the study. Terms or concepts often
have different definitions depending on who is reading the study. To minimize confusion about what the
terms and phrases mean, the researcher must specifically define them for the study. In the obesity study, the
concept of individuals health can be defined in hundreds of ways, such as physical, mental, emotional,
or spiritual health. For this study, the individuals health is defined as physical health. The concept of
physical health may also be defined and measured in many ways. In this case, the programmer decides to
more narrowly define individual health to refer to the areas of weight, percentage of body fat, and
cholesterol. By defining the terms or concepts more narrowly, the scope of the study is more manageable
for the programmer, making it easier to collect the necessary data for the study. This also makes the
concepts more understandable to the reader.

Step 5: Define the Population

Research projects can focus on a specific group of people, facilities, park development, employee
evaluations, programs, financial status, marketing efforts, or the integration of technology into the
operations. For example, if a researcher wants to examine a specific group of people in the community, the
study could examine a specific age group, males or females, people living in a specific geographic area, or
a specific ethnic group. Literally thousands of options are available to the researcher to specifically identify
the group to study. The research problem and the purpose of the study assist the researcher in identifying
the group to involve in the study. In research terms, the group to involve in the study is always called the
population. Defining the population assists the researcher in several ways. First, it narrows the scope of the
study from a very large population to one that is manageable. Second, the population identifies the group
that the researchers efforts will be focused on within the study. This helps ensure that the researcher stays
on the right path during the study. Finally, by defining the population, the researcher identifies the group
that the results will apply to at the conclusion of the study. In the example in table 2.4, the programmer has
identified the population of the study as children ages 10 to 12 years. This narrower population makes the
study more manageable in terms of time and resources.
Step 6: Develop the Instrumentation Plan
The plan for the study is referred to as the instrumentation plan. The instrumentation plan serves as the road
map for the entire study, specifying who will participate in the study; how, when, and where data will be
collected; and the content of the program. This plan is composed of numerous decisions and considerations
that are addressed in chapter 8 of this text. In the obesity study, the researcher has decided to have the
children participate in a walking program for six months. The group of participants is called the sample,
which is a smaller group selected from the population specified for the study. The study cannot possibly
include every 10- to 12-year-old child in the community, so a smaller group is used to represent the
population. The researcher develops the plan for the walking program, indicating what data will be
collected, when and how the data will be collected, who will collect the data, and how the data will be
analyzed. The instrumentation plan specifies all the steps that must be completed for the study. This ensures
that the programmer has carefully thought through all these decisions and that she provides a step-by-step
plan to be followed in the study.
Step 7: Collect Data
Once the instrumentation plan is completed, the actual study begins with the collection of data. The
collection of data is a critical step in providing the information needed to answer the research question.
Every study includes the collection of some type of datawhether it is from the literature or from subjects
to answer the research question. Data can be collected in the form of words on a survey, with a
questionnaire, through observations, or from the literature. In the obesity study, the programmers will be
collecting data on the defined variables: weight, percentage of body fat, cholesterol levels, and the number
of days the person walked a total of 10,000 steps during the class.
The researcher collects these data at the first session and at the last session of the program. These two sets
of data are necessary to determine the effect of the walking program on weight, body fat, and cholesterol
level. Once the data are collected on the variables, the researcher is ready to move to the final step of the

process, which is the data analysis.

Step 8: Analyze the Data
All the time, effort, and resources dedicated to steps 1 through 7 of the research process
culminate in this final step. The researcher finally has data to analyze so that the research
question can be answered. In the instrumentation plan, the researcher specified how the data will
be analyzed. The researcher now analyzes the data according to the plan. The results of this
analysis are then reviewed and summarized in a manner directly related to the research
questions. In the obesity study, the researcher compares the measurements of weight, percentage
of body fat, and cholesterol that were taken at the first meeting of the subjects to the
measurements of the same variables at the final program session. These two sets of data will be
analyzed to determine if there was a difference between the first measurement and the second
measurement for each individual in the program. Then, the data will be analyzed to determine if
the differences are statistically significant. If the differences are statistically significant, the study
validates the theory that was the focus of the study. The results of the study also provide valuable
information about one strategy to combat childhood obesity in the community.
As you have probably concluded, conducting studies using the eight steps of the scientific
research process requires you to dedicate time and effort to the planning process. You cannot
conduct a study using the scientific research process when time is limited or the study is done at
the last minute. Researchers who do this conduct studies that result in either false conclusions or
conclusions that are not of any value to the organization.

Discourse analysis (DA), or discourse studies, is a general term for a number of approaches to
analyze written, vocal, or sign language use, or any significant semiotic event.
The objects of discourse analysisdiscourse, writing, conversation, communicative eventare
variously defined in terms of coherent sequences of sentences, propositions, speech, or turns-attalk. Contrary to much of traditional linguistics, discourse analysts not only study language use
'beyond the sentence boundary', but also prefer to analyze 'naturally occurring' language use, and
not invented examples. Text linguistics is a closely related field. The essential difference between
discourse analysis and text linguistics is that it aims at revealing socio-psychological
characteristics of a person/persons rather than text structure.[1]

Discourse Analysis

Discourse analysis is a qualitative research method that investigates the use of language in social
contexts. Concerned with the creation of meaning through talk and texts, discourse analysis
provides insights into the way language works to help shape and reproduce social meanings and
forms of knowledge (Tonkiss, 2012, p. 403). Grounded in social constructivism, which
emphasizes the sociocultural interactions as sources of knowledge, discourse analysis is based on
the following three theoretical assumptions (based on Potter,1996) :

First, knowledge cannot be gained by pure objectivity as scientific and positivist

researchers believe it can. A researcher brings his or her own set of beliefs, cultural
values, expectations, subjectivity and bias into the study when conducting his or her
research: A researcher recognizes his or her own beliefs, and acknowledges how these
beliefs influenced by his or her own personal, cultural, and historical experiences shape
his or her interpretations of reality and knowledge.

Second, reality is socially and culturally constructed. Unlike scientific approaches in

which reality, ideas, or constructs (e.g. intelligence & attitudes) are categorized as
naturally occurring things, in social constructivist or interpretive approaches, these
categories and constructs are shaped by the language and since language is a
sociocultural phenomenon, our sense of reality is socially and culturally
constructed.These realities which are often varied and multiple lead researchers to look
for the complexity of the views rather than reduce meanings into a few categories or
ideas. The goal of research, then, is to give insights into the different views and
perspectives of participants and how these views and perspectives are socially and
historically negotiated.

Third, in social constructivism, a researcher is more interested in studying the language

(discourse) and the role it plays in construction of meaning and knowledge in society. As
such, the emphasis of such research is placed on the discursive patterns of talk in
societies, their impact on the formation and reproduction of social meanings and
identities as well as their role in empowering and disenfranchising institutions and
individuals.

Quasi-Experimental Design

A quasi-experimental design is one that looks a bit like an experimental design but lacks the key
ingredient -- random assignment. My mentor, Don Campbell, often referred to them as "queasy"
experiments because they give the experimental purists a queasy feeling. With respect to internal
validity, they often appear to be inferior to randomized experiments. But there is something
compelling about these designs; taken as a group, they are easily more frequently implemented
than their randomized cousins.
I'm not going to try to cover the quasi-experimental designs comprehensively. Instead, I'll present
two of the classic quasi-experimental designs in some detail and show how we analyze them.
Probably the most commonly used quasi-experimental design (and it may be the most commonly

used of all designs) is the nonequivalent groups design. In its simplest form it requires a pretest
and posttest for a treated and comparison group. It's identical to the Analysis of Covariance
design except that the groups are not created through random assignment. You will see that the
lack of random assignment, and the potential nonequivalence between the groups, complicates
the statistical analysis of the nonequivalent groups design.
The second design I'll focus on is the regression-discontinuity design. I'm not including it just
because I did my dissertation on it and wrote a book about it (although those were certainly
factors weighing in its favor!). I include it because I believe it is an important and often
misunderstood alternative to randomized experiments because its distinguishing characteristic -assignment to treatment using a cutoff score on a pretreatment variable -- allows us to assign to
the program those who need or deserve it most. At first glance, the regression discontinuity
design strikes most people as biased because of regression to the mean. After all, we're assigning
low scorers to one group and high scorers to the other. In the discussion of the statistical analysis
of the regression discontinuity design, I'll show you why this isn't the case.
Finally, I'll briefly present an assortment of other quasi-experiments that have specific
applicability or noteworthy features, including the Proxy Pretest Design, Double Pretest Design,
Nonequivalent Dependent Variables Design, Pattern Matching Design, and the Regression Point
Displacement design. I had the distinct honor of co-authoring a paper with Donald T. Campbell
that first described the Regression Point Displacement Design. At the time of his death in Spring
1996, we had gone through about five drafts each over a five year period. The paper (click here
for the entire paper) includes numerous examples of this newest of quasi-experiments, and
provides a detailed description of the statistical analysis of the regression point displacement
design.
There is one major class of quasi-experimental designs that are not included here -- the
interrupted time series designs. I plan to include them in later rewrites of this material.