Evaluation
A Systematic Approach
Peter H. Rossi, Mark W. Lipsey, and Gary T. Henry
1
Chapter 1:
What Is Program Evaluation
and Why Is It Needed?
What is Program Evaluation?
• Evaluation is usually focused on social interventions.
• Can be used in a variety of sectors
– Politics
– Advertising
– Government work
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 3
Why Is Program Evaluation
Needed?
• Ensuring that the program meets outcomes
• Main tasks of program evaluation:
– Assess the effectiveness of social programs
– Identify the factors that drive or undermine their
effectiveness
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 4
Why Systematic Evaluation?
• The relativity of program effects
• Difficulty in identifying program effects brief
• Confirmation bias: the tendency to see things in ways
that favor preexisting beliefs
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 5
Systematic Program Evaluation
• Program evaluation: applying social research methods
to systematically investigate the effectiveness of social
intervention programs
• Peter H. Rossi was a pioneer in the field.
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 6
Application of Social Research
Methods
(1 of 2)
• The concept of evaluation entails:
– A description of the performance of the entity being
evaluated
– Some standards or criteria for judging that
performance
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 7
The Effectiveness of Social
Programs
(2 of 2)
• A program evaluation generally involves assessing one
or more of five domains:
(a) the need for the program
(b) its design and theory
(c) its implementation and service delivery
(d) its outcome and impact
(e) its efficiency
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 8
Adapting to the Political and
Organizational Context
• Evaluation sponsor: person who commissions the
evaluation
• Stakeholders: individuals, groups, or organizations with
a significant interest in how well a program is working.
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 9
Influencing Social Action to Improve
Social Conditions
(1 of 2)
• Purpose of an evaluation:
– Provide useful answers to questions about a program
– To influence action
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 10
Influencing Social Action to Improve
Social Conditions
(2 of 2)
• Demonstration programs: social intervention projects
designed to test the value of an innovative program
concept.
• The findings are significant because of what they reveal
about the program concept and how promising it is
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 11
The Central Role of Evaluation
Questions
(1 of 3)
• The Purpose of the Evaluation:
– Help management improve a program
– Support advocacy by proponents or critics
– Gain knowledge about the program’s effects
– Provide input for decisions
– Respond to political pressures
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 12
The Central Role of Evaluation
Questions
(2 of 3)
• The Evaluator-Stakeholder Relationship:
– Decision-makers, Program sponsors, Evaluation
sponsors, Target participants, Program managers,
Program staff, Program competitors, Contextual
stakeholders, Evaluation and research community.
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 13
The Central Role of Evaluation
Questions
(2 of 3)
• The Evaluator-Stakeholder Relationship
– Independent evaluation
– Participatory or collaborative evaluation
– Empowerment evaluation
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 14
The Central Role of Evaluation
Questions
(3 of 3)
• Criteria for Program Performance
– Performance criterion
– Professional standards
– Specific program objectives
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 15
The Central Role of Evaluation
Questions
(3 of 3)
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 16
The Five Domains of Evaluation
Questions and Methods
(1 of 7)
• Need for the program
• Program theory and design
• Program process
• Program impact
• Program efficiency
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 17
The Five Domains of Evaluation
Questions and Methods
(2 of 7)
• Need for the Program: Needs Assessment
– First step in planning a new program
– Provide guidance for improvement
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 18
The Five Domains of Evaluation
Questions and Methods
(3 of 7)
• Assessment of Program Theory and Design
– Must reflect valid assumptions about the nature of the
problem
– Must represent a feasible approach
– Often in the form of a logic model
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 19
The Five Domains of Evaluation
Questions and Methods
(4 of 7)
• Assessment of Program Process
– Process evaluation or an implementation assessment
– Program monitoring
– Outcome monitoring
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 20
The Five Domains of Evaluation
Questions and Methods
(5 of 7)
• Effectiveness of the Program: Impact Evaluation
– Gauged by the change it produces in outcomes
– Asks whether the desired outcomes were actually
attained
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 21
The Five Domains of Evaluation
Questions and Methods
(6 of 7)
• Cost Analysis and Efficiency Assessment
– Cost analysis
– Efficiency assessment
– Cost- benefit or cost-effectiveness analysis
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 22
The Five Domains of Evaluation
Questions and Methods
(7 of 7)
• The Interplay Among the Evaluation Domains
– Implementation failure
– Theory failure
Peter H. Rossi and Gary T. Henry, Evaluation: A Systematic Approach 8e. SAGE Publishing, 2019. 23
