Euroguidance Network Meeting
Impact Assessment Training
Prague
07th April 2016
Outline
1. Concepts and principles
2. Structuring an impact assessment
3. Collecting data
4. Analysing and validating data
5. Conducting an impact assessment
6. Choice of tools and techniques
7. Exercises
2
What is Impact?
Impact is a measure of the changes made
What actually
What we
What we What actually changed
wanted to
proposed happened (desired and
change
undesired)
3
What is Impact Assessment?
Impact assessment seeks to establish a causal
connection between inputs and changes.
Impact assessment should be seen as the
contribution of the outputs and outcomes to
purpose and overall goal.
The approach to impact assessment is very
similar to the planning that is required to
establish a monitoring system.
Main difference: the type of information and data
that are needed to be able to assess impact.
4
What is Impact Assessment?
Impact assessment is closely linked to an
organisation's mission
A mission statement is:
• A written declaration of an organization's core purpose and
focus that normally remains unchanged over time. Properly
crafted mission statements (1) serve as filters to separate what
is important from what is not, (2) clearly state which
stakeholders will be served and how, and (3) communicate a
sense of intended direction to the entire organization.
• A mission is different from a vision in that the former is the
cause and the latter is the effect; a mission is something to be
accomplished whereas a vision is something to be pursued for
that accomplishment.
5
Importance of Impact assessment
Measuring effectiveness of organisational
activities and judging significance of
changes
Closely linked to the objectives of an
organisation / Euroguidance
Powerful way of communicating,
internally and externally the contribution
of given activities to the identified
Mission.
6
Key elements
Evidence-based
Independence
Quality control
7
A multi-dimensional approach
•Survey
COLLECTION
DATA
•Focus group
•Interviews Convergence of multiple
sources
•Past
evaluations
RESEARCH
DESK
•Monitoring
•Statistics
8
Instruments, tools and methods
Stucturing Tool A
Collection Tool B
Analysis Tool C
….
Judgement
Methods
9
Methodological choice
- Econometric models
- Cost-benefit analysis
- Cost-effectiveness analysis
- Case studies Objective of the assessment
- Surveys
- Expert panels
- Statistical analysis Prospective or retrospective
- Desk research
Dimensions of the assessment
- Interviews
-Focus groups
- Logical framework Phase of the assessment
Internal and external resources, deadlines, …
10
Identify
problem/issue
Analyse
actual vs. Further analyse
expected problem/issue
Evaluation of Impact
Monitor results Set broad
and assess goals/objectives
to be achieved
progress
Impact Assessment for
Policy design
Identify
Implementation « best »
solution
Monitoring
Adoption Finalise
objectives
Identify
Progress/Success
Indicators
11
Outline
1. Concepts and principles
2. Structuring impact
3. Collecting data
4. Analysing and validating data
5. Conducting an impact evaluation
6. Choice of tools and techniques
12
Instruments for structuring impact
• The Logical Framework
• SWOT
• Stakeholder matrix
13
The Logical Framework
• Helps clarify the objectives
• Facilitates evaluation by translating the
objectives into a hierarchy of expected
effects
• Suggests questions about the effects
• Helps judge the internal coherence of the
intervention
• Provides the starting point for integrating
impact assessment within monitoring and
evaluation
14
Developing the logical framework
Objective tree
Effects tree
Logical framework
15
Developing the logical framework
General
objective
Priorities
Sub priorities
16
Effects tree
Expected
Expected results Expected impacts
outputs (short term) Expected Expected
Expected
intermediate intermediate
Output result impact impact
Output
Expected
Expected Expected
result intermediate
Output intermediate
impact impact
Expected Expected
Output
intermediate global
impact impact
Output Expected
Expected intermediate Expected
result impact intermediate
Output impact
Expected
result
17
Logical framework
Objectively Means of Assumptions
verifiable verification and risks
indicators
Global
objective
Priorities
Impacts – Impact indicators –
assessment of the changes
changes made Quantitative and
(positive, negative,
by action(s)
intended, unintended) qualitative
techniques
made by action(s)
Results
Activities
inputs costs 18
SWOT analysis
Aid to strategic decision-making :
Analyses - strengths and weaknesses (internal)
- opportunities and threats (external)
Aims to:
• Highlight the dominant and determining factors
• Produce relevant strategic guidelines
19
SWOT analysis
• A “scan” of the environment
• Preparation of an inventory of possible actions
• Internal analysis of strengths and weaknesses
• External analysis of opportunities and threats
• Highlight the dominant and determining
factors
• Classification of possible actions
• Produces relevant strategic guidelines
20
21
Stakeholders matrix
1. Identify who your stakeholders are. Be
precise. Identify their needs
2. Define the best approach for addressing
their needs.
22
Stakeholders needs analysis
Name Needs Response
Guidance counsellors
23
Stakeholders
• Policy and decision makers (specify)
• Guidance counsellors
• Students and learners (specify)
• professional / sectoral bodies
• local / regional authorities
• researchers / think tanks
• companies / entrepreneurs
• recruitment specialists
• awarding bodies
• mobility practitioners
24
Outline
1. Concepts and principles
2. Structuring an evaluation
3. Collecting data
4. Analysing and validating data
5. Conducting an impact evaluation
6. Choice of tools and techniques
25
Collecting data
•Desk research
• Individual Interviews
• Group interviews
• Questionnaire surveys
• Case studies
26
Desk research
1. Tool to collect existing information
2. Identification of the sources of
information
3. Understanding the scale of a
phenomenon
4. Definition of the scope of research
5. Identification of the channels to use
6. Implementation of the research
7. Synthesis of results
27
Individual Interviews
1. Useful to observe change
2. Collect opinion and information
3. Selection of the interviewees
4. Planning the interview
5. Selection and training of interviewers
6. Course of the interview
7. Analysis of results
28
Group interview
1. Useful to observe changes
2. Selection of participants
2. Defining the interview topics
3. Choice and training of facilitators
4. Analysis and report on results
5. Risk of dominance of majority of opinion
29
Questionnaires surveys
1. Tool to observe changes
2. Collect diverse information, opinions
but also facts
3. Designing the questionnaire
2. Sampling
3. Pre-test or pilot
3. Administration of the questionnaire
4. Codifying the data
5. Interpreting and disseminating the
results
30
Case studies
1. In-depth study of an action in a natural
setting, drawing on a multitude of perspectives
2. Illustration of a general situation
3. Study of good or bad practices
4. Study of certain key aspects of an intervention
5. Study of the effects of an initiative
6. Component of a multiple case study carried out
in view of generating an overall assessment
31
Outline
1. Concepts and principles
2. Structuring an evaluation
3. Collecting data
4. Analysing and validating data
5. Conducting an impact evaluation
6. Choice of tools and techniques
32
Analysing and validating data
• Descriptive statistics
• Regression analysis
• Sampling
• Bibliometric analysis
• Delphi survey
• Comparative analysis
33
Outline
1. Concepts and principles
2. Structuring an assessment
3. Collecting data
4. Analysing and validating data
5. Conducting an impact assessment
6. Choice of tools and techniques
34
Some tools
• Experts
• Multicriteria analysis
• Cost-benefit analysis
• Cost-effectiveness analysis
35
Experts
1. Judgement on the value of a
programme and its effects
2. Identification of a list of potential
experts
2. Selection and mandating of the experts
3. Investigations
4. Synthesis
36
Multicriteria analysis
1. Compare alternatives taking into
account multiple criteria including
prospective and/or retrospective
situations
2. Definition of judgement criteria
3. Analysis of the impacts of the
actions
4. Judgement of the effects of the
actions in terms of each of the
selected criteria
5. Aggregation of judgements
37
Cost- benefit analysis
1. Tool to make a judgement
2. In depth evaluation
3. Calculate the net impact of a
project, where impacts are
measured in monetary units
4. Analysis to determine whether a
project is desirable from the
viewpoint of the population
38
Cost- effectiveness analysis
1. Requires detailed data
2. Measure the impact
3. Highlight the relationship between the
main effect of an intervention and its
cost
4. Easy to understand conclusions that
reflect a key public preoccupation
5. Only useful to compare programmes that
are simple to implement and have the
same type of impact
39
Examples
50 Millions € programme
Objective: to support the European
cinematographic production
Results: coproduction of 20 films that were
seen by 12 000 000 persons
Unit cost: 4,16 € / person
40
Exercises
• Mission statement
• Benchmarks identification
• The pyramid model
• The logical framework
• The SWOT analysis
• The stakeholder matrix
41
Exercise 1 – Mission statement
Define a mission statement for
Euroguidance network
You can get inspiration from the EU mission statement
42
Example of mission statement
EU’s mission is:
• To guarantee peace, freedom and security in and
around Europe.
• To promote and protect democracy and universal
rights in Europe and around the world.
• To strengthen Europe's economy and to promote
solidarity around Europe by working in partnership
with national, regional and local government.
• To make it easy for Europe's citizens to live and
work throughout the Union.
43
Exercise 2 – Indicators identification
Providing and
Providing quality
Promoting the maintaining input
information on
European to the "Learning
lifelong guidance
dimension in Opportunities and
and mobility for
lifelong guidance Qualifications in
learning purposes
Europe" Portal
Indicators
44
Exercise 3: The pyramid
Write on the side your suggestions
General objective
Priorities
Sub priorities
45
Exercise 4: Logical framework
Fill in the table
Objectively Means of Assumptions
verifiable verification and risks
indicators
Global
objective
Priorities
Impacts –
assessment of
changes made
by action(s)
Results
Activities
46
Exercise 5: SWOT ANALYSIS
Identify up to 5 strengths and 5 weaknesses internal to your
organizations
Strengths Weaknesses
47
Exercise 5: SWOT ANALYSIS
Identify up to 5 opportunities and 5 threats external to your
organizations
Opportunities Threats
48
Exercise 6: Stakeholders needs analysis
Fill in the table (be as precise as possible)
Name Needs Response
Guidance counsellors
49
Contacts
[email protected]
[email protected]
51
