USING ADULT DATA
TO INFORM
DECISION-MAKING
February 27, 2024
1
Facilitator
Amanda Nickerson
CLSD TA Specialist
Objectives
Understand Guskey's Five Critical Levels of PD
Theory
Evaluation.
Identify various data sources to evaluate
Best Practices
various dimensions of professional learning.
Utilize the professional learning decision
Resources
framework to scrutinize adult data.
Session Outline
New Content Reflection and Action Debrief and
Learning Discussion Planning Feedback
45 minutes 10 minutes 30 minutes 5 minutes
QUESTIONS?
New Content Learning
fa
EDUCATION.OHIO.GOV
45 minutes
5
K-5 CLSD CoP Resources
Delivering, Supporting, Evaluating and Sustaining Professional Learning
QUESTIONS?
Supportingfa
EDUCATION.OHIO.GOV
Adult Learners
7
Maintaining Momentum
"To build educator capacity to deliver
instruction that is aligned to the science of
reading and the foundational components of
Ohio’s Plan to Raise Literacy Achievement
will require leaders to provide intensive,
embedded, collaborative, and
instructionally focused professional
learning and coaching” (ODE Literacy
Achievement and Reading Success, 2023, p.
3).
Maintaining Momentum
What does implementation look like now?
Are we satisfied with how implementation looks?
What would we like implementation to look like?
What would we need to do to make implementation look like that?
How would we know if we’ve been successful with implementation?
What can we do to keep implementation like that?
What can we do to make implementation more efficient & durable?
(Adapted from Sugai, 2004)
Phases of Implementation
Exploration & Program Initial Full
Innovation Sustainability
Adoption Installation Implementation Implementation
Assess need; Assure resources; Initiate practice; Practice is Refine practices; Provide ongoing
Examine fit develop supports Use data to consistent; Differentiate and data-driven support
and feasibility improve supports positive outcomes improve fit
Phases of Implementation
Exploration & Program Initial Full
Innovation Sustainability
Adoption Installation Implementation Implementation
• Programs need various forms of data to diagnose issues that emerge during implementation.
• Data disambiguates whether there is a process, program, or performance issue.
• Continuous improvement is continuous. It is recursive and doesn't have an expiration date.
Implementation Drivers
Training
Adult Data
Leadership
Decision Support Data System (DSDS)
"A Decision Support Data System (DSDS) is a system for
identifying, collecting, and analyzing data that are useful to the staff
and leadership of the implementing agencies. The data system
needs to provide timely, accurate, and reliable data for decision-
making" (NIRN & SISEP, 2013).
13
Different Types of Data
Program or Process Fidelity and Integrity Outcome
Data that are relevant to Data that measures the extent to which Results data, measures the impact of
administration of the strategy, the strategy, program, or practice has the strategy, practice, or program
program, or practice been implemented as intended
Examples:
Examples: Constructs: • Improved achievement
• Enrollment • Adherence • Improved skills or behavior
• Reach or scale • Dosage/Exposure • Teacher retention
• Cost of participation • Quality/Competency • Increased knowledge
• Financial data • Participant Responsiveness • Changes in beliefs or perceptions
• Staffing • Component Differentiation
Examples:
• Observation
• Self-Report
• Product Reviews
• Interviews
(NIRN & SISEP, 2013)
Competency Drivers
Selection Training Coaching
1 2 3
"Competency Drivers are mechanisms to develop, improve, and
sustain one’s ability to implement an innovation as intended to
benefit students" (NIRN & SISEP, 2013).
Improving Professional Learning
Typical Professional Development Optimal Professional
Learning
● free standing ● aligns with school, district and state
● short-term priorities
● nonsystematic ● focuses on core content and modeling
● infrequent ● includes opportunities for active learning
● sit-and-forget and applications of new strategies
● smattering ● includes opportunities for collaboration
● smorgasbord ● involves follow-up and feedback
smattering sit-and-forget smorgasbord surface deep transfer
A Closer Look at Training
SURFACE DEEP TRANSFER
High-Quality Professional Learning
Lead Support Sustain
Professional Professional Professional
Learning Learning Learning
● Use standards to ensure quality ● Utilize varied formats to support ● Evaluate professional learning
● Conduct needs assessments at professional learning ● Use adult implementation data
the macro- and micro-levels ● Create a purposeful professional to inform how professional
● Align PL with local and state learning plan/sequence learning is led and supported
goals and priorities ● Remove barriers and create ● Create a feedback loop and/or
● Design and lead PL that meets meaningful incentives decision rules to sustain
observed and perceived needs ● Foster accountability professional learning
● Foster accountability ● Commit and refine the PL plan ● Foster accountability
● Commit to being a “lead learner” over time ● Commit and refine processes
over time
QUESTIONS?
Evaluating
fa
EDUCATION.OHIO.GOV
Adult Data
19
Why Evaluate?
We use evaluations to determine the value of
something—to help answer important questions:
Is this program or activity achieving its intended results?
Is it better than what was done in the past?
Is it better than another, competing activity?
Is it worth the costs?
(Guskey, 2002)
Why Evaluate?
"Good evaluations don't have to be complicated.
They simply require thoughtful planning, the ability
to ask good questions, and a basic understanding
of how to find valid answers. What's more, they can
provide meaningful information that you can use to
make thoughtful, responsible decisions about
professional development processes and effects."
(Guskey, 2002)
Guskey's Five Critical Levels of PD Evaluation
Level 1: Participants' Reactions
Did participants feel their time was well spent?
Did the material make sense to them?
Were the activities well planned and meaningful?
Was the leader knowledgeable and helpful?
Did the participants find the information useful?
Level 1: Participants' Reactions
Did participants feel their time was well spent?
Did the material make sense to them?
Were the activities well planned and meaningful?
Was the leader knowledgeable and helpful?
Did the participants find the information useful?
Level 2: Participants' Learning
Can participants describe the crucial attributes of a particular routine or skill (e.g., using
Elkonin boxes; explicit vocabulary routine; querying)
Can participants give examples of how these might be applied in typical classroom situations?
When prompted, can participants apply knowledge and skills acquired through professional
learning?
Presented with a variety of instructional challenges, can participants diagnose each situation to
apply relevant knowledge and skills?
Can participants prescribe and carry out an appropriate and actionable plan?
Are participants applying knowledge and skills as intended?
Level 3: Organization Support and Change
Did the professional development activities promote changes
that were aligned with the mission of the school and district?
Were changes at the individual level encouraged
and supported at all levels?
Were sufficient resources made available, including time for
sharing and reflection?
Were successes recognized and shared?
Level 4: Participants' Use of New Knowledge and Skills
Did the new knowledge and
skills that participants learned
make a difference in their
professional practice?
Level 5: Student Learning Outcomes
How did the professional
development activity affect students?
Did it benefit them in any way?
Determining Needs and Supports
1 Identify data sources to
illuminate professional
learning needs.
6 Create a professional
learning scope and sequence
2 Review and analyze these
w ith discrete next steps and data.
measurable goals.
5 Determine how this w ork will
be integrated into sustainable 3 Determine know ledge and
skills to be directly taught
district/building
and/or modeled.
plans/activities.
4 Determine how coaching and
collaboration w ill be leveraged
to refine implementation.
Data Sources: Revisited
Level 1: Level 2: Level 3: Level 4: Level 5:
Participants' Participants' Learning Organization Support Participants' Use of New Student Learning
Reactions and Change Know ledge and Skills Outcom es
Self-reported Data Self-reported Data Self-reported Data Self-reported Data Student Data: Cognitive
• Prof essional learning • Teacher surv ey s • Questionnaires • Questionnaires • Student assessment data
f eedback responses • Coaching requests • Structured interv iews • Structured interv iews (e.g.,
using a mix of rating • OTES goals • Personal ref lections (oral or screener, diagnostic,
scale and open-ended • Personal ref lections (oral or Meeting Minutes written) progress-monitoring,
questions (can be written) • Teacher-Based • Portf olios outcome assessment)
univ ersally used across • Portf olios Team (TBT) notes • Student work samples
prof essional learning • Building Leadership Team Observational Data (e.g., v ocabulary
sessions) Teacher Knowledge Assessments (BLT) notes • OTES observ ation notes template; phonics
• Ohio's Teacher Knowledge • District Leadership Team • Walkthrough data dictation sheet)
Assessment (DLT) notes (e.g., v ocabulary
• Prof essional learning asses • Multidisciplinary Team routine; section of Student Data: Affective
sments (e.g., LETRS unit a (MDT) notes the phonics • Attitudes
ssessments) lesson; read aloud • Dispositions
• Locally created assessment District and Building Records lesson to teach
s (e.g., brief pretest, post- • R-TFI sentence Student Data: Psychomotor
test, exit tickets) • Local Literacy Plan structure/sy ntax) • Skills
• Coaching records • Video observ ation notes • Behav iors
Simulations • Lesson plans
School-wide Indicators
Full-scale Demonstrations • Attendance
• Disciplinary actions
• Retention
• Participation in school-
related activ ities
• Homework completion
Backwards Planning
LEVEL 5: Which student learning outcomes do you want to
achieve?
• Critical analyses of relevant data from assessments of student
learning, examples of student work, and school records are especially useful in
identifying specific goals.
LEVEL 4: What instructional practices and policies will most
effectively and efficiently produce those outcomes?
• What evidence verifies that these particular practices and policies will lead to
the desired results?
• How good or reliable is that evidence?
• Was it gathered in a context similar to ours?
31
Backwards Planning
LEVEL 3: What aspects of organization support need to be in place for
those practices and policies to be implemented?
• Consider barriers to implementation and look for ways to provide support.
LEVEL 2: What knowledge and skills must professionals have to implement
the prescribed practices and policies?
• Consider what educators and leaders must know and be able to do to improve student
outcomes.
LEVEL 1: What experiences will help educators and leaders acquire
necessary knowledge and skills?
• Which format(s) will you use to facilitate surface learning?
• Which format(s) will you use to facilitate deep learning?
• Which format(s) will you use to facilitate transfer?
32
Backwards Planning
Graphic by Marsyitah Ismail 33
The Implementation Formula for Success
Socially Effective Enabling Effective
Significant
Outcomes =
Implementation
x
Context
x Practices
Student Learning Participants' Use of New Organizational Support Participants' Learning
Outcomes Knowledge and Skills and Change Participants' Reactions
ns
ns ns
ns
(Adapted from Fixsen et al., 2015)
34
QUESTIONS?
Reflection & Discussion
fa
EDUCATION.OHIO.GOV
10 minutes
35
Reflection
Examine the Adult Data Decision-Making Framework:
https://docs.google.com/document/u/1/d/1l8SDkerFre6VYD
L7p7UB8nC_I35UF1gcKJUQfkaTyZw/copy?usp=sharing
Stop and jot for 5 minutes:
• How have you monitored professional learning (PL)
across the various levels?
• How could you implement a deeper level of evaluation
to inform PL?
Discussion
In your small group, discuss the following for five minutes:
• How have you monitored professional learning (PL) across the
various levels?
• How could you implement a deeper level of evaluation to inform PL?
As you listen, consider:
• What resonates with you when you listen to others?
• What is the same or different to what you jotted down?
QUESTIONS?
Action Planning
fa
EDUCATION.OHIO.GOV
30 minutes
38
Action Planning
As you continue to review the Adult Data Decision-Making Tool, consider
how you might support and sustain adult implementation moving
forward:
• What will you START doing to use adult data more purposefully?
• What will you STOP doing because it is interfering with your plans to
sustain this work?
• What will you CONTINUE doing because it provides opportunities to
leverage past and present successes?
Action Planning
What will you What will you What will you
START doing? STOP doing? CONTINUE
doing?
QUESTIONS?
Debrief & Feedback
fa
EDUCATION.OHIO.GOV
5 minutes
41
Debrief & Feedback
What is your biggest takeaway
from this session?
What is your next step when you
return to your district?
education.ohio.gov/ReadOhio
