INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Instructional Policy and Classroom Performance:

The Mathematics Reform in California

David K. Cohen
Heather C. Hill

CPRE Research Report Series

RR-39

Consortium for Policy Research in Education

University of Pennsylvania
Graduate School of Education

© Copyright 1998 by the Consortium for Policy Research in Education

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Contents

Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Authors’ Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

The Reform: Policy and Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Professional Learning and Reform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Opportunities to Learn and Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Impact of Opportunities to Learn on Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

The Mediating Role of Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Effects on Student Achievement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

Appendix C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

End Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

CPRE Research Report Series, RR-39

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Biographies

David K. Cohen is John Dewey Collegiate Professor of Education, and Professor of Public
Policy at the University of Michigan. In addition to his current work on educational policy and the
relationships between policy and practice, his previous research includes studies on the effects of
schooling, efforts to reform teaching, evaluation of educational experiments and large-scale
intervention programs, and relations between research and policy.

Heather C. Hill is a doctoral candidate in the Department of Political Science at the University of
Michigan. In addition to her current work with David K. Cohen on mathematics reform in
California, she is completing her dissertation on the role non-governmental resources play in
teaching enactors about public policy and changed educational practices.

Authors’ Note

This paper is part of a continuing study of the origins and enactment of the reforms, and their
effects, that was carried out, beginning in 1988, by Deborah Loewenberg Ball, David K. Cohen,
Penelope Peterson, Suzanne Wilson and a group of associated researchers at Michigan State
University. The research was supported by a grant (No. OERI-R308A60003) to CPRE from the
National Institute on Educational Governance, Finance, Policymaking and Management, Office of
Educational Research and Improvement, U.S. Department of Education, and by grants from the
Carnegie Corporation of New York, and The Pew Charitable Trusts, to Michigan State
University. We are grateful to these agencies, but none is responsible for the views in this paper.

An earlier version of this paper was presented at the 1997 meeting of AERA, and draws on a
larger book manuscript. This paper is not for any other publication or use without our written
permission. Address correspondence to either Cohen or Hill at the School of Education, The
University of Michigan, 610 East University, Ann Arbor 48109-1259.

CPRE Research Report Series, RR-39 i

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Abstract
Educational reformers increasingly seek to manipulate policies regarding assessment, curriculum,
and professional development in order to improve instruction. They assume that manipulating
these elements of instructional policy will change teachers’ practice, which will then improve
student performance. We formalize these ideas into a rudimentary model of the relations among
instructional policy, teaching, and learning. We propose that successful instructional policies are
themselves instructional in nature: because teachers figure as a key connection between policy
and practice, their opportunities to learn about and from policy are a crucial influence both on
their practice, and, at least indirectly, on student achievement. Using data from a 1994 survey of
California elementary school teachers and 1994 student California Learning Assessment System
(CLAS) scores, we examine the influence of assessment, curriculum, and professional
development on teacher practice and student achievement. Our results bear out the usefulness of
the model: under circumstances that we identify, policy can affect practice, and both can affect
student performance.

CPRE Research Report Series, RR-39 ii

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Introduction more central authority for instruction by

devising and implementing intellectually
remarkable realignment ambitious curriculums and assessments. By

A occurred in American education

between 1980 and 1994. The
era began as a conservative
president vowed to abolish the federal
Bill Clinton’s 1992 inauguration, many states
were moving more forcefully on instruction,
and many sought coordinated change in
instructional frameworks, curriculum, and
assessment.
Department of Education and turn schooling
back to states and localities, but the
Department of Education persisted, and The reformers faced two central problems.
Ronald Reagan’s administration exerted an One was political: power and authority were
impressive centralizing influence on public extraordinarily dispersed in U.S. education,
education. It helped to mobilize powerful especially in matters of instruction. Could
national pressures for better academic state or national agencies actually steer
performance, stiffer state and national teaching and learning in thousands of far-
standards, and even stiffer state and perhaps away classrooms?2 Reformers argued that
national tests. Ironically, conservatives new assessments, or instructional
helped to push public education toward frameworks, or professional development, or
much more power for central agencies in some combination of them, would do the
state and perhaps even national government. trick, but such things are unprecedented in
Some members of Reagan’s administration the United States. The other problem was
even attacked local control of schools as a pedagogical: reformers wanted teaching and
dangerously outmoded idea. learning to become much more thoughtful
and demanding, but researchers reported that
The same years also saw dramatic changes in most teaching in U.S. schools was no better
ideas about the purposes and content of than basic. That was a key argument for
schooling. School improvement had focused reform, but it also raised a question: Can
on the “basics” in the mid-1970s and early anyone steer teaching and learning so sharply
1980s,1 but by the end of Reagan’s first term away from long-established practice?
researchers and reformers had begun to Reformers say that instead of just offering
argue for more intellectually ambitious the basics, teachers must help students to
instruction. They contended that teaching understand mathematical concepts, to
and learning should be more deeply rooted in interpret serious literature, to write creatively
the disciplines and much more demanding. about their own ideas and experiences, and
Reformers also began to argue that schools to converse thoughtfully about history and
should orient their work to the results that social science. But ever since researchers
students achieve rather than the resources began to investigate instruction, they have
that schools receive. been reporting that most of it was dull, and
that intellectual demands generally were
Politicians, business leaders, and educators modest. Recent research shows that few
proposed fundamental changes in politics teachers have deep knowledge of any
and policy to achieve these new goals. academic subject, especially in elementary
Beginning with California in the mid-1980s, schools. Until now the sort of instruction
state education agencies began to exercise that reformers proposed has mostly been

CPRE Research Report Series, RR-39 1

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

confined to protected enclaves in a few student achievement through teachers’

public and private secondary schools. practice.

As instructional policy moved to the top of The Reform: Policy and

many states’ education agendas in the past
ten or fifteen years, it raised fundamental Instruments
questions about the relations between policy
and practice. Researchers began to State reform of mathematics instruction in
investigate those questions, and we continue California has been remarkable both for the
the effort here. Using data from a 1994 sustained energy that reformers and
survey of California elementary school educators brought to the enterprise and for
teachers, we probe the classroom effects of the controversies that ensued. The
state efforts to reform mathematics teaching California Department of Education took the
and learning in California. In order to do so first step in 1985, when it issued a new
we devised a model of the relations between Mathematics Framework, and the endeavor
policy and practice. Like more and more continues today, though much modified.
states, California sought to improve student This state reform has been one of the longer-
achievement by using state policies and other running efforts in the history of U.S.
means to manipulate a range of instruments education.
that are specific to instructional policy,
including student curriculum, assessments, The 1985 Mathematics Framework called for
and teachers’ knowledge, beliefs, and much more intellectually ambitious
practices. Notice that the effective operation instruction, for more mathematically
of these instruments would depend in engaging work for students, and for teachers
considerable part on professionals’ to help students understand math rather than
learning—that is, teachers would have to just memorizing facts and operations. The
learn new views of mathematics and math Framework was a central part of state
teaching from the revised assessments and instructional policy, though it was formally
student curriculum, in order for the policies only advisory to local districts. It
to affect practice. Teachers’ opportunities to encouraged teachers to open up discourse
learn would be a key policy instrument. about math in their classrooms, to pay more
attention to students’ mathematical ideas,
In the pages that follow we develop this and to place much more emphasis on
rudimentary model: students’ achievement is mathematical reasoning and explanation
the ultimate dependent measure of rather than the mechanics of mathematical
instructional policy, and teachers’ practice is facts and skills.
both an intermediate dependent measure of
policy enactment and a direct influence on Shortly after issuing the new Framework, the
students’ performance. Teachers, therefore, State Board of Education tried to use
figure in the model as a key connection textbook adoption as an instrument of the
between policy and practice. Teachers’ policy; state approval carries great weight
opportunities to learn what the policy implies with localities because they receive state aid
for instruction is a crucial influence on their for using approved texts.3 The State Board
practice, and at least an indirect influence on used the Framework to reject most texts.
After much debate, some negotiation, and a

CPRE Research Report Series, RR-39 2

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

good deal of acrimony, some of the books once the state began to test students on new
were somewhat revised, and the Board mathematical content and methods, scores
declared most of them fit for children’s use. would drop because the material would be
But state officials were not happy with the unfamiliar and more difficult. Teachers and
result and decided that text revision might the public would notice the lower scores,
not be the best way to encourage reform. which would generate pressure for better
results; teachers would pay attention to the
Reformers then began to encourage the pressure and thus to the new tests, and
development of other curriculum instruction would change. As one state
materials—small, topic-centered modules official told us, “...tests drive instruction.”4
called “replacement units”—that would
support changed math teaching without The Department of Education had some
challenging textbook publishers. The difficulty revising the tests, in part because it
California Department of Education also was a formidable task, in part because of Bill
tried to encourage professional development Honig’s disputes with then-Governor
for teachers around the reforms, although Dukmejian, and in part because of Honig’s
continuing budget cuts had weakened the own tribulations and trials. But the revisions
Department’s capacity to support such work. finally were completed and the new tests
The new Mathematics Framework called for were administered in 1993 and 1994. As
a substantial shift in teachers’ and students’ state education leaders had thought, scores
views of knowledge and learning, toward were lower and the public noticed, but that
views that most Americans would see as understates the matter: a storm of protest
unfamiliar and unconventional. If the new erupted after the 1993 test results were
ideas were to be taken seriously, teachers published. Not only were scores generally
and other educators would have a great deal quite low, but a technical panel also gave
to learn. Moreover, the Framework offered low grades to some features of the
such general guidance that the California assessment and its administration. Things
reform was quite underspecified. That was were modified for 1994, partly in response to
only to be expected, both because the ideas the outcry over low scores, but it was too
were relatively new to most advocates and late, for the opposition had organized an
hence underdeveloped, and because assault on the whole enterprise. Conserv-
reformers wanted complex teaching that atives criticized the new tests on the grounds
could only be constructed in response to that they gave little attention to the “basics”
students’ ideas and understandings, and thus and instead encouraged “critical thinking,” or
could not be captured in any set recipes. “outcomes-based education,” activities that
many rejected. Questions were raised about
The California Department of Education the technical quality of the test and its
used its student assessment system as administration, reporting, and analysis,
another means to change teaching, and especially the “subjectivity” of items and
devoted considerable attention to revising scoring. Governor Wilson was running for
the tests so they were aligned with the new the Republican Presidential nomination at the
Framework. Though some reformers were time; he attacked and then canceled the
uneasy about testing, others assumed that testing program.
new tests could help. They reasoned that

CPRE Research Report Series, RR-39 3

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Professional Learning and As members of the Michigan State research

group studied classrooms and mathematics
Reform teaching in California, we soon saw that the
reforms entailed extensive learning: they
Larry Cuban (1984) once wrote of such could not be enacted unless educators,
political controversies that they only weakly parents, and policymakers revised many
affect schools and classrooms. Like storms beliefs about mathematics, teaching, and
on the surface of a deep ocean, they roil the learning, and developed new ways to teach
surface but have little impact on develop- and learn mathematics. Unless one believed
ments further below. Much research on that everyone could do all that on his or her
policy implementation has probed the failures own, implementation of these reforms would
of central policies to shape practices in have to include many learning opportunities
street-level agencies, but most researchers that did not exist in 1985.7 California’s
seemed to assume that policy was normative instructional policy could be thought of as
and practice should follow suit. They wrote implying a program for the re-education of
from the perspective of policy, trying to teachers and others concerned with schools.
explain why practice had gone awry. Only a Since teachers would have to teach the
few tried to understand practice, or to dramatically new curriculum for students that
consider policy from the perspective of policymakers had proposed, and since few
practice.5 teachers could teach as the new Framework
advised, the policy could not be enacted
The research reported here began nearly a unless these professionals had many
decade ago in a research group at Michigan opportunities to learn new conceptions of
State University.6 Members of that group mathematics teaching and learning. If one
sought to learn about what was happening believed that teachers and parents could not
below the surface of policy in California and teach and learn the new policy on their own,
several other states, and to use it to improve then implementation would depend on the
understanding of policy and its implement- actions of state and other agencies that might
ation. One of us worked with that group, create opportunities for teachers to learn.
studying documents, visiting elementary
classrooms in several schools in three school From that perspective it seems that the
districts in California, and following the same connections between policy and practice,
teachers for four or five years. We also between Sacramento and local schools,
followed developments in state and district would be crucial. If implementation was in
offices, interviewing many state and district part a matter of teaching professionals and of
administrators and reformers, and studying their learning, and if most teachers could not
efforts to improve teachers’ knowledge and do it all by themselves, then some agencies
skill in various professional development would have to do the teaching, and
projects. But the project staff considered its encourage the learning. This implies that the
local work to be crucial: not only would relations between events on the surface of
reform be made or broken in schools and policy and far beneath that surface would be
classrooms, but past research had tended to significant, and that the content of those
ignore that part of the story, or to touch on it relations would in a sense be instructional. If
quite incompletely. so, then the key issues about those relations

CPRE Research Report Series, RR-39 4

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

would be similar to those one might leading reform ideas, their opportunities to
encounter in any case of teaching and learn about improved mathematics
learning: What opportunities did teachers instruction, and their reports of their
and other enactors have to learn? What mathematics teaching.
content were they taught? Did teachers who
reported participating in these opportunities These ideas imply a conception of the
report a different kind of practice than those relations between policy and practice in
who did not have them? Analysts would which teachers’ opportunity to learn would
investigate who taught the new ideas and be a critical mediating instrument. But the
materials, and what materials or other content of those opportunities is not self-
guidance for learning teachers had. On this evident, and, if they might play the crucial
view it would not do to look solely beneath role we propose, a more precise idea is
the surface of policy, in practice; rather, one required. Our work and previous research
would want to look anywhere one could find suggests that several features would be
agents and opportunities that might connect central:
policy and practice via professional learning.
Beginning in 1988, the Michigan State • General orientation: exposure to key
research group explored some features of the ideas about reform.
response to reform in detailed longitudinal
field studies of teachers’ practice—how • Specific content: exposure to such
teachers understood the reforms, whether educational instruments as improved
their practice changed, and what learning mathematics curriculum for students,
opportunities they had. In 1994 we or assessments that inform teachers
supplemented those studies with a one-time about what students should know,
survey of 1,000 elementary school teachers, and how they perform.
in order to extend the breadth of our findings
about the extent of change in math teaching.8 • Consistency: the more overlap there
A survey instrument was designed, and a was among the educational
stratified random sample selected to instruments noted above, the more
represent the population of second through likely teachers’ learning would be to
fifth grade teachers in California. move in the direction that state policy
proposed.
Teachers’ Opportunity to Learn and
Practice. We report the initial analysis of • Time: teachers who had more
that survey data here. Our opening exposure to the educational
conjecture was that the greater the teachers’ instruments would be more likely to
opportunities to learn the new mathematics move in the direction proposed by
and how to teach it, the more their practice the state policy.
would move in the direction proposed by the
state policy. To probe that conjecture we These ideas imply two points about any
needed to know what learning opportunities analysis of the relations between instructional
the teachers had, what they learned, and policy and teaching practice. One is that we
what they did in math class. Thus, the view teachers’ reported practice as evidence
survey probed teachers’ familiarity with the of the enactment of state instructional policy,

CPRE Research Report Series, RR-39 5

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

and thus as a key dependent measure. The practice would influence students’
other is that we view teachers’ opportunities performance. But teachers’ practice is not
to learn as a bundle of independent variables the only influence on students’ learning.
that are likely to influence practice. Such a policy also could influence learning
by way of students’ exposure to specific
The connections between policy and practice educational instruments such as improved
thus are our central concern, and learning for mathematics curriculum, or tests that
professionals is one of several key directed teachers’ and students’ attention to
connecting agents. We conjecture that the goals and content of reform. Other
relations between events on and below the factors are also likely to influence either the
surface would depend less on the depth of opportunities that teachers are provided,
the water than on the extent to which their learning, or students’ learning.
government or other agencies built Inequalities among families would create
connections or made use of those already differences in students’ capacity to take
extant. Ours is thus an instructional model advantage of improved curriculum and
of instructional policy: although it seems an teaching, and inequalities among schools and
obvious way to explain variation in the communities could inhibit teachers’ capacity
effects of such policies, it has not been used to learn from new curriculum and
until now. assessments. Neither learning nor
opportunities to learn are independent of
Student Achievement. Teaching practice is politics, money, social and economic
not the sole outcome of interest. Students’ advantages, and culture. Hence we take
performance is no less important, since several of these into account in the analysis
reformers’ justification for asking teachers to that follows. But in developing our
learn new math instruction was that students’ conception of links between policy and
learning would improve. From that practice we keep most attention on factors
perspective, teachers’ practices become closest to the production of student
crucial intervening measures, for if growth—teachers’ learning and practices,
instructional reform was to affect most related curricula, and time.
students, it would be mainly through
teachers’ practice. While teachers’ practice is Opportunities to Learn and
a dependent measure of policy
implementation from one perspective, from Practice
another it is an independent measure that
mediates the effects policy may have on We want to know how teachers’ practice
students’ work, which is the final dependent compares with reform ideals, so we asked
measure. Therefore, we probe links between teachers to report on their classroom
teachers’ opportunity to learn, their practice, practice in mathematics along some of the
and scores on California’s math test in 1994. dimensions advocated by the new
In this conception of the relations between Mathematics Framework. But since we—
policy and practice, teachers’ learning and the reformers—were interested in
opportunities (their general orientation, change, we also wanted to know how their
specific content, consistency, and time) teaching compared with conventional
would influence their practice, and their practice, so we also asked teachers to report
on that. Both sorts of measures would be

CPRE Research Report Series, RR-39 6

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

required to probe whether teachers’ learning For now, we stick to the first part of this
opportunities influenced their practice, and investigation, asking whether teachers’
to explore whether reform-oriented practice practice is correlated with their own learning
is related to students’ achievement. opportunities. We start by more closely
defining how we measured “practice,” and
investigating how opportunities to learn are
distributed through California’s population
of teachers.

Table 1
Teacher Reports of Conventional Mathematics Practices

About how often do students in your class take part in the following activities during
mathematics instruction? (CIRCLE ONE ON EACH LINE.)
Never A few Once or Once or Almost
times a twice a twice a daily
year month week
Practice or take tests .6 9.7 33.4 42.6 13.7
on computational skills
Work individually on 3.0 4.3 9.7 38.1 45.0
mathematics problems
with the text/workbook

Q. 35. a. Which statement best describes your use of a mathematics textbook? (CIRCLE
ONE.)
A textbook is my main curriculum resource......................................................... 30.9

I use other curriculum resources as much as I use the 39.1

text.................................
21.0
I mainly use curriculum resources other than the text.........................................
9.1
I do not use a textbook. I use only supplementary
resources..............................
Note: Numbers are percentages of respondents selecting that category, weighted to represent
statewide population.

CPRE Research Report Series, RR-39 7

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Practice along which these items lined up. The first

consisted of more conventional instructional
Teachers’ self-reports of classroom practices activities (see Table 1). The responses to
associated with mathematics instruction were each item were individually standardized and
measured by fourteen survey items. A averaged by teacher to form the scale we call
factor analysis9 revealed two dimensions “conventional practice;” its mean is zero

Table 2
Teacher Reports of Framework Practices

9. About how often do students in your class take part in the following activities during
mathematics instruction? (CIRCLE ONE ON EACH LINE.)
Never A few Once or Once or Almost
times a twice a twice a daily
year month week
Make conjectures and 1.0 7.4 18.3 42.8 30.4
explore possible
methods to solve a
mathematical problem
Discuss different ways .9 5.0 14.6 46.6 32.9
that they solve
particular problems
Work in small groups 1.2 3.8 20.4 46.4 28.2
on mathematics
problems
Work on individual 23.5 36.7 26.5 10.6 2.6
projects that take
several days
Work on group 25.4 36.2 26.1 9.4 2.9
investigations that
extend for several
days
Write about how to 11.6 16.9 33.7 28.8 9.0
solve a problem in an
assignment or test
Do problems that have 8.1 14.1 33.0 32.8 12.0
more than one correct
solution
Note: Numbers are percentages of respondents selecting that category, weighted to represent
statewide population.

CPRE Research Report Series, RR-39 8

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

and standard deviation .75. The scale’s able to return the next summer. Replacement
reliability is .63. units were curriculum modules designed to
be consistent with the reforms that center on
The second set of items that emerged from specific topics, like fractions, or sets of
our factor analysis was composed of topics. Unit authors devised these units to be
activities more closely keyed to practices coherent and comprehensive in their
that reformers wished to see in classrooms exploration of mathematical topics—to truly
(see Table 2). We averaged teachers’ replace an entire unit in mathematics texts,
responses to these seven items to make our rather than just add in activities to existing
“framework practice” scale. The scale has a curricula—and to support teacher as well as
mean 3.26, a standard deviation of .72, and a student learning. Teachers who attended
reliability of .85. these workshops worked through the units
themselves, and often had a chance to return
Opportunity to Learn to the workshops during the school year for
de-briefing and discussion about how the
Most teachers had much to learn if they were unit worked in their own classrooms.
to respond deeply to the new ideas about
mathematics teaching and learning. We Workshops like EQUALS, Family Math, and
report here on three very different sorts of cooperative learning (in section B of Table
opportunities to learn: study of specific math 3) had a different focus. Each was loosely
curriculum materials for students that were related to the Framework (for the curriculum
created to advance the reforms; study of frameworks had many goals) but none of the
certain special topics and issues related to three was focused directly on students’
reform; and more general participation in mathematical curriculum. EQUALS, for
learning opportunities, reform networks and instance, deals with gender, linguistic, class
activities. and racial inequalities in math classrooms.
Family Math helps teachers involve their
Table 3 contains evidence from our first students’ parents in math learning, and
inquiry into teachers’ opportunities to learn. cooperative learning workshops come in
A single question, reproduced in the table, many different flavors, such as de-tracking,
asked teachers to estimate how much time but all encourage learning together.10
they invested in mathematics-related
activities within the past year. The question Two-thirds of the teachers who responded to
refers to two somewhat different sorts of our survey participated in professional
workshops. Section A of the table focused development activities in at least one of the
on what we refer to as “student curriculum;” five curricula listed in Table 3. But the
these are workshops that dealt with new breadth of these professional development
mathematics curriculum for students. For opportunities was not matched by their
instance, Marilyn Burns Institutes are offered depth. Our chief indicator of depth was the
by experienced trainers that Burns selects amount of time that teachers reported
and teaches, and are focused on teaching spending in the activities. While we
specific math topics; some focus on recognize that more time is no guarantee of
replacement units that Ms. Burns has more substantial content, it creates the
developed. In some cases, teachers who
attended these workshops one summer were

CPRE Research Report Series, RR-39 9

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 3
Teachers’ Opportunities to Learn

Which of the following mathematics-related activities have you participated in during the past
year and approximately how much total time did you spend in each? (e.g., if four two-hour
meetings, circle 2—”1 day or less”). (CIRCLE ONE ON EACH LINE.)*
None One day 2-6 1-2 More
or less days weeks than 2
weeks
A. Student Curriculum
Marilyn Burns 83.2 9.8 5.3 1.3 .3
Mathematics Replacement 58.9 22.7 14.2 1.7 2.5
Units
B. Special Topics/Issues
EQUALS 96.5 2.4 .9 .2 0
Family Math 81.7 12.9 4.3 .8 .3

Cooperative Learning 54.5 28.9 13.7 1.8 1.1

Note: Numbers are percentages of respondents selecting that category, weighted to represent
statewide population.
* Missing data assumed to be “none.”

opportunity for substantial work that could One way to place these numbers in context
not occur in a few hours or a day. Table 3 would be to compare California’s teachers’
shows that most teachers spent only nominal learning opportunities to those available to
amounts of time in either sort of professional teachers in other parts of the nation.
development activity. By tabulating each Unfortunately, few studies contain similar
teachers’ total investment across the five descriptions of teachers’ professional
options above, we found roughly half of all development in the U.S., so precise
teachers who reported comparisons with previous work are
attending one of the workshops in the past impossible.11 But Table 4 accords with what
year indicated they spent one day or less. most observers report: the modal teacher’s
Roughly 35 percent reported spending opportunity for professional development
between two and six days. A smaller typically consists of a few days of learning
fraction of those who attended workshops— each year about a discrete topic (Little,
and a very small fraction of the sample as a 1993; Lord, 1994; O’Day and Smith, 1993;
whole—attended workshops for one week or Weiss 1994). A few teachers managed to
more.

CPRE Research Report Series, RR-39 10

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 4
Participation in Reform Networks and Leadership Roles*

Activities Percent that Percent that

did participate did not
participate
Attended a national mathematics teacher association 5.7 94.5
meeting
Attended a state or regional mathematics teacher 12.3 85.1
association meeting, including California Mathematics
Council affiliates
Taught an in-service workshop or course in 13.6 83.5
mathematics or mathematics teaching
Served on a district mathematics curriculum committee 13.7 84.6
Note: Numbers are percentages of respondents selecting that category, weighted to represent
statewide population.
* Teachers were asked to report only for the year prior to the survey.

connect themselves to relatively rich learning or served on local curriculum committees.

opportunities, but most encountered the Teacher contact with the reforms via these
reforms in conventional settings—in a day- leadership activities, in other words, was less
long or shorter introduction to a particular frequent than their contact through more
instructional technique or curriculum. conventional professional development
avenues.
Another way to put these numbers in context
is to ask how they related to teachers’ So far, we have reported on teachers’
more general opportunities to learn about learning opportunities within the year before
California’s Mathematics Framework. the survey. We also asked teachers to tell us
Besides encounters with student curriculum whether they had had career-long
or special topics/issues workshops, teachers opportunities to learn about the new
could have engaged in a variety of activities standards, although we did not here inquire
designed to familiarize them with reform, into the specifics of those experiences.
like participating in reform networks, According to our tabulations, 65% of
attending meetings of math teachers, serving teachers reported that they had at some time
on committees, and the like. Table 4 shows attended school or district workshops related
that few teachers did so: for example, fewer to the new mathematics standards, and 45%
than six in every hundred reported attending said they had been given time to attend off-
a national mathematics teacher association site workshops or conferences related to
meeting, and only twelve or thirteen in every those standards. Merged, somewhere near
hundred participated in other state or seven out of ten teachers did one of these
regional meetings, taught local workshops, two activities—many did both. But because

CPRE Research Report Series, RR-39 11

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

these are general measures only, we have no curriculum workshops” is a dummy variable
sense of the character of the learning marking attendance at the workshops which
opportunities—whether they were long or used students’ new curriculum to investigate
short, focused on specific problems or mathematics instruction. “Special
general principles, or whether the formats topics/issues workshops” marks attendance
were innovative or traditional. at workshops associated with special topics
or issues in mathematics reform. Roughly 45
One view of the evidence in Tables 3 and 4 is percent of teachers had at least some
that reformers in California wanted to opportunity to learn about student
leverage deep changes in mathematics curriculum in either the Marilyn Burns or
instruction with very modest investments. mathematics replacement units workshops,
Recent research suggests, however, that and around 50 percent of teachers spent
altering the core elements of teaching some time learning about EQUALS, Family
requires extended opportunities for teachers Math, or cooperative learning.
to learn, generous support from peers and
mentors, and opportunities to practice, Those variables permit us to probe the links
reflect, critique, and practice again (Ball and between the type of learning opportunities
Rundquist, 1993; Heaton and Lampert, teachers had and their self-reported practice.
1993; McCarthy and Peterson, 1993; Wilson, Because teachers’ time investments in these
Miller and Stokes, 1993; see also Schifter opportunities to learn varied, we could also
and Fosnot, 1993). Such opportunities were ask: what effects do difference in time spent
unlikely in the brief professional activities of learning about new curricula have on
most California teachers. teachers’ practice? To pursue this issue we
created two additional variables to mark the
Another view of the evidence is that some duration of the learning opportunity that
reformers took a novel departure: they teachers reported in the two types of
grounded some teachers’ professional workshops.12 These time measures are
development in the improved student correlated with their respective dummy
curriculum that state policy had helped to variables (r=.4 or more), but entering them
enable. Most professional development is not into our models predicting teacher practices
so grounded in student curriculum. It also is should tell us whether spending more time in
a happy event for the interested researcher, a certain kind of workshop is linked to
for comparing the two approaches in Table 4 different kinds of practice—an outcome one
enables us to ask a central question: Did would expect if teachers were indeed
teachers who attended the student learning.13
curriculum-centered workshops in Table 4
report different kinds of practice from those Finally, we created a more general variable
who attended the special topics/ issues known as “previous Framework learning.”
workshops? The variables in Table 3 capture teachers’
learning opportunities only in the year prior
We used the raw data reported in Tables 3 to the survey, so we tried to control for
and 4 to create several aggregate variables earlier learning opportunities in predicting
that represent the broader classes of learning “Framework” and “conventional” practice.
opportunities we identified earlier. “Student Not doing so could lead to a type of omitted

CPRE Research Report Series, RR-39 12

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

variable bias, for teachers who had some frameworks led teachers to certain
earlier learning about the content of the new workshops, or because those workshops
Framework would be lumped with teachers caused teachers to be more enthusiastic. We
who had none. Our simple measure of want to control for selectivity—the former
earlier learning showed that about 30 percent case—because leaving it out of the model
of teachers had not attended one of the might result in a workshop variable picking
student curriculum- or math-related up this selectivity and artificially inflating.
workshops in the past year but did report a Because “affect” could also pick up some
career-long learning opportunity.14 effects of workshops, thus understating any
relationship between opportunities to learn
Controls and practice, this may act as a conservative
control.
Causality is difficult to determine in a one-
time survey. It would not be surprising, for The second control is teachers’ familiarity
instance, if teachers who took advantage of with the themes of the state reform.
professional development that was centered Teachers who are more familiar with these
in students’ mathematics curriculum were broad policy objectives may have at least
different from teachers who spent their time learned to use the language of the
in brief workshops on peripheral matters. frameworks and know what is “in” and
Teachers of the first sort might be more “out.” We found, for example, that
committed to the reforms, or more “familiarity” is linked to teachers’ attitudes
knowledgeable about them already, or both. toward conventional math instruction;
Were that the case, our measures of teachers who know what classroom practices
teachers’ learning opportunities would are approved by the frameworks will much
include effects of such selectivity, and less often report approval of spending math
relationships with practice would be suspect. time in drill and skill.16 Familiarity was
measured by asking teachers to identify the
We tried to err on the side of caution by themes central and not central to the reforms
including two controls; while these do not from a list of statements about instruction
completely mitigate the possibility of and student learning. We include this in our
selection bias, they go some distance toward analysis of the relationship between
safeguarding against inflation of teacher opportunities to learn and classroom practice
learning effects. The first, “affect,” is since teachers who were more familiar with
teachers’ reports about their views of the the reform might report practices more
state mathematics reforms. Teachers consistent with the reforms, just because
answered this item on a scale of 1 to 5, with they know what is approved.17 Other
1 labeled “extremely negative” and 5 teachers whose classrooms were identical
“extremely positive.” The scale mean is 3.77 but who were less familiar with the reforms
and its standard deviation .93. We include it might have been less likely to report
in our models since teachers’ view of reform practices acceptable to reformers. The mean
is likely to be linked to the classroom of this measure is .83 on a scale of 0-1,
practices they report.15 Affect also might be which indicates considerable familiarity with
correlated with taking certain workshops, the leading reform ideas.
either because being enthusiastic about the

CPRE Research Report Series, RR-39 13

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Familiarity also may be a conservative check practices to a traditional core, but also
on our analysis: though some portion of changed that core.
teachers’ familiarity may pre-date the
workshops and thus signal selection, another In contrast, the variable for the special
portion may be an effect of workshops. By topics/issues workshops has nearly a zero
including this measure we may be reducing regression coefficient in both cases.
any possible associations between Workshops not closely tied to student
professional development and practice.18 curriculum seem unrelated either to the kinds
of practices reformers wish to see in schools
Impact of Opportunities to or to conventional traditional practices like
worksheets and computational tests. We
Learn on Practice suspect that this is because the special
topics/issues workshops, though consonant
We now turn to the results. We report first with the state math Framework in some
(see Table 5) on the impact that workshop respects, are not centered on the
curricula have on teachers’ reports of both mathematics teaching practices that are
Framework and conventional practices, then central to instruction, but focus instead on
turn to the combined impact of curriculum other things that may be relevant to
and time. instruction but are not chiefly about
mathematical content. Such workshops may
Curriculum Alone. The results of this OLS be useful for some purposes teaching—such
regression states a central finding quite as adding cooperative learning groups or
bluntly: the content of teachers’ professional new techniques for girls or students of
development makes a difference to their color—but would likely be peripheral to
practice. Workshops that offer teachers an mathematics, and to changing core beliefs
opportunity to learn about student math and practices about mathematics teaching.
curriculum are associated with teacher
reports of more reform-oriented practice. The coefficients on “previous math
The average teacher who attends a Marilyn Framework learning” shows a more modest
Burns or replacement unit workshop reports effect on Framework practice, and none for
more Framework practice (nearly three- conventional practice. That is as expected,
quarters of a standard deviation) than does for variable was constructed from a question
the average teacher who did not attend those that invited teachers to lump together
workshops. Moreover, the relationship different learning opportunities—those
works in both directions. Teachers who centered on student curriculum and others.
report attendance at either Marilyn Burns or
replacement unit workshops report fewer So when teachers’ opportunities to learn
conventional practices (about four-tenths of from instructional policy are focused directly
a standard deviation) than teachers who did on student curriculum that exemplifies the
not attend these student curriculum-centered policy, that learning is more likely to affect
workshops. These learning opportunities their practice. Capable math teachers must
seem not only to increase Framework know many things, but their knowledge of
practice but to decrease conventional
practice; teachers did not just add new

CPRE Research Report Series, RR-39 14

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 5
Associations Between Teachers’ Learning Opportunities and Practice

Curriculum Only Curriculum Plus Time

Equations Equations
Conventional Framework Conventional Framework
Practice Practice Practice Practice
Intercept 1.6* 1.78* 1.56* 1.83*
(se) (.19) (.18) (.19) (.17)
Student Curriculum -0.30* 0.54* -0.15** 0.36*
Workshop
(se) (.08) (.07) (.09) (.08)
Time in Student Curriculum -0.08* 0.09*
Workshop
(se) (.02) (.02)
Special Topics/Issues 0.02 0.01 -0.03 0.04
Workshop
(se) (.06) (.06) (.07) (.06)
Time in Special 0.05 -0.04
Topics/Issues Workshop
(se) (.03) (.03)
Previous Framework 0.02 0.20* 0.02 0.21*
Learning
(se) (.08) (.07) (.08) (.07)
Affect -0.21* 0.22* -0.21* 0.22*
(.03) (.03) (.03) (.03)
Familiarity -0.85* 0.42* -0.79* 0.36**
(se) (.21) (.20) (.21) (.20)
R2 (adjusted) 0.17 0.22 0.19 0.25
Note: Estimation by OLS.
* Indicates significance at P<.05 level
** Indicates significance at P<.10 level

CPRE Research Report Series, RR-39 15

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

mathematics, and how it is taught and practice and less conventional practice they
learned, are central. This explanation points reported. The effect persists even when
to the unusual coherence between the controlling for such markers of possible
curricula of students’ work and teachers’ selectivity as teachers’ familiarity with and
learning that the Marilyn Burns/replacement views of reform. The result parallels
unit professional development created. research on students’ opportunities to learn,
Teachers in these workshops were learning where researchers have found the
about the mathematics that their students combination of time and content focus to be
would study and about teaching and learning a potent influence on learning.
it.
Time expenditures in the special topics/issues
Such learning differs quite sharply from most workshops did not have the same payoff in
professional development, which seems to be practice. Instead, the coefficients and
either generic (“classroom significance levels drift toward a contrary
management,” for example), or peripheral to effect—that is, teachers who spent more
subject matter (such as “using math time in such special topics/issues workshops
manipulatives”). Generic and peripheral report practices that are a bit more
professional development do not have deep conventional than their peers, although the
connections to central topics in school difference is not statistically significant. This
subjects (Little, 1993; Lord, 1994). There is a very important point: even large
was a modest move in the 1980s away from investments of time in less content focused
generic pedagogy workshops, toward workshops are not associated with more of
subject-specific workshops like cooperative the practices that reformers advocate, nor
learning for math, that several observers with fewer of the conventional practices.
considered an improvement (see Little, 1989, Again, the effects of these workshops seem
1993; McLaughlin, 1991). But our results tangential to the central classroom issues
suggest that teachers’ learning opportunities measured by our practice scales and on
may have to go one level deeper than just which the mathematics reform focused.
subject specificity. Providing teachers with
more concrete, topic-specific learning This effect of time bears on our concerns
opportunities— fractions, measurement, or about selectivity. A critic might argue that
geometry— seems to help to change the results of the curriculum-only regressions
mathematics teaching practices. This (columns one and two in Table 5) could be
conjecture is consistent with recent research explained by teachers having selected
in cognitive psychology which holds that workshops that mirror their teaching styles
learning is domain-specific. and interests. But it seems extremely
unlikely that teachers would arrange
Curriculum and Time. We found clear themselves neatly by level of enthusiasm and
effects of time. They are reported in the practice into different levels of time
curriculum-and-time models, the next set of investment as well. Thus, when we see that
equations in columns three and four in Table adding hours or days in a student curriculum
5. The more time that teachers spent in workshop means scoring progressively
Marilyn Burns/Replacement Unit learning higher on our Framework practice scale and
situations, the more Framework-related lower on conventional practice, especially

CPRE Research Report Series, RR-39 16

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

when controlling for teachers’ familiarity factors—which may be correlated with

with and views of reform, we surmise that teacher practice—that may have led teachers
learning, not fiendishly clever self-selection, to select themselves into certain
was the cause.19 workshops.20 The results show that decisions
to enroll appear to be only modestly related
What does all this mean for the average to teachers’ pre-existing dispositions toward
teacher in California? As we have said, certain types of mathematics teaching. In so
nearly half of the teachers in the survey far as we can tell from these data, teacher
reported attending a Marilyn Burns or selection into workshops does not appear to
replacement unit workshop within the year be rational, in the sense that teachers
before the survey. This is impressive breadth carefully seek out workshops that fit with
in the coverage of reform in the state, and strongly held convictions about reform. That
suggests that many teachers had at least a further suggests our findings are robust, an
chance to rethink some of the practices impression that is strengthened by Little’s
central to mathematics instruction. But (1989, 1993) account of the professional
breadth is not the same as depth, and in this development “system.” She describes
vein we note again that many teachers’ teachers’ workshop choices as usually
opportunities to learn were quite shallow. A related to very general subject-matter
re-inspection of Table 3 shows that only a interest like “math” or “technology” but only
very modest slice—five percent or less—of weakly related to things like specific
the population of California elementary workshop content, quality, or potential
school teachers reported spending one week effects for students’ learning. Lord (1994)
or more in either of the student curriculum goes one step further, arguing that teachers’
workshops during 1993-94. staff development choices are “random” with
regard to the factors reformers might care
This first picture of the impact that about. The sort of selection that concerns us
professional learning can have on teacher does not seem to be characteristic of
practice is grainy, for surveys of this sort are professional development.
relatively crude instruments. But the
associations are substantively significant and The Mediating Role of Tests
fairly consistent in size across different
model specifications. They support the idea Tests are widely believed to be a significant
that the kind of learning opportunities influence on teaching, and the California
teachers have matters to their practice, as Learning Assessment System (CLAS) was
does the time that they spend learning. designed partly for this purpose. California
reformers and educators advocated
Because of our concerns about causality we assessments that would focus on the new
subjected the findings to some fairly rigorous conceptions of mathematics and
tests for selection, such as using fairly strict mathematical performance advanced by the
control variables like “affect” and “familiar,” state’s Mathematics Framework. California
to mitigate against selection effects in our revised its testing program between the late
models. But since these are far from perfect, 1980s and early 1990s; the new system
we also performed a two-stage least squares comprised a set of statewide assessments
regression to control for those that were administered to all students in the

CPRE Research Report Series, RR-39 17

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

fourth, eighth, and tenth grades in 1993 and intellectual bent of the CLAS report more
1994. The tests were revised so they would Framework practice and less conventional
help reform instruction across the state either practice than teachers who did not? If so,
by aligning the messages sent by the state how did the tests affect practice? If some of
about curriculum, instruction, and the reformers were correct, the test should
assessment, by providing an incentive for have provided an incentive for fourth-grade
teachers or schools to investigate the new math teachers, or an opportunity for them to
curriculum, by proffering educators another learn more about the new mathematics
means by which to become familiar with teaching, or both. That question is especially
reform ideas, or by some combination of salient because there is disagreement about
these. the means by which tests influence
practice—is it learning or incentives?
Efforts of this sort raise several issues for Finally, do the effects of tests on teachers’
anyone concerned about California’s practice wash out the effects
reforms. One is straightforward: Did the that teachers’ learning opportunities have on
tests affect practice? Did teachers who knew practice? That could occur if teachers who
about, administered, or shared the took the CLAS seriously had attended the
student curriculum workshops, but had done
so, and changed their practice, because of
the test rather than the workshops.

Table 6
Learning about the California Learning Assessment System (CLAS)
vs. Administering CLAS

Learned About CLAS

Yes No Total
Administered CLAS
Yes 312 93 405
(53%) (16%) (68%)
No 58 131 190
(10%) (22%) (32%)
Total 371 224 595
(62%) (38%) (100%)

CPRE Research Report Series, RR-39 18

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

To investigate these issues we That modest effect of test administration

operationalized two variables: whether might disappoint supporters of assessment-
teachers “learned about CLAS,”21 and based reform, because it suggests that the
whether teachers administered CLAS. incentives associated with testing alone are
About one-third of the teachers reported not great. But the CLAS lasted for only two
they had learned about the CLAS, and years and published results only at the school
another third reported that they had level, which may not have been sufficient for
administered it. Not every teacher who incentive effects to develop.24 There also
learned about the mathematics CLAS said seems to be little solace in these results for
they also administered the test, and vice advocates of a contrary view: that any effect
versa. Table 6 shows that there is an of assessment-based reform will occur only
association between these two through teachers’ learning opportunities.
variables—teachers who administered the The other new variable in this
CLAS were more likely to have had an model—whether teachers reported learning
opportunity to learn about it. The off- about the CLAS—fared even worse: it was
diagonal cases, however, show that unrelated to teachers’ descriptions of their
there is enough variance to enable us to sort classroom practice in mathematics.
out the effects of learning about the test from
the effects of actually administering CLAS. One might conclude both that the incentive
that CLAS presented to teachers who
Set I of Table 7 contains the results of that administered it caused mild change in their
effort. As one would expect, there is a math instruction, and that the test prompted
statistically significant and positive little independent learning about new
relationship between administering CLAS mathematics practices. That alone would be
and reporting more Framework practice. But humble yet hopeful news for assessment-
the relationship is quite modest; it does not based reform: because teachers certainly did
come at all close to the size of the not select themselves into administering the
association between curriculum workshop test, the effect associated with test
learning and practice.22 In addition, this administration should be a true estimate of
CLAS-practice association does not decrease practitioners’ response to policy. But there
teachers’ reports of conventional practices is more to the story. To further probe
like bookwork and computational tests. It teachers’ views of the assessment we
seems that any incentive associated with the generated cross tabs that described the
administration of CLAS only adds new relationship between administering the CLAS
practices to existing conventional practice. and various measures of agreement with the
Rather than redecorating the whole house, test. Table 8 shows that there is a strong
teachers supplemented an existing motif with relationship among administering the CLAS,
more stuff—a result that also was clear in teachers’ view of the test, and adopting
our field work.23 By way of contrast, the classroom practices that it might seem to
teachers who spent extended time in student imply. But the table also shows that not all
curriculum workshops reported both less teachers who reported administering the
conventional practice and more Framework- CLAS either agreed with the test’s
oriented practice. orientation or tried to fit their teaching to it.
This implies that teachers were quite

CPRE Research Report Series, RR-39 19

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 7
Association Between Teachers’ Learning Opportunities,
Teachers’ Practice, and CLAS Measures

Set I Set II
Conventional Framework Conventional Framework
Practice Practice Practice Practice
Intercept 1.58* 1.82* 1.62* 1.62*
(.19) (.17) (.20) (.16)
Student Curriculum -0.16* 0.37* -0.14* 0.37*
Workshop
(.08) (.08) (.09) (.07)
Time in Student Curriculum -0.07* 0.08* -0.06* 0.07*
Workshop
(.02) (.02) (.02) (.02)
Previous Framework 0.02 0.21* 0.06 0.23*
Learning
(.08) (.07) (.08) (.07)
Affect -0.21* 0.22* -0.17* 0.11*
(.03) (.03) (.04) (.03)
Familiarity -0.84* 0.34** -0.61* 0.35**
(.21) (.19) (.23) (.19)
Learned about CLAS 0.06 0.002 0.11** -0.01
(.06) (.06) (.07) (.06)
Administered CLAS -0.004 0.14* 0.06 -0.02
(.07) (.06) (.07) (.06)
CLAS useful -0.14* 0.21*
(.04) (.03)
R2 (adjusted) 0.18 0.25 0.21 0.34
* Indicates significance at P<.05 level
** Indicates significance at P<.10 level

CPRE Research Report Series, RR-39 20

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 8
Attitude Toward the California Learning Assessment System (CLAS) by
Test Administration

Administered Did Not

CLAS Administer
The mathematics CLAS corresponds well with the
mathematics understanding that I want my student to
demonstrate.
Agree 57% 50%
Neutral 32% 39%
Disagree 12% 11%
Total 101% 100%
I currently use performance assessments like CLAS in
my classrooms.
Agree 48% 21%
Neutral 34% 32%
Disagree 18% 46%
Total 100% 99%
Math CLAS has prompted me to change some of my
teaching practices.
Agree 71% 40%
Neutral 15% 30%
Disagree 14% 30%
Total 100% 100%

Learning new forms of assessment has been valuable

for my teaching.
Agree 64% 36%
Neutral 22% 32%
Disagree 13% 32%
Total 99% 100%
Note: Totals do not always equal 100% due to rounding.

CPRE Research Report Series, RR-39 21

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

selective in attending to the new test. Many 3. Math CLAS has prompted me to
who administered the CLAS liked it and change some of my teaching
used it as a learning opportunity, but others practices.
did not. The same can be said for those who
did not administer the test: even without the 4. Learning new forms of assessment
direct incentive supplied by the test’s has been valuable for my teaching.
presence in their classroom, some found it
instructive in changing their mathematics The scale thus links several elements of the
teaching, while others paid it little heed.25 role that an assessment might play: (1)
teachers’ sense of the congruence between
This throws a bit more light on how the CLAS and their work; (2) their use of
statewide testing may influence teaching and and thus familiarity with such assessments;
curriculum, at least in states that resemble (3) their sense of whether the test had
California. Instead of compelling teachers to changed their teaching, which could occur
teach the mathematics to be tested, the through learning or an incentive, or both; and
CLAS seems to have provided teachers with (4) their view of whether they had learned
occasions to think about, observe, and revise from CLAS-like assessments and whether
mathematics instruction. Some teachers the learning was pedagogically useful.
seized on the occasion while others ignored
it. Administering or learning about the test We then re-ran the equations that probed the
increased the probability that a given teacher effects of testing on practice in Table 7, with
would attend to the test and thus to the state this new variable included. Doing so
reform, but did not guarantee that result. rendered the two test-related variables that
Many California teachers seem to have felt we initially discussed quite insignificant (see
quite free to reject the test and its Table 7, set II). Moreover, teachers who
concomitant view of mathematics— score relatively high on this scale report
probably without penalty and possibly with more reform-oriented practices but fewer
support from principals, school boards, and conventional practices, which indicates a
parents. more thorough revision of practice, and
perhaps greater internal consistency in
To pursue this more teacher-dependent teachers’ work than if teachers had reported
representation of teachers’ relationship with more Framework practice but no less
the test, we made the four survey items in conventional practice. This supports a view
Table 8 into a scale, called “CLAS useful.”26 that it is neither learning alone nor incentives
The items were: alone that make a difference to teachers’
practice, but a combination of experience,
1. The mathematics CLAS corresponds knowledge, beliefs and incentives that seem
well with the mathematics to condition teachers’ responses to the test.
understanding I want my students to The effects of assessment on practice appear
demonstrate. among those teachers who constituted
themselves as learners about and
2. I currently use performance sympathizers with the test—and this group
assessments like CLAS in my itself seems constituted both of teachers
classroom. whose approaches already concurred with

CPRE Research Report Series, RR-39 22

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

the test and those for whom the test spurred associate with standards and testing—i.e.
new thought and learning about mathematics one tied to external rewards or punishments.
Though reformers laid great stress on the
This complex interrelationship between role of CLAS in promoting change, its
learning and incentives is also evident in the external accountability element was relatively
observations of California elementary weak: school scores were published, but no
teachers themselves. One teacher, further official action was required or even
interviewed by Rebecca Perry in a study advised. The incentives connected with this
related to ours, reported that: test instead seemed internally constructed by
individual teachers.
“...the CLAS test....It was a shock to me. Another major reason the new assessment
They [students] really did fall apart. It system worked as it did is that it provided
was like, ‘Oh! What do I do?’ And I opportunities for teachers to learn. To start,
realized, I need to look at mathematics the California Department of Education
differently. You know, I really was doing involved a small number of teachers in the
it the way I had been taught so many development and pilot testing of the CLAS.
years before. I mean, it was so dated. The state department then paid many more
And I began last year, because of the teachers—several hundred—to grade student
CLAS test the year before, looking to see responses to open-ended tasks on the 1993
what other kinds of things were and 1994 assessments. These teachers then
available.” (Perry, 1996, p. 87) returned to their districts and taught others
about performance assessment in general,
This suggests that the teacher’s learning and about the CLAS in particular. Other
(“...looking to see what other kinds of things opportunities to learn about the test were
were available”), and her efforts to change made available through the California
her practice, were associated with the Mathematics Council and its regional
incentive for change that was created when affiliates, various branches of the California
she noticed that her students “...really did fall Math Projects, and through assessment
apart” when trying to take the new test. Her collaboratives in the state. Finally, the state
students’ weak performance as test-takers published in 1991 and 1993 “Samplers of
stimulated her to find ways to help them do Assessment” to help familiarize teachers with
better before she saw any scores. the novel problems and formats of the new
test.
Thus, California’s brand of assessment-
driven instructional reform did not When teachers came into contact with the
automatically ensure change in practice. new assessment, they had opportunities to
Many teachers who came in contact with it examine student work closely, to think about
through test administration or professional children’s mathematical thinking, and to
development were spurred to reevaluate their learn about the activities and understandings
math instruction; others were not. The test associated with the state’s reform. Such
was a resource or incentive only to those work would have offered participants
who perceived it as such. One reason may elements of a “curriculum” of improved math
have been that the incentive embedded in the teaching. Simply administering the CLAS
test was not what many policymakers also may have served as a curriculum for

CPRE Research Report Series, RR-39 23

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

many teachers, for it provided those • grounded in the curriculum that

unfamiliar with the frameworks a chance to students study;
observe how children react to challenging
math problems, and novel exercises and • connected to several elements of
activities. In either event, the closer a instruction (for example, not only
teachers’ contact with the test—via its curriculum but also assessment); and
administration or by learning about it—the
more likely s/he was to have had both • extended in time.
internal incentives to change and
opportunities to learn. Such opportunities are quite unusual in
American education, for professional
Our third question about testing was whether development rarely has been grounded either
the effects of CLAS on teachers’ practice in the academic content of schooling or in
washed out the effects of their workshop knowledge of students’ performance. That
learning on practice. Table 7 shows that it is probably why so few studies of
does not. When we ran models with only professional development report connections
“administered CLAS” and “learned about with teachers’ practice, and why so many
CLAS” (Table 7, Set I), the coefficients on studies of instructional policy report weak
the curriculum workshop variables declined implementation: teachers’ work as learners
very slightly. When we entered “CLAS was not tied to the academic content of their
useful” (Table 7, Set II), the student work with students.
curriculum by time coefficient declined a bit,
suggesting modest overlap between teachers’ Effects on Student
learning about CLAS and learning from
curriculum. But it was a small overlap: the Achievement
coefficients on “student curriculum
workshops” remains quite near its former Reformers took several steps intended to
size, and statistically significant.27 Teachers’ improve mathematical instruction and
learning through student curriculum student learning: they made available new
workshops and their learning via CLAS were and better student curriculum units; they
more independent than overlapping paths to encouraged professional development
framework-oriented practice. around these units and reform ideas more
generally; and they used the state assessment
These remarkable effects tend to support our program both as an example of and as
conjecture that teachers’ opportunities to incentive toward change. Many reformers
learn can be a crucial link between reasoned that teachers would respond to
instructional policy and classroom practice. these initiatives by learning new things about
Many educators believe that such links exist, math and implementing a new kind of
but research generally has not supported that practice in their classrooms, and that
belief. Our results suggest that one may students would learn more or better as a
expect such links when teachers’ result. We have organized their reasoning in
opportunities to learn are: more formal terms as a conjecture or model
of how policy might affect student
performance: teachers who had substantial
learning opportunities, who adopted the

CPRE Research Report Series, RR-39 24

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

curriculum or learned about the assessments learning opportunities. But the survey
designed to promote change, and whose sampled only four or fewer teachers per
math teaching was more consistent with the school, so the averages provided only a
state reforms would have students with crude estimate of our independent measures.
higher math scores on assessments that were These measures of school engagement with
consistent with the aims of state instructional reform are therefore error-filled, that is, most
reforms. likely to bias the investigation against finding
significant results, because random noise in
To explore this reasoning we merged student equations is known to diminish the effects on
scores on the 1994 fourth grade mathematics affected variables. Working with school
CLAS onto the school files in our data set. averages also reduced the size of the sample
The CLAS included a good deal of (n=162), for we deleted school files in which
performance based assessment. To do well, only one teacher responded or lacked CLAS
students would have had to answer scores.30
adequately a combination of open-ended and
multiple-choice items designed to tap their We created three additional variables for
understanding of mathematical problems and each school in the reduced sample. One
procedures. State scorers assigned students a variable is the 1991 state report of the
score from Level-1 to Level-6 based on their percent of students in each school who
proficiency level, and school scores were qualified for free lunch (%FLE), so we can
reported as “percent of students scoring allow for the influence of students’ social
Level-1,” and so on. We created an average class on test scores. The next variable is the
of these for each school to represent our school average of teachers’ estimates of the
CLAS dependent variable, with the higher school environment, called “school
school scores representing a more proficient conditions.” This consists of a five-point
student body. The mean of CLAS in our scale that includes teacher reports on
sample of schools was 2.76, and the standard parental support, student turnover, and the
deviation at the school level .57.28 Because condition of facilities, with five indicating
assessment officials corrected problems from better conditions.31 Finally, we took
the previous year, the 1994 assessment was teachers’ reports of the number of
technically improved—all student booklets replacement units they used and averaged
were scored, and measurement problems them by school; the mean for this measure is
reduced. Moreover, it was administered in .61, its standard deviation .59. In addition to
the spring of 1994, roughly six months these three, we continued to use the
before this survey, so our estimates of variables that mark other potential
teachers’ learning opportunities and practice connections between policy and practice,
corresponded in time to the assessment.29 including time in student curriculum
workshops32, our control for teachers’
Despite that good timing, we faced several previous Framework learning experiences33,
difficulties. Because the California teachers’ reports of Framework practice, and
Department of Education reported only the CLAS-OTL measure, all averaged for
school-level scores, we had to compute schools. Table 9 shows the school averages
school averages of all independent variables, for all these measures.
including teachers’ reports of practice and

CPRE Research Report Series, RR-39 25

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

The central issue in this analysis is whether the policy and practice markers are
the evidence supports our model of relations correlated at the .14 to .29 level with the
between policy and performance, but this school average CLAS scores we think they
question is difficult to handle empirically. might explain.
Reformers and researchers argue that the
more actual overlap among policy With this knowledge, we built an analysis
instruments, the more likely teachers, strategy: we started with a base equation
students, and parents are to get the same including the demographic measures, and
messages and respond in ways that are tested the primary conjecture of this
consistent with policy. The more highly section—that changes in teacher practice will
correlated are any possible measures of those lead to improvements in student
policy instruments, however, the greater the performance. But because our practice
problems of multicolinearity. Thus the more scale is an imperfect measure, tapping only
successful agencies are at aligning the one subset of the ways instruction might
instruments of a given policy, the more improve, we also tested the separate effects
headaches analysts will have in discerning the of each of the policy variables—teacher
extent to which they operate jointly or learning about CLAS, use of replacement
separately. units, and learning about that student
curriculum—on student achievement in
Table 10 displays some reasons for such successive equations. These models will
headaches, for it reveals that the correlations provide some overall impressions about the
among the independent variables of interest effect of policy on student performance
in our analysis range from mild to because each of the variables roughly
moderately strong. At the stronger end of summarizes a type of intervention that
this continuum, school average incidence of policymakers or others can organize. Yet
using replacement units is correlated at .44 the coefficient estimates in these first four
with the school average teacher report of models will be compromised by the high
participation in the student curriculum correlations among the policy variables as
workshops within the past year34, and at .47 evidenced in Table 10.
with school average reports of Framework
practice. This makes sense, since student Hence we devised a second strategy: put all
curriculum workshops should provide three policy variables in the base equation at
teachers with replacement unit materials and once, to see if it is possible to sort out the
know-how, and encourage them to change independent effects on student achievement
their practices. At the weak end of the of new student curriculum, teacher learning,
continuum, school average reports of and learning about the test. If this second
teachers’ learning about CLAS is correlated method enables us to distinguish the relative
at only the .13 to .15 level with schools’ use importance of policy variables, it would offer
of replacement units, teachers’ reports of evidence about which paths to reform
Framework practice, and their average
participation in the student curriculum
workshops. Special topics/issues workshops
and conventional practices also evidenced
low correlations with other variables. Finally,

CPRE Research Report Series, RR-39 26

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 9
Basic Data Statistics for Analysis of Achievement and Policy

Variable N Mean Std Dev Minimum Maximum

CLAS-OTL 162 0.3843101 0.3089501 0 1.0000000
FRAMEWORK PRACTICE 162 3.3068741 0.4746628 1.5714286 4.3571429
CONVENTIONAL 162 - 0.5631793 - 1.0027002
PRACTICE 0.0494945 2.3706506
STUDENT CURRICULUM 162 1.0123898 1.1543056 0 5.2500000
TIME
SPECIAL TOPICS/ 162 0.5337735 0.6670798 0 3.3333333
ISSUES TIME
REPLACEMENT UNIT 162 0.6103528 0.5922475 0 2.5000000
USE

Table 10
Intercorrelations Among Measures of Policy Instruments and
Math Performance (School Level)

Student Special Repl. Frame- Convent CLAS- CLAS

Curric. Topics/ Units work . OTL
Issues Practice Practice

Student 1.0
Curriculum
Special Topics/ .29 1.0
Issues
Replacement .446 .04 1.0
Units
Framework .386 .13 .479 1.0
Practice
Conventional -.39 -.06 -.33 -.39 1.0
Practice
CLAS-OTL .132 .02 .157 .148 .02 1.0
CLAS .252 .00 .264 .293 -.06 .142 1.0

CPRE Research Report Series, RR-39 27

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

might be most effective. Finally, we also The significant coefficient on “Framework

want to know whether these policy activities practice” also helps to answer one possible
were independently influential in improving criticism of our earlier analysis, namely that
student performance, or whether they the relationship between workshop
operate through teachers’ practice. So our attendance and Framework practice results
third analysis strategy is to add back our from teachers learning to talk the talk of
practice variable to this fuller model. We reform rather than making substantial
include the demographic measures in all changes in their classrooms. A critic might
equations to control the influence of social argue that the relationship was an artifact of
and economic status on student performance. teachers’ rephrasing their descriptions of
classroom work to be more consistent with
We start with teachers’ practice alone, the reform lingo; in that critic’s scenario,
because we have already shown that practice only the talk would be different, and
at least in part results from some of the classroom practice would be the same. But
learning opportunities provided by if teachers learned only new talk, it is
reformers, and because it provides the most difficult to imagine how schools with
logical link between policymakers’ efforts to teachers who report more Framework-
affect what happens in the classroom and related instruction should post higher scores
how students score on tests. Equation 1 in on the CLAS. Thus the association between
Table 11 shows a modest relationship: Framework practice and student scores
schools in which teachers report classroom seems to ratify the link between teacher and
practice that is more oriented to the math student learning, and to imply that teachers
Framework have higher average student were doing roughly what they reported. It
scores on the fourth grade 1994 CLAS, also seems to indirectly confirm our earlier
controlling for the demographic finding that teachers who had substantial
characteristics of schools. No such opportunities to learn did substantially
relationship, however, was found between change their practice.
schools high on our conventional practice
scale and student achievement scores. This Our second model concerns the effect of
provides evidence that teachers’ practice teachers’ learning on student achievement.
links the goals and results of state policy: Given the analysis just above, we would
students benefitted from having teachers expect a modest relationship between teacher
whose work was more closely tied to state attendance at student curriculum workshops
instructional goals. Though this and CLAS scores (absent other things) for
interpretation is based on aggregate data, it we have seen teachers who attend these
is difficult to think of any other reasonable workshops do more Framework practice.
inference than that teachers’ learning That relationship does occur when
opportunities can pay off for their students’ controlling for teachers’ previous
performance if the conditions summarized in Framework learning as is evident from
our model—grounded in student curriculum, Equation 2 in Table 11.
connected to several elements of instruction,
and extended in time—are satisfied. A more important query, perhaps, is the
effect of teacher learning in the special
topics/issues workshops on student

CPRE Research Report Series, RR-39 28

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Table 11
Associations Between Teachers’ Practice, Their Learning Opportunities,
and Student Math Scores

Equation-1 Equation-2 Equation-3 Equation-4 Equation-5 Equation-6

CLAS CLAS CLAS CLAS CLAS CLAS

Intercept 2.14* 2.65* 2.69* 2.66* 2.57* 2.27*

0.32 0.22 0.21 0.21 0.21 0.31
Percent FLE -1.17* -1.23* -1.22* -1.24* -1.21* -1.18*
0.13 0.13 0.12 0.12 0.12 0.12
School 0.19* 0.17* 0.17* 0.18* 0.17* 0.17*
Conditions*
0.05 0.05 0.05 0.05 0.05 0.05
Framework 0.17* 0.087
Practice
0.07 0.67
Conventional -.00
Practice
0.05
Student 0.065* 0.041** 0.034
Curriculum-Time
0.028 0.028 0.028
Special Topics/ 0.03
Issues-Time
0.04
Replacement 0.14* 0.11* 0.09*
Units-Average
Number Used 0.05 0.05 0.06

Learned About 0.21* 0.15** 0.147**

CLAS
0.09 0.09 0.09
Previous 0.11 0.14 0.14
Framework
Learning 0.1 0.1 0.1

R2 (Adjusted) 0.60 0.60 0.61 0.60 0.62 0.62

Note: All survey-based measures are averages from the teachers within a school who responded to the survey.
* Indicates significance at P<.05 level
** Indicates significance at P<.15 level

CPRE Research Report Series, RR-39 29

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

achievement. We saw earlier that this piloting, scoring, and so forth. We saw
variable contributed little to explaining earlier that this kind of learning affected
differences among teachers in Framework or teachers’ practices under certain conditions,
conventional mathematics practice. Hence and that learning may then translate into
any effect we might find on student changed practice and improved student
achievement would be through pathways not achievement. But it also is possible that
detected by these scales, such as increasing teachers prepared their students by
teacher knowledge, improving equity within administering CLAS-like assessments, used
classrooms, or helping teachers better performance-based assessments year-round,
understand student learning. But we found or learned something more about
no such effect of special topics/issues mathematics while learning about the CLAS.
workshops on student achievement. This is
a very important result: whatever
improvements these workshops may bring to In principle, then, both our practice and
California’s classrooms, they do not affect policy measures relate positively to student
what many see as the bottom line of achievement. This suggests that state efforts
schooling—student performance. to improve instruction can affect not only
teaching but also student learning. The
The third component of the policy mix, the relatively close relations among these
use of replacement units, also shows a markers call the point estimates in these
positive relationship to student achievement. models into question, however, since
Equation 3 in Table 11 indicates schools in omitting any one variable will allow another
which teachers reported they each used one variable to pick up its effects via their
replacement unit have student test scores correlation. So we ask next about the true
which average about one-quarter of a influence of each policy instrument on
standard deviation higher than schools in student achievement, controlling for the
which no teachers reported replacement unit effects of others: do the three instruments of
use. policy exert their influence jointly, each
having some independent effect on
Finally, we come to the effect on performance, or does one dominate? This is
achievement associated with teacher learning an important theoretical and practical
about the CLAS.35 The coefficient on CLAS- question, for if one instrument were
OTL (Equation 4 in Table 11) suggests a overwhelmingly influential we would draw
clear effect: when comparing student different inferences for action than if several
achievement scores, schools where all instruments were jointly influential. To this
teachers learned about the CLAS had student end, we entered the CLAS-OTL, student
test scores that were roughly one-quarter of curriculum workshop, and replacement unit
a standard deviation higher than schools markers into the CLAS regression along with
where no teachers learned about CLAS. It is the important control variable “previous
easier to report this result than to decide Framework Learning,” hoping there was
what it means. The CLAS-OTL measure enough statistical power to sort among them.
consists of the question whether teachers had
an opportunity to learn about the new test Equation 5 in Table 11 offers a version of
through professional development, test the joint influence story. Schools in which

CPRE Research Report Series, RR-39 30

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

teachers reported using an average of one that variable gathering an effect and zeroing
replacement unit appeared about one-fifth of out the three policy measures.
a standard deviation higher in the distribution
of CLAS scores than schools where no This does not occur in Equation 6 in Table
replacement units were used. Though 11. With the exception of “learned about
modest, this effect is statistically significant. CLAS,” which remains significant, our policy
Teacher learning in student curriculum and practice measures drop below strict
workshops added less power to student levels of significance while remaining
learning than did replacement unit use—but positively related to student achievement.
the effect is still discernible from none at all Most notably, the coefficient on our measure
by loose statistical standards.36 And schools of Framework practice is cut nearly in half,
in which teachers had opportunities to learn indicating it shares variance with markers
about the CLAS itself continued to post like student curriculum workshops and
scores about one-quarter of a standard replacement unit use. Even with this
deviation higher than schools in which evidence, however, we do not imagine we
teachers did not. All interventions organized have discovered a hitherto unnoticed magical
by reformers were associated with higher effect of teacher knowledge or curriculum
student scores on the CLAS. use on student achievement. Instead, we are
inclined to stick to our learning-practice-
One reason all three major policy variables learning story. One reason is that the three
might appear significant in this equation is variables which split variance are the most
that, to some degree, all might contribute to colinear, suggesting that the regression
or correlate with Framework practice. If algorithm will have difficulty sorting among
instructional policy is to improve student their effects, and that we might do better to
achievement, it must do so directly through conceive of the three as a package, rather
changes in teacher practice, for students will than as independent units. A joint F-test
not learn more simply because teachers know finds these three variables together a
different things about mathematics or have significant influence on student performance.
been exposed to new curricula or tests.
Instructional interventions like those studied A second reason is that our practice scale is
here must change what teachers do in the imperfect. Recall the types of items that
classroom—including what they do with comprise this measure: students do problems
curricula and tests—even if very subtly, in that have more than one correct solution;
order to affect student understanding. students make conjectures; students work in
Teachers who used new curricula but small groups. While this represents one
understood nothing about how to use them aspect of the ways teachers’ practices may
would not be likely to have students who change as a result of reformers’ efforts, it
learned significantly more from those fails to represent others, such as the changes
curricula. Following this reasoning—and in practice which might occur when teachers’
assuming that we had measured Framework understanding of mathematics deepens, when
practice perfectly—we would expect that teachers understand student learning
adding that measure of Framework practice differently, when teachers reconceive
to Equation 5 in Table 11 would result in assessment, or when teachers’ pedagogical
content knowledge increases. It is hard to

CPRE Research Report Series, RR-39 31

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

imagine these interventions not teaching practice, and thus at least indirectly an
teachers some of these things, yet these influence on student achievement through
dimensions of instruction are omitted from teachers’ practice.
the Framework practice scale. If, as we
expect, they do affect student achievement, The results that we have reported seem to
they would be picked up by the policy bear out the usefulness of such a model. We
variables in model 6. Equation 6 in Table 11 were able to operationalize measures of each
thus teaches us as much about the limits of important element, and the analysis seems to
survey research in instructional policy as it show that an instructional view of
does about the pathways to improved instructional policy can work. Teachers’
student achievement. opportunities to learn about reform do affect
their knowledge, and when those
Conclusion opportunities are situated in curriculum that
is designed to be consistent with the reforms,
We began this paper by sketching an and which their students study, teachers
instructional view of instructional policy. report practice that is significantly closer to
We argued that educational policies the aims of the policy. In such cases there is
increasingly seek to improve student a consistent relationship among the
achievement by manipulating elements of professional curriculum of reform, the
instruction—including assessment, purposes of policy, assessment and teachers’
curriculum, and teachers’ knowledge and knowledge of assessment, and the student
practice. To implement such policies, we curriculum. Finally, when the assessment of
wrote, requires the deployment of a range of students’ performance is consistent with the
instruments that are specific to instructional student and teacher curriculum, teachers’
policy, including student curriculum, learning opportunities pay off for students’
assessments, and teachers’ opportunities to math performance. These results confirm the
learn. Because the effects of these analytic usefulness of an instructional model
instruments would depend in considerable of instructional policy, and suggest the
part on professionals’ learning, teachers’ potent role that the education of
knowledge and practice and their professionals can play in efforts to improve
opportunities to learn would be key policy public education.
instruments.
It has been relatively unusual for researchers
We proposed a rudimentary model of this to investigate the relations between teachers’
sort, in which students’ achievement was the and students’ learning, but when they have
ultimate dependent measure of instructional done so it has been even more unusual to
policy, and in which teachers’ practice was find evidence that teachers’ learning
both an intermediate dependent measure of influenced students’ learning. A few recent
policy enactment and a direct influence on studies, however, are consistent with our
students’ performance. Teachers figure in results. Wiley and Yoon (1995) investigated
the model as a key connection between the impact of teachers’ learning opportunities
policy and practice, and teachers’ on student performance on the 1993 CLAS,
opportunities to learn what the policy implies and found higher student achievement when
for instruction is a crucial influence on their teachers had extended opportunities to learn

CPRE Research Report Series, RR-39 32

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

about mathematics curriculum and students’ achievement was changed by the

instruction. Brown, Smith and Stein (1995) professional development most California
analyzed teacher learning, practice, and teachers had experienced. Still, very large
student achievement data collected from four amounts of money are spent every year on
QUASAR project schools, and found that just such activities (Little, 1989). Our results
students had higher scores when teachers therefore challenge those who make policy
had more opportunities to study a coherent for and practice professional development:
curriculum designed to enhance both teacher can they design programs, policies, and
and student learning.37 requirements that focus more closely on
improved teaching for improved student
If our analysis is correct, when educational learning?
improvement is focused on learning and
teaching academic content, and when Our analysis also seems to confirm
curriculum for improving teaching overlaps arguments for standards-based reform in that
with curriculum and assessment for students, it broadly supports any approach to school
teaching practice and student performance improvement that leads to the creation of
are likely to improve. Under such better curriculum for students, that makes
circumstances educational policy is an suitable provision for teachers to learn that
instrument for improving teaching and curriculum, that focuses teaching on
learning. Policies that do not meet these learning, and that thoughtfully links
conditions—new assessments or curricula curriculum and assessment to teaching.
that do not offer teachers adequate Some examples of standards-based reform
opportunities to learn, or professional meet these criteria, but so do other
development that is not grounded in approaches to school improvement.39
academic content—are less likely to have
constructive effects.38 The story told here is not one in which the
efforts of state agencies carried the day.
These points have important bearing for the Rather, it is a story in which the related
professional development system—or non- actions of government and professional
system. Professional development that is organizations were crucial. California state
fragmented, not focused on curriculum for agencies played a key role in framing a set of
students and does not afford teachers ideas about improved math teaching and
consequential learning opportunities cannot learning, in supporting those ideas, and in
be expected to be a constructive agent of changing some state education requirements
state or local policy. Yet, that seems to be to be more consistent with the ideas. The
the nature of most professional development state alone, however, did not have the
in the U.S. today. Teachers typically engage educational resources to frame those ideas.
in a variety of short-term activities that fulfill The state did not have the intellectual,
state or local requirements for professional political, or fiscal resources to support the
learning but rarely are deeply rooted either in reforms. Most of the salient resources,
the school curriculum or in thoughtful plans including professional development, were
to improve teaching and learning. This offered by education professionals and their
study confirms that picture, and shows organizations, in agencies as diverse as
further that neither teachers’ practice nor National Council of Teachers of

CPRE Research Report Series, RR-39 33

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Mathematics and its California affiliate, California politics since the late 1980s,
home-office curriculum developers, opponents battle in a Manichean world: basic
university schools and department, among skills are diametrically opposed to true
others. Changes in teaching practice understanding, hard knowledge is totally
depended as much on professional as on opposed to fuzzy romanticism. California
state action.40 teachers are exhorted to radically change
their practice to avoid rote exercises, or they
Working together, these agencies were able are charged with irresponsibly ignoring
to create rational relationships among conventional math instruction as they
teachers’ learning, their practice, school embrace foolish radical reforms. But our
curriculum and assessments, and student reports on teachers’ behavior from below the
achievement. Such relationships were not surface suggest that most California teachers
easy to organize, and our evidence shows hold fast to conventional math teaching, and
that California reformers, after years of hard that even teachers who have taken the
work, achieved them for only fifteen to reforms most to heart attend to computation
twenty percent of the state’s teachers. That and other elements of conventional math
squares with what we know about instruction. Reformers’ hopes for deep and
fragmentation in the U.S. public education speedy change seem as misguided as
system (it is more nearly a non-system) conservatives’ worries about being
whose sprawling organization makes it very overtaken by the deluge. Both have
difficult to organize coherent and concerted something to learn from evidence about how
action even within a single modest-sized teachers actually do learn and change.
school district, let alone an entire state
(Cohen and Spillane, 1992). It also fits with Finally, all of this analysis rests on non-
recent research on teachers’ learning and experimental evidence, which is not
change, which shows that although certain conclusive. The relationships that we have
sorts of learning opportunities do seem to reported should be investigated with a larger
alter teachers’ practice and student learning, population of schools and teachers, in a
change typically occurs slowly and partially. longitudinal format, so that more robust
Few teachers in our sample—even those causal attributions might be probed, and
who had the most abundant learning more precise measures tried. We are trying
opportunities—wholly abandoned their past to organize such a study. But the results do
mathematics instruction and curriculum to not come from left field: they seem
embrace those offered by reformers. Rather, reasonably robust, and are quite consistent
the teachers who took most advantage of with several related lines of recent research.
new learning opportunities blended new We think better research on these issues is
elements into their practice while reducing essential, but we would be surprised if the
their reliance on conventional practices. direction of the effects we have found, and
our model of causation, do not stand up in a
These remarks about the pace of change more powerful design. We think it would be
return us to the opening of this essay, where wise for policymakers and practitioners to
we distinguished between life at and below ground teachers’ professional education
the surface of policy. At the surface, in more firmly in deeper knowledge of the
debates about math reform that have roiled student curriculum. When designing new

CPRE Research Report Series, RR-39 34

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

curricula and assessments, we think it would

wise to make more adequate provisions so
teachers could learn about and from the new
curricula and assessments. And we think it
would be wise to offer teachers more
opportunities to relate assessments to
curricula, and to relate both to their
pedagogy.

CPRE Research Report Series, RR-39 35

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

References
Ball, D.L., and Rundquist, S. S. (1993). Cohen, D. K. (1990). Governance and
Collaboration as context for joining instruction: The promise of
teacher learning with learning about decentralization and choice. In Witte
teaching. In D. Cohen, M. McLaughlin, and Clune (Eds.), Choice and Control in
and J. Talbert (Eds.), Teaching for American Education Vol 1, (pp. 337-
Understanding: Challenges for Policy 86). New York: Falmer.
and Practice (pp. 13-42). San Francisco
CA: Jossey Bass. Cohen, D. K., and Spillane, J. P. (1993).
Policy and practice: The relations
Barr, R., and Dreeben, R. (1983). How between governance and instruction. In
Schools Work. Chicago: University of S. Fuhrman (Ed.), Designing Coherent
Chicago Press. Education Policy: Improving the System
(pp. 35-95). San Francisco: Jossey-Bass.
Berman, P., and McLaughlin, M. W. (1978).
Federal programs supporting educational Cuban, L. (1984). How Teachers Taught :
change. Vol. VIII: Implementation and Constancy and Change in American
sustaining innovations. Santa Monica Classrooms, 1890-1980. New York:
CA: RAND Corporation. Longman.

Berliner, D. (1979). Tempus Educare. In Dow, P. B. (1991). Schoolhouse Politics:

P. Peterson and H. Walberg (Eds.), Lessons from the Sputnik Era.
Research on Teaching: Concepts, Cambridge: Harvard University Press.
Findings, and Implications. (pp. 120-
135). Berkeley, CA: McCutchan. Elmore, R. F. (1979). Backward mapping:
Implementation research and policy
Brown, C. A., Smith, M. S., and Stein, M. decisions. Political Science Quarterly,
K. (1996). Linking teacher support to 94(4), 601-616.
enhanced classroom instruction. Paper
presented at the annual meeting of the Heaton, R.M., and Lampert, M. (1993).
American Educational Research Learning to hear voices: Inventing a new
Association, New York, NY. pedagogy of teacher education. In D.
Cohen, M. McLaughlin, and J. Talbert
Clogg, C. C., Petkova, E., and Haritou, A. (Eds.), Teaching for Understanding:
(1995). Statistical methods for Challenges for Policy and Practice (pp.
comparing regression coefficients 43-83). San Francisco, CA: Jossey-Bass.
between models. American Journal of
Sociology, 100 (5), 1261-1312. Lash, A., Perry, R., and Talbert, J. (1996).
Survey of elementary mathematics
Cohen, D., and Ball, D. L. (1996). Capacity education in California: Technical report
and capacity building for instruction. and user’s guide to SAS files. Stanford:
Paper presented at AERA, New York, Author.
NY.

CPRE Research Report Series, RR-39 36

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Leinhardt , G., and Seewaldt, A. M. (1981). McLaughlin, M. W. (1991). Enabling

Overlap: What’s tested, what’s taught. professional development: What have we
Journal of Educational Measurement, learned? In A. Lieberman and L. Miller
18(2), 85-95. (Eds.), Staff Development for Education
in the '90s (pp. 61-82). New York:
Linn, R.L. (1983). Testing and instruction: Teachers College Press.
Links and distinctions. Journal of
Educational Measurement, 20(2), 179- O’Day, J. and Smith, M. (1993). Systemic
189. reform and educational opportunity. In
S. Fuhrman (Ed.), Designing Coherent
Lipsky, M. (1980). Street level bureaucracy: Education Policy: Improving the System
Dilemmas of the individual in public (pp. 250-312). San Francisco: Jossey-
services. New York NY: Russell Sage Bass.
Foundation.
Perry, R. (1996). The role of teachers’
Little, J. W. (1989). District policy choices professional communities in the
and teachers’ professional development implementation of California
opportunities. Educational Evaluation mathematics reform. Unpublished
and Policy Analysis, 11(2), 165-179. doctoral dissertation, Stanford
University, Stanford.
Little, J. W. (1993). Teachers’ professional
development in a climate of educational Pressman, J. L., and Wildavsky, A. B.
reform. Educational Evaluation and (1984). Implementation. Berkeley:
Policy Analysis, 15, 129-151. University of California Press.

Lord, B. (1994). Teachers’ professional Schifter, D., and Fosnot, C. T. (1993).

development: Critical colleagueship and Reconstructing Mathematics Education.
the role of professional communities. In New York NY: Teachers’ College Press.
N. Cobb (Ed.), The Future of Education:
Perspectives on National Standards in Weiss, I. (1994). A profile of science and
Education (pp. 175-204). New York: mathematics education in the United
College Entrance Examination Board. States: 1993. Chapel Hill, NC: Horizon
Research Inc.
McCarthy, S. J., and Peterson, P. L. (1993).
Creating classroom practice within the Wiley, D., and Yoon, B. (1995). Teacher
context of a restructured professional reports of opportunity to learn: Analyses
development school. In D. Cohen, M. of the 1993 California learning
McLaughlin, and J. Talbert (Eds.), assessment system. Education
Teaching for Understanding: Challenges Evaluation and Policy Analysis, 17(3),
for Policy and Practice (pp. 130-166). 355-370.
San Francisco, CA: Jossey-Bass.

CPRE Research Report Series, RR-39 37

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Wilson, S. M., Miller, C., and Yerkes, C.

(1993). Deeply rooted change: A tale of
teaching adventurously. In D. Cohen, M.
McLaughlin, and J. Talbert (Eds.),
Teaching for Understanding: Challenges
for Policy and Practice (pp. 84-129).
San Francisco, CA: Jossey-Bass.

CPRE Research Report Series, RR-39 38

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Appendix A
A Note on Sampling
Our primary sampling unit was the school From the 250 schools sampled, one teacher
district. Because the number of students in from each of grades 2-5 was selected at
each district varies greatly, districts were random and mailed a long-form survey.
stratified into five categories by student Because some schools did not support four
population and unevenly sampled in order to teachers for these grades, the final number of
achieve probabilities proportionate to size. teachers in our sample is 975, rather than
1,000.

Stratum Size of District Number of Number of

(in students) districts schools
sampled/total sampled/district
districts in strata
1 (LA) 1/1 10
2 35,000+ 10/10 5
3 10,000-35,000 50/97 2
4 1,000-10,000 70/367 1
5 LT 1,000 20/421 1

CPRE Research Report Series, RR-39 39

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Appendix B
A two-stage-least-squares was performed on treatments of mathematics and
the student curriculum models to help student curriculum. A regression
mitigate against “selection effects”—that is, analysis also shows that this variable
the possibility that teachers who attended has few direct relationships with
one of these workshops did so because they traditional and reform practices,
were somehow predisposed to teach to the controlling for workshop and
Frameworks. In order to do so, we assessment-related learning.
identified variables which affect the
probability teachers would attend a Marilyn • District development, a variable
Burns or Replacement Unit workshop and marking teacher participation in
estimated a logit equation representing that district mathematics committees or in
relationship. We then took the predicted teaching math in-services. Again,
values from this first equation and used them knowledge of the content of those
instead of the Student Curriculum Workshop activities is key to understanding
(SCW) marker in the student curriculum whether this should affect teacher
models. practice or not. In the absence of this
information, however, we proceed on
As is necessary to resolve endogeneity the basis of results from a regression
problems, we needed to identify factors analysis which shows this marker
which affect the probability a unit will select unrelated to teacher practices.
into the “treatment” condition but which do
not affect the final outcome variable. In • Administrative support. A three-item
other words, we searched for factors which measure of teachers’ reports of the
might encourage teachers to take these extent to which their principal,
workshops but would not have a direct effect school, and district are well-informed
on their practice. Using both theory and and favorable toward the Frame-
empirical investigation, we have identified works. One item specifically asked
three such factors: about the amount of staff develop-
ment supplied by the district. School
• Policy, a variable marking teacher and district instructional policy,
attendance at national or regional however, is not thought to have great
mathematics meetings. Such direct impact on teacher practice, and
participation should affect teacher this measure has no direct effect on
practice if the content of meetings our practice scales. 41
focuses on substantive matters of
instruction and mathematics; where We also chose to include two more variables
the focus is administrative or political in the first-stage selection equations—
matters, practice is less likely to be teacher affect toward the reforms, and
affected (Lichtenstein et al, 1992). teacher familiarity with the reforms—on the
The content of California’s meetings view that these markers might indicate
was mixed during this time period, teacher desire to learn about both the
but tended toward more superficial reforms and children’s curriculum as a

CPRE Research Report Series, RR-39 40

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

vehicle for those reforms. To the extent Teachers’ reports on all these measures were
these capture teacher “will” they will act as entered into the first stage probit equation
important controls. predicting whether or not teachers attended a
MB/RU activity in 1993-1994:

AttendSCW = b0 + b1 affect + b3 familiar + b4 policy + b5 district + b6 adminsup

Attended Student Curriculum

Workshop
Intercept -5.44*
.80
Affect .21*
0.11
Familiar 2.07*
.71
Policy Networks 1.20*
.34*
District Development .74*
.23
Administrative Support .20*
.80
Log Likelihood -336.29
P =71.43, p=.000

All five proved moderately strong predictors weaker or non-existent relationships to

of student curriculum workshop attendance. workshop attendance. It is noteworthy that
Other variables—teacher math background, teacher affect toward the reforms is
classes in mathematics teaching, student outperformed by other predictors in this
race and class—were examined but yielded equation.

CPRE Research Report Series, RR-39 41

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Based on the first stage model above, a observation in the sample, and this predicted
predicted level of SCW (zero or one) was value was entered into a pair of practice
generated using the probit model for each equations similar to those in Table 5:

practice = b0 + b1 SCW + b2 affect + b3 familiar

1 2
Traditional Framework
Practice Practice
Intercept 1.83* 2.06*
0.22 0.17
Predicted SCW Attendance -0.74* 1.04*
0.22 0.19
Affect -0.22* .17*
0.04 0.03
Familiar -.77* 0.12*
0.25 0.22
R-squared (adjusted) .20 .19

Here, the coefficient on SCW increases from The same procedure was accomplished for
.54 (se=.06) to 1.04 (se=.19) in the the regressions using the variable “time in
framework practice regression. The increase student curriculum workshop” instead of the
in the coefficient is likely due to the simple dummy MB/RU. Similar results
decreased precision with which our statistical obtained.
package can estimate the two-stage equation
rather than to substantive differences in its This method—specifying a two-stage model
real value. The coefficient in the traditional in which the first stage is a probit—tends to
practice regression likewise dropped from - inflate standard errors for the regressors in
.28 (se=.07) to -.74 (se=.22) but likewise the model. Because our regressors remained
saw higher standard errors. Despite the significant predictors of teacher practice
decrease in precision with which we could outcomes, however, we did not pursue
estimate both equations, we note that both methods to correct this problem.
measures of SCW remain significant and
related to the dependent variables in the
expected direction.

CPRE Research Report Series, RR-39 42

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

Appendix C
Clogg, Petkova and Haritou’s (1995) test for and thus warranting of a claim that the
difference in nested coefficient compares regression is in fact incorrect without the
point estimates within models with and competitor variable included. Here, we
without one or a set of predictors. Point examined the point estimate on student
estimates for the variable in question—here curriculum workshop both with and without
“student curriculum workshop”—are the CLAS variables—“CLAS Useful,”
compared with and without the competing CLAS-OTL, CLASADM—in and out of the
explanatory variable(s) in the equation to see equation. For more details, see Clogg,
if the difference in its effect is significant, Petkova, and Haritou (1995).

Restricted Full Model d t

Model (se of (se of
(se of estimate) estimate)
estimate)
Practice Framework
Student Curriculum .378 .366 .012 1.15
Dummy (.076) (.073) (.0104)

Student Curriculum .083 .065 .018 3.6

Time (.0189) (.0174) (.004)
MSE (full) =
.904 MSE
(restricted)
Conventional Framework
Student Curriculum -.155 -.141 -.014 .5
Dummy (.083) (.088) (.028)
Student Curriculum -.067 -0.62 .005 .83
Time (.021) (.020) (.006)
MSE (full) =
.998 MSE
(restricted)

CPRE Research Report Series, RR-39 43

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

End Notes
1
In the 1970s and early 1980s, in response to worries about relaxed standards and weak
performance by disadvantaged students, states and the federal government pressed basic skills
instruction on schools, supporting the idea with technical assistance and enforcing it with
standardized “minimum competency” tests. Those tests were America’s first post-war brush with
performance-oriented schooling.
2
One of us has dealt with the political issues in several recent essays (see Cohen, 1991 and Cohen
and Spillane, 1992).
3
In California, as in Texas, the State Board of Education decides what texts are suitable for local
adoption. Local districts can use other texts, but by so doing they lose some state subsidies.
4
Denham original interview.
5
The chief exceptions to this rule were the RAND Change Agent studies (Berman and
McLaughlin 1987), Elmore (1979), and Pressman and Wildavsky (1984). Lipsky (1980) offers
one of the few efforts at extended explanation of policy failures from a perspective of practice.
6
This paper is part of a continuing study of the origins and enactment of the reforms, and their
effects. The study began in 1988, led by Deborah Loewenberg Ball, David K. Cohen, Penelope
Peterson, and Suzanne Wilson, and it involved an extended group of associated researchers at
Michigan State University.
7
See, for example, Educational Evaluation and Policy Analysis 12 (3).
8
The survey was designed by Ball, Cohen, Peterson and Wilson, in partnership with Dr. Joan
Talbert at the Stanford University Graduate School of Education—and carried out by Dr. Talbert
(see Appendix A for a summary of the sampling frame). We owe many thanks to Deborah Ball,
Penelope Peterson, Joan Talbert, and Suzanne Wilson for help at many points, and are especially
indebted to Dr. Talbert. The survey was supported by the National Science Foundation (Grant #
ESI-9153834 ).
9
As is often the case with factor analyses, the “results” were dependent on statistical
specifications. When different types of factor analyses turned up conflicting results for specific
items, theoretical judgements were made concerning where those items belonged. In the main,
however, every factor analysis run turned up two dimensions—conventional and Framework
practice.
10
It is common in workshops like EQUALS and cooperative learning for teachers to engage in
mathematical activities which they may then try out with their classes. We feel it is important to
distinguish between these activities, which tend to be short exercises intended to motivate or
introduce students to a topic, from the kind of curriculum offered by a replacement unit.

CPRE Research Report Series, RR-39 44

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

11
Iris Weiss’ 1993 National Survey of Science and Mathematics Education suggests that teachers
in California may be getting more time in staff development in mathematics than their peers
elsewhere. Weiss reported that 32 percent of first to fourth grade teachers attended more than 16
hours of staff development over the past three years. In our data, nearly 20 percent of second
through fifth grade teachers attended sixteen hours or more total staff development in the last year
alone.
12
The survey asked teachers to circle an amount of time ranging from “one day or less” to “more
than two weeks” rather than write the number of days they spent at each activity. To calculate
time spent, we assumed the following: “One day or less” = 1 day; “2-6 days” = 4 days; “1-2
weeks” = 10.5 days; and “More than 2 weeks” = 14 days. We then added the teachers’ reports of
workshop attendance.
13
Only a modest proportion of teachers reported more than one day at either kind of workshop,
and the mean of our “time spent” markers was .91 for student curriculum, and .5 for the special
topics/issues workshops.
14
A number of respondents in this category, for instance, reported using replacement units,
indicating they had perhaps attended a replacement unit workshop in a past year.
15
Our scale actually had six levels: 1-5 negative-positive and a level 6 for “don’t know.” Since
several analyses showed individuals who answered “6 ” to be quite similar to those who answered
3 (to indicate neutrality) on the scale we transformed the don't knows into neutrals. The
regression results presented here do not change in the absence of this “fix”—but making the
replacement does reduce the number of cases lost to missing data in all models.
16
Our hypothesis is not that knowing of broad policy objectives will, ceteris paribus, lead
teachers to the greater classroom enactment; knowledge of broad policy prescriptions is not the
same as practice, many of these practices require learning and resources, and the scale of
familiarity does not measure knowledge deeply.
17
We say “smaller scale” because that is what we have found; familiarity with reform has a
stronger influence on teachers’ beliefs than on their practice.
18
When we run the models in Table 5 without affect and familiar with controls, the size of the
coefficients on the student curriculum workshop variables increases.
19
Because of the unique format in which time-in-workshop was reported on this survey, an
additional analysis not presented here was necessary to confirm this point. This was accomplished
by breaking each workshop into a set of five dummy variables representing a discrete time
investment (Marilyn Burns—1 day, Marilyn Burns—2-6 days, etc.), and entering these alone into
the practice scales. Greater increments of time did in general “add” to teachers’ reports of
Framework practice and “subtract” from their reports of conventional practice. No time
measurement was available for our variable measuring previous Framework learning.
20
See Appendix B.

CPRE Research Report Series, RR-39 45

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

21
The question asked if teachers “...participated in any activities that provided [them] with
information about the CLAS (for example, task development, scoring, pilot testing, staff
development).”
22
By size of association, we mean to say that the simple effect associated with a teacher attending
a student curriculum workshop or not—about 3/4 of a standard deviation of Framework practice,
and about 4/10 of a standard deviation in conventional practice—is not matched by the impact of
administering CLAS, which has an impact of only about 2/10 of a standard deviation on the
practice scale.
23
See again Educational Evaluation and Policy Analysis 12 (3).
24
Thanks to Jennifer O’Day for this point.
25
Making the four survey items in Table 8 into a dependent measure and regressing it on
“administered CLAS” and “learned about CLAS” show that both learning and doing add about
the same amount of “enthusiasm” to teachers’ responses.
26
This scale runs from 1 (CLAS did not correspond...) to 5 (CLAS corresponded well...). Its
mean is 3.24, its standard deviation 1.02, and its reliability .85.
27
According to the test suggested by Clogg, Petkova and Haritou (1995), the change in three of
the four coefficients in question is non-significant. See Appendix C for details.
28
The same statistics for all elementary schools in the state are:

N Mean Std Dev Minimum Maximum

4228 2.8135951 0.6242373 0 5.0200000

The student-level standard deviation for our sample (constructed from schools’ reports of student
distributions) is 1.728.
29
To the extent teachers’ workshop learning occurred in the summer of 1994 (after the test) we
could underestimate the effect of these workshops on student learning.
30
The CLAS scores also have some measurement error, most of it consistent with the usual
problems associated with psychometric research. Also, the California Department of Education
reports that school CLAS scores were not reported in the case where error in the score crossed
above a threshold of acceptability, the number of students on which the score was based was low,
or the number of students who opted out of taking the test was too high. We compared schools
that we did use in the CLAS analysis against those we could not use (because they had missing
school scores, had only one teacher who responded to our survey, or were unusable for some
other reason). Of our independent variables, significant differences between the two groups
occurred in only a handful of cases: schools with CLAS data tended to have fewer free-lunch-
eligible students; schools with CLAS data tended to have teachers who reported more

CPRE Research Report Series, RR-39 46

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

opportunities to learn about the assessment, were more likely to have teachers who said they had
administered the test, and had higher scores on the “CLAS useful” scale; schools with CLAS data
also had more teachers, on average, who attended student curriculum workshops, although there
is no significant difference in the “time” correlate of this variable used in the CLAS analysis.
31
We include this variable in our equations because educational environments are not perfectly
correlated with student socio-economic status; some schools enrolling many free-lunch-eligible
children, for example, have teachers who report quite orderly environments, with lots of parental
support and good building facilities. In response to the question, “How well does each of the
following statements describe general conditions and resources for mathematics teaching in your
classroom, school, and district?” The scale items are: (1) Adequate parent support of your
instruction; (2) High student turnover during the school year; and (3) Well-maintained school
facilities.
32
We did not enter two separate variables showing whether and how long teachers attended the
learning opportunities as we did in the practice analysis, since the second captures the information
of the first, for the purposes of this investigation.
33
This variable is under-specified, but not including it biases the coefficients on the remaining
variables, since teachers with some previous learning opportunities would be marked as zero, and
throw the “baseline” off.
34
Likely, this correlation would rise if we had career-long estimates of teachers’ attendance at
student curriculum workshops.
35
We tried both “Learned about CLAS” and “CLAS useful” in this model, since both could be
measures of teachers’ attempts to prepare students for the test. “CLAS Useful” was not
significant, and evidenced colinearity with “Framework Practice.”
36
There is reason to expect that the coefficient on student curriculum-time in this model—and
elsewhere—is actually underestimated. Remember that the survey asked teachers to report
workshop learning of this type within the last year—leaving teachers who attended student
curriculum workshops in previous years and now use replacement units represented by only the
replacement unit marker. This will bias the effect of replacement unit use up, and student
curriculum-time down.
37
These studies are supported indirectly by other work on learning opportunities, including
Cooley and Leinhardt’s Instructional Dimensions Study, other research concerning the
significance of time on task, and studies of the relationship between the purposes and content of
instruction (Barr and Dreeben, 1983; Berliner, 1979). The results also are consistent with
research on domain-specific learning in cognitive psychology, and psychometric research on the
importance of consistency between assessment and curriculum in assessing educational
interventions (Leinhardt and Seewaldt, 1981; Linn, 1983).
38
Efforts to improve schools typically have focused only on one or another of the influences that
we discussed. Challenging curricula have failed to broadly influence teaching and learning at least

CPRE Research Report Series, RR-39 47

INSTRUCTIONAL POLICY AND CLASSROOM PERFORMANCE Cohen and Hill

partly because teachers had few opportunities to learn and improve their practice (Dow 1991).
Countless efforts to change teacher’s practices in various types of professional development have
been unrelated to central features of the curriculum that students would study, and have issued in
no evidence of effect on students’ learning. Many efforts to drive instruction by using high-stakes
tests failed to either link the tests to the student curriculum or to offer teachers substantial
learning opportunities. These and other interventions assume that working on one of the many
elements that shape instruction will affect all the others, but lacking rational relationships among
at least several of the key influences, that assumption seems likely to remain unwarranted.
39
For example, Success For All embodies such coherence.
40
We have profited from reading portions of Suzanne Wilson’s book manuscript that concern
educators learning in and from the California reforms.
41
We have so far only performed the check for “administrative support” in SAS; a more proper
estimation technique might be HLM, given that this is a school or district-level variable. It would
be surprising, given the very low coefficient on this variable, if HLM changed the results to any
great extent. There is also an argument for the view that different communities of support exist
within the same schools—and therefore the individual-level measure is more appropriate.

CPRE Research Report Series, RR-39 48