Downloaded 18 times
Uploaded byOECD
2010 e-madrid (spanish presidency) - rev 1.0
The document discusses indicators of education systems and frameworks for measuring educational quality and performance across countries. It addresses the need to (1) understand policy objectives and context, (2) consider multiple domains and levels from individual learners to national systems, and (3) recognize indicators that provide policy-relevant insights while also being technically feasible and comparable cross-culturally. International assessments like PISA aim to meet these criteria by measuring student performance, highlighting disparities, and informing policy, but require balancing breadth, depth, and country-specific needs.
More Related Content
Similar to 2010 e-madrid (spanish presidency) - rev 1.0
![2011 c-oslo [english] - rev 1.1](https://cdn.slidesharecdn.com/ss_thumbnails/2011-c-osloenglish-rev1-1-110330055917-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![2010 C Washington (Future Of Assessments) [No Notes] Rev 1 1](https://cdn.slidesharecdn.com/ss_thumbnails/2010-c-washingtonfutureofassessmentsnonotes-rev1-1-100311054332-phpapp01-thumbnail.jpg?width=640&height=640&fit=bounds)
2010 e-madrid (spanish presidency) - rev 1.0
- 1. Indicators of EducationSystemsSeeing your education system in the mirror of other systemsMadrid, 24 May 2010Andreas SchleicherEducation Policy Advisor of the OECD Secretary-General
- 2. Indicators of educationsystemsOr the Alchemists’ Stone?The Holy Grail?Know why you are lookingYou cannot improve what you cannot measureThe yardstick for success is no longer just improvement by national standards but the best performing education systems globallyKnow what you are looking forSystemic frameworkPolicy relevanceCross-country and cross-cultural validityKnow howyou will recognise it when you find itWhat makes a good indicator system?Implications and lessons learned .
- 3. Know what youare looking forThe Holy Grail was a well-described object, and there was only one true grail…
- 4. More than compilationsof statisticsDo we understand the policy objectives
- 5. What countries careabout, what they intend to do about it and how they define success
- 6. Do we understandthe policy context and can we anticipate its future development ?
- 7. Do we understandthe strategic requirements for change?
- 8.
- 9.
- 10. Politically and sociallysuitable?
- 11.
- 12. Do we understandthe delivery challenge and delivery capacity?
- 13. Nature and sizeof the barriers that systems face to deliver reform goals
- 14. Can what worksin one country by done in another by real people in real situations?
- 15. Avoiding big timeand energy traps?
- 16. Do we understandpast and present performance vis a vis the policy goals as well as the drivers of performance and their underlying system activities?
- 17. What is addedvalue of international comparisons?Domain 1Individual learnerLevelALevelBInstructional settingsLevelCSchools, other institutionsCountry or systemLevelDDimensions of an indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourSocio-economic background of learnersStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policies
- 18. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 19. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Cost per studentGraduate supplyTertiary-type A graduation rate
- 20. A world ofchange – highereducationExpenditure per student at tertiary level (USD)United StatesCost per studentFinlandJapanSpainGraduate supplyTertiary-type A graduation rate
- 21. A world ofchange – highereducationExpenditure per student at tertiary level (USD)AustraliaFinlandUnited KingdomTertiary-type A graduation rate
- 22. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Tertiary-type A graduation rate
- 23. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Tertiary-type A graduation rate
- 24. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Tertiary-type A graduation rate
- 25. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Tertiary-type A graduation rate
- 26. A world ofchange – highereducationExpenditure per student at tertiary level (USD)Tertiary-type A graduation rate
- 27. A world ofchange – highereducationWhat about international students?Expenditure per student at tertiary level (USD)United StatesAustraliaAAUnited KingdomFinlandSpainATertiary-type A graduation rate
- 28. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 29. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 30. Components of theprivate net present value for a male with higher education27K$56K$170K$105K$35K$26K$367K$Net present value in USD equivalent
- 31. Public cost andbenefits for a male obtaining post-secondary educationPublic costsPublic benefitsNet present value, USD equivalent(numbers in orange shownegative values)USD equivalent
- 32. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 33. Spending choices onsecondary schoolsContribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004)Percentage points
- 34. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 35. Domain 1Dimensions ofan indicators frameworkDomain 3Domain 2Antecedentscontextualise or constrain ed policyPolicy Leversshape educational outcomesOutputs and Outcomesimpact of learningQuality and distribution of knowledge & skillsIndivid attitudes, engagement and behaviourLevelASocio-economic background of learnersIndividual learnerLevelBStudent learning, teacher working conditionsQuality of instructional deliveryTeaching, learning practices and classroom climateInstructional settingsThe learning environment at schoolCommunity and school characteristicsOutput and performance of institutionsLevelCSchools, other institutionsNational educ, social and economic contextSocial & economic outcomes of educationStructures, resource alloc and policiesCountry or systemLevelD
- 36. OECD’s PISA assessmentof the knowledge and skills of 15-year-oldsCoverage of world economy83%77%81%85%86%87%
- 37. High science performanceAverageperformanceof 15-year-olds in science – extrapolate and applyPoland 2000… 18 countries perform below this lineLow science performance
- 38. High science performanceAverageperformanceof 15-year-olds in science – extrapolate and applyHigh average performanceLarge socio-economic disparitiesHigh average performanceHigh social equityStrong socio-economic impact on student performanceSocially equitable distribution of learning opportunitiesLow average performanceLarge socio-economic disparitiesLow average performanceHigh social equityLow science performance
- 39. High science performanceDurchschnittlicheSchülerleistungen im Bereich MathematikHigh average performanceLarge socio-economic disparitiesHigh average performanceHigh social equityStrong socio-economic impact on student performanceSocially equitable distribution of learning opportunitiesLow average performanceLarge socio-economic disparitiesLow average performanceHigh social equityLow science performance
- 40. Consistency in qualitystandardsVariation in the performance of 15-year-olds in mathematics20
- 41. Consistency in qualitystandardsVariation in the performance of 15-year-olds in mathematicsVariation of performance within schoolsVariation of performance between schoolsOECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.
- 42. Pooled international dataset,effects of selected school/system factors on science performance after accounting for all other factors in the modelSchool principal’s positive evaluation of quality of educational materials(gross only)Schools with more competing schools(gross only)Schools with greater autonomy (resources)(gross and net)School activities to promote science learning(gross and net)One additional hour of self-study or homework (gross and net)One additional hour of science learning at school (gross and net)School results posted publicly (gross and net)Academically selective schools (gross and net) but no system-wide effectSchools practicing ability grouping (gross and net)One additional hour of out-of-school lessons (gross and net)20Each additional 10% of public funding(gross only)School principal’s perception that lack of qualified teachers hinders instruction(gross only)Effect after accounting for the socio-economic background of students, schools and countriesMeasured effectOECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a
- 43. Know how youwill recognise it when you find itThe Alchemists’ stone was to be recognisedby transforming ordinary metal into gold…
- 44.
- 45. Increased likelihood ofpostsec. particip. at age 19/21 associated with PISA reading proficiency at age 15 (Canada)after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group PISA Level 1)Odds ratioCollege entrySchool marks at age 15PISA performance at age 15
- 46.
- 47. Latin America thenand now…Hanushek 2009
- 48. Latin America thenand now…Why quality is the keyHanushek 2009
- 49. High policy relevanceQuickwinsMust havesOutcomes of educationFinancing of educationParticipation and progressionProcesses and structuresLow technical quality/costlyHigh technical qualityMoney pitsLow-hanging fruits…Low policy relevance
- 50. Implications and lessonslearnedThe medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong foundationsThe search for the Holy Grail was overburdened by false clues and cryptic symbols
- 51. Some things arealways difficultFocus on added value of international comparisonsNot every policy question canbebestanswered through international comparisons Trading off breadth and depthsNot everything that isimportantneedstobe dealt withwithexcruciatingdetailSeek outputs that are as comparable as possible…… but as country-specific as necessaryFocus coverage as much as feasible…… but keep as large as necessary to be useful for policy formation .
- 52. High policy valueAreal-time assessment environment that bridges the gap between formative and summative assessment .Quick winsMust havesExamine individual, institutional and systemic factors associated with performanceExtending the range of indicators through which educational quality is assessedMonitor educational progressMeasuring growth in learningLow feasibilityHigh feasibilityEstablish the relative standing of students and schoolsAssuming that every new indicator is orthogonal to all othersMoney pitsLow-hanging fruitsLow policy value
- 53. www.oecd.org; www.pisa.oecd.orgAllnational and international publicationsThe complete micro-level databaseemail: [email protected]@OECD.org… and remember: Without data, you are just another person with an opinionThank you !
Editor's Notes
- #5 Such an OECD knowledge base needs to reflect three dimension, (1) an understanding of what countries are trying to achieve, (2) the evidence that we have, and (3) what we know about policy delivery in different contexts. As concerning the policy context: Do we understand what countries care about, what they intend to do about it and how they define success? Do we understand the policy context, and that is not just about the context as it stands, but how we anticipate its future development, because in an area like education where the delivery chain of reforms is so long, we need to think far into the future. Do we understand the strategic requirements for change, and do we judge various approaches to change technically feasible? And not everything that is feasible is politically suitable. And not everything that can be done is robust and cost-effective. With regard to the evidence: Are our indicators and analyses adequately capturing past and present performance of the education system vis a vis the reform goals, and have we been able to pin-point the drivers of performance and their underlying system activities? How does what we produce at the OECD, whether that is data, country reviews or thematic review relate to and add value to what other players provide?Finally, to what extent is our analysis useful and actionable in national contexts. Do we understand the challenges for reform delivery in countries and their delivery capacity? Is what we propose doable by real people in real situations and avoiding big time and energy traps?
- #6 The pace of change is most clearly visible in higher education, and I want to bring two more dimensions into the picture here. Each dot on this chart represents one country. The horizontal axis shows you the college graduation rate, the proportion of an age group that comes out of the system with a college degree. The vertical axis shows you how much it costs to educate a graduate per year.
- #7 *Lets now add where the money comes from into the picture, the larger the dot, the larger the share of private spending on college education, such as tuition.The chart shows the US as the country with the highest college graduation rate, and the highest level of spending per student. The US is also among the countries with the largest share of resources generated through the private sector. That allows the US to spend roughly twice as much per student as Europe. US, FinlandThe only thing I have not highlighted so far is that this was the situation in 1995. And now watch this closely as you see how this changed between 1995 and 2005.
- #8 You see that in 2000, five years, later, the picture looked very different. While in 1995 the US was well ahead of any other country – you see that marked by the dotted circle, in 2000 several other countries had reached out to this frontier. Look at Australia, in pink.
- #14 Thatwasallveryquick, letusgothroughthisdevelopmentonceagain
- #25 The best way to find out whether what students have learned at school matters for their life is to actuallywatch what happens to them after they leave school. This is exactly what we have done that with around 30,000 students in Canada. We tested them in the year 2000 when they were 15 years old in reading, math and science, and since then we are following up with them each year on what choices they make and how successful they are in their transition from school to higher education and work.The horizontal axis shows you the PISA level which 15-year-old Canadians had scored in 2000. Level 2 is the baseline level on the PISA reading test and Level 5 the top level in reading.The red bar shows you how many times more successful someone who scored Level 2 at age 15 was at age 19 to have made a successful transition to university, as compared to someone who did not make it to the baseline PISA level 1. And to ensure that what you see here is not simply a reflection of social background, gender, immigration or school engagement, we have already statistically accounted for all of these factors. The orange bar. …How would you expect the picture to be like at age 21? We are talking about test scores here, but for a moment, lets go back to the judgements schools make on young people, for example through school marks. You can do the same thing here, you can see how well school marks at age 15 predict the subsequent success of youths. You see that there is some relationship as well, but that it is much less pronounced than when we use the direct measure of skills.
- #26 Let us go back to the 1960s. The chart shows you the wealth of world regions and the average years of schooling in these regions, which is the most traditional measure of human capital. Have a look at Latin America, it ranked third in wealth and third in years of schooling, so in the 1960s the world seemed pretty much in order.
- #27 But when you look at economic growth between 1960 and 2000, you see that something went wrong. Despite the fact that Latin America did well in terms of years of schooling, only Sub-Saharan Africa did worse in terms of economic growth. So in 2000, Latin America had fallen back considerably in terms of GDP per capita.You can draw two conclusions from this: Either education is not as important for economic growth as we thought, or we have for a long time been measuring the wrong thing.
- #28 Now let me add one additional element, and that is a measure of the quality of education, in the form of the score of the different world regions on international tests like PISA or TIMSS. And you see now that the world looks in order again, there seems a close relationship between test scores and economic growth. You can see that even more clearly when you put this into graphical form. This is one of the charts produced by Professor Hanushek. And, as Professor Hanushek will explain, the relationship holds even when you account for other factors, it even holds when you compare growth in economies with growth in learning outcomes, which is the closest we can come to examining causality.So what this tells you is that it is not simply years of schooling or the number of graduates we produce, but indeed the quality of learning outcomes that counts.