PISA education ranking naming Canada as a superpower is ‘misleading’

The working paper, published today from the IOE, finds that although it remains plausible that educational achievement in Canada is higher than in the average OECD country, low participation rates to the PISA international education ranking system - an influential large-scale study of 15-year-olds achievement in reading, science and mathematics which is now conducted in more than 70 countries - raises questions over the county’s performance. 

Joint first author, Professor John Jerrim said: “After the release of the PISA 2015 results, Canada was described as an “education superpower”, with various theories, from the strong academic performance of immigrants through to high levels of student motivation, put forward to explain this result. Andreas Schleicher – the man who has led the OECD’s PISA programme – himself suggested that the strong commitment to equity in Canada is the key.”

The new paper highlights how the PISA 2015 data for Canada suffers from a much higher rate of student exclusions, school non-response and pupil non-response and omission of students with special education needs compared to other high-performing countries such as Finland, Estonia, Japan and South Korea. 

“Canada only just meets – and sometimes even misses – the minimum threshold the OECD sets for several key criteria that are meant to ensure the quality of the PISA data. A combination of these factors leads us to believe that there are serious problems with comparing the PISA 2015 data for Canada to other countries.”

When looking at the PISA data the researchers found that only around half of 15-year-olds in Canada were covered within the PISA 2015 assessment compared to 90% or more of 15-year-olds in Japan and South Korea.

One of the reasons fewer students took part was because schools in Canada were more likely to refuse to take part. This was a particularly big problem in Quebec, where less than half the schools approached agreed to participate. 

Secondly, Canada was much more likely to exclude pupils from taking the PISA assessment for reasons such as special educational needs – a group who are likely to be very low achievers. In total, 7.5% of 15-year-olds were excluded in Canada compared to 2.4% in Japan and less than 1% in South Korea.

Finally, students in Canada were less likely to actually sit the PISA assessment – even within schools that agreed to take part. The figures show that one-in-five Canadian teenagers were counted as absent on the day of the PISA test compared to less than 3% of those in Japan and South Korea.

Professor Jerrim added: “All this adds up to a sizeable problem, which I believe significantly undermines our confidence in the PISA 2015 data for Canada and ability to draw comparisons with other high-performing countries where a genuinely representative cross-section of children took part. After scratching below the surface, evidence of Canada being an “education superpower” is misleading and does not seem to be particularly strong.”

The authors suggest that such problems may not be isolated to Canada alone. They note how there are question marks over the quality of the PISA sample for other countries such as New Zealand, while the Netherlands should have been excluded from the 2015 rankings were the OECD’s data quality criteria strictly enforced. 

The researchers also caution that similar problems could affect the PISA 2018 data for the United Kingdom – with the latest release of these closely watched education rankings just two weeks away.  

In order to restore faith in the quality of the PISA sample, the academics suggest the OECD who lead the study raise the minimum student response rate from 80% to 90% and that the 5% criterion for student exclusions be much more strictly applied. School response rates that are below 65% which is labelled ‘unacceptable’ should also be automatically excluded. 

Media contact

Mark Greaves, UCL Media Relations
+44 (0)20 3108 9485
m.greaves@ucl.ac.uk

Links

• Read the paper ‘Is Canada really an education superpower? The impact of exclusions and non-response on results from PISA 2015’
• View Professor John Jerrim’s research profile
• Department of Social Science