Government’s education statistics “seriously flawed”

The government rely on the Organisation for Economic Co-operation and Development (OECD) PISA data which tests 15-year-olds in reading, science and mathematics in around 80 countries to see how pupils in the UK rank internationally.  

PISA results have consistently driven changes to schooling systems across the globe and in the UK. PISA has also become the main resource to compare outcomes across its four devolved nations, making it the only cross-national assessment, measured on a regular basis.

The study, published today online and in the forthcoming edition of the Review of Education, analyses results from the OECD PISA study and highlights serious concerns about the 2018 edition data for England, Wales, Northern Ireland and Scotland.

The peer-reviewed research also finds that key limitations with the data have not been transparently reported – particularly by the Scottish Government and Professor Jerrim is now calling for the Office of Statistics Regulation to conduct an urgent review of how this key piece of education data is reported.

In addition, the study finds that low levels of participation in countries such as England and Scotland may have resulted in potential biases in the PISA data with low achieving pupils being underrepresented. In the case of Wales, he finds that PISA mathematics scores could be inflated by as much as 15 points. This could lead Wales to fall seven places in the world rankings; significantly below the international average, taking Wales on a par with countries such as Malta and Belarus.

Report author, Professor Jerrim (UCL Social Research Institute, part of UCL Institute of Education) said: “PISA is meant to be a representative study of 15-year-olds across the UK. But there are serious flaws with some children being excluded from the study, schools being unwilling to participate, and some pupils not showing up for the test.

“In England and Wales, there is clear evidence that some lower-achieving pupils have been systematically excluded. While what has happened in Scotland is, frankly, a bit of a mess.”

Other key findings from the paper include:

• In the UK, around 40% of students are not included in the PISA data. This is amongst the highest anywhere in the world.
• Scotland changed the PISA test date in 2018 to later in the school year. This meant that half the sample were in a later school year (S5) than in previous editions of the test. Yet this was not commented upon by either the Scottish government or by the OECD.
• A number of other anomalies emerge with the Scottish data – such as a very high number of “ineligible” pupils – which the Scottish Government have not adequately explained or discussed.
• In England and Northern Ireland, bias analyses were produced but not published. These were obtained by Professor Jerrim via freedom of information request and revealed – in the case of England – schools with lower historical GCSE grades were more likely to refuse to participate in the study.  
• The PISA sample for Wales seems to systematically underrepresent those who achieve low GCSE grades. For instance, according to official government data, 41% of young people in Wales failed to achieve a GCSE C grade in mathematics in the 2018/19 academic year.  Yet, the figure for the PISA 2018 cohort was just 31%.

One limitation of the paper is that it has focused upon the PISA 2018 data for the UK, and not how similar issues may have affected other countries and previous PISA rounds. Other countries, particularly those with low overall participation rates, accompanying the UK such as Canada, Sweden, New Zealand, Portugal and Hong Kong may also be experiencing similar issues.

Professor Jerrim added: “The most pressing issue is for the Office for Statistics Regulation to conduct an independent review of the UK’s PISA data. Clear guidelines need to be put in place to ensure more transparent reporting in the future.

“Crucially, at an international level, the OECD needs to reconsider its technical standards, the strictness of which these are applied, and its data adjudication processes. The processes currently in place are nowhere near robust enough to support the OECD’s claims that PISA provides truly representative and cross-nationally comparable data.”

Links

• Professor John Jerrim's academic profile
• UCL Institute of Education
• Paper in full

Image

Credit: F1Digitals/111 images Source: Pixabay CC 2.0 

Media contact

Rowan Walker

Tel: +44 (0)20 3108 8515

Email: rowan.walker@ucl.ac.uk