UCL Study Finds Research Quality Insufficient to Explain Citations in SDG Policy

This suggests that research quality alone is not a strong predictor of policy impact; instead, non-academic factors appear to play a more significant role.

The paper analysed over 16 million research articles related to the United Nations Sustainable Development Goals (SDGs) and their citations across 2 million policy documents. The study assessed the effect of conventional bibliometric indicators of research quality, including academic citations, journal impact factor, and author h-index. It found that while these metrics had only a modest relationship in explaining how a paper is used in policy, overall, academic citations were the most significant predictor of whether scholarly articles are cited in policy documents across SDGs. This suggests that policymakers focus on the quality and relevance of the research content over the prestige of the publishing journals or the reputation of the authors.

Commenting on the implications of this study for governments, Sir Geoff Mulgan, Professor of Collective Intelligence, Public Policy, and Social Innovation at UCL’s Department for Science, Technology, Engineering, and Public Policy (STEaPP), said, "By combining global policy and research data, we can identify whether governments are using the most relevant and high-quality evidence to guide their decisions at an unprecedented scale.” 

“What we see in this study is that governments often face challenges in effectively using the best available evidence for policymaking, highlighting an opportunity to better support decision-makers by leveraging AI-driven tools and data analysis techniques," he added.

Discussing the implications for researchers at UCL and beyond, Licia Capra, Professor of Pervasive Computing at UCL Computer Science said, "conventional measures of academic success, such as publishing in high-impact journals or accumulating academic citations, do not necessarily mean that research will be used in policy”, adding that “researchers who want to influence policy should ensure that their work is not only addressing an academic knowledge gap but is also relevant to current policy issues and accessible to policy makers outside of academic circles." 

One approach for increasing the likelihood of policy impact is by communicating research results through media channels. Across all SDGs, media mentions improved the model fit by 14%, meaning that media coverage, or the lack of it, plays an important factor in explaining why some research is cited in policy or not. This effect is particularly pronounced for research related to SDG 13: 'Climate Action' and SDG 9: 'Industry, Innovation & Infrastructure', where the media is as likely as research quality to explain policy impact.

As one of the first and largest studies of its kind, this research provides empirical evidence supporting the long-held belief that there is a disconnect between science and policy. The disparities observed between different SDGs highlight the need for tailored approaches that account for the unique nuances of each field.