Vice-Provost’s View: Global league tables and prospects for UCL research

Since 2014 there have been at least half a dozen global rankings tables published each year, which, in addition to that from the THE, include those published by QS, URAP, ARWU and NTU. Each of these league tables have their own idiosyncratic combination of metrics, which claim to reflect the factors which are the most important characteristics of a university. Some rely on qualitative, subjective views from around the world; others give weighting to the number of Nobel Prize winners, or the citizen mix of the academic community; while yet others focus on quantitative (albeit no less controversial) metrics like the number of papers published, citations found or research degrees awarded.

A number of these tables have a degree of year-on-year volatility that often triggers headlines about the “decline of the west”, or the rise of China’s universities. Given the shortcomings of any one of these tables, it is surprising, and perhaps concerning, that some institutions, government agencies and politicians seem to use their outcomes to determine future policy, funding and even national infrastructure investment.

Robustness, volatility and misuse

The league table industry has been criticised as being unscientific, contradicting the responsible use of metrics embedded in DORA and of attempting to measure the unmeasurable. Work by Elizabeth Gadd, Richard Holmes and others has highlighted some of issues with league tables. Holmes recently wrote:

“A pilot study, Rating the Rankers, conducted by the International Network of Research Management Systems (INORMS), has found that on four significant dimensions, transparency, governance, measuring what matters, and rigour, the performance of six well known rankings is variable and that of the big three is generally unimpressive.”

Furthermore, small, high-quality universities often miss out in these rankings, because some are biased in favour of scale, while others are biased towards technical, biomedical or professional school metrics. Whatever their methodology, such league tables can only ever capture a subset of information about some aspects of university activity.

Certainly, when league tables are used by institutional management or governments to make decisions that significantly affect the livelihoods of individuals, they are being abused, and given more weight and significance than they deserve.

Alternative measures

At the highest level of granularity, these league tables do contain elements of data that reflect the impact that a university has in the world of research and innovation. However, one of the troubles is that volatility and subjectivity make it impossible to infer meaningful year-on-year trends from any one listing.

As suggested above, this volatility in the year-on-year outcomes may be an essential feature of some tables, as they can be used to make headlines, sell adverts or drive up subscriptions. In reality, the nature of the university is one of long-term substantive evolution, rather than rapid change, much to the frustration of many a commentator, politician or even some university leaders.

One way to see through such ‘noise’ however, is to look at the average or an aggregate ranking that universities get in any one year from a basket of individual tables, and then to measure changes or trends over time. This approach was used by Richard Holmes in 2009, and was called Global Ranking of Academic Performance (GRAPE) – he states he chose not to call it the Comparative Ranking of Academic Performance!

With the passing of time, a set of longitudinal ‘data’ from the ranking organisations is now available, and so we in UCL Research, Innovation & Global Engagement analyse these trends over time. Such an analysis is shown here, where the ranked position is based on the average ranking of the five tables mentioned at the beginning of this article, plotted over the most recent eight-year period. The number quoted after each institution is its average ranking in 2021.

As might be expected, the year-on-year fluctuations are generally small, with very little relative movement amongst the top few universities over the past five years or more, which shoots the fox of many a headline-seeking sub-editor. One take-home message from this list, however, is that ‘money talks’. The majority of these top universities are characterised by having considerable private endowments or medieval inheritances. Those that do not are either benefiting from huge state injections of funding or dependent on income from the fees from large numbers of students.

Although following the waxing and waning of league table rankings might be an amusing pastime for some, it is far better for an institution to assess progress in terms of its own objectives over the longer term (the Netherlands provides an interesting example of this approach).

I have always held that UCL should measure our research success by our outputs and impact, rather than by the inputs of funding that we attract. Obviously some researchers need direct project grant income to carry out research and achieve impact, but in a multi-faculty, comprehensive university many researchers can achieve impact without specific grant support, as long as the university has excellent library and infrastructure facilities. (As an aside, this is why QR income is so important for universities in the UK.) As such, I have always favoured a metric which measures the impact of outputs generated by UCL, for example a five-year-rolling citation count, as shown below:

On this criterion, UCL seems to be growing in impact over the past decade. Research outputs from the university (2016–2020) generate the 4th-highest number of citations globally. This is up from 10th place six years ago, putting us ahead of Oxford this year (Essential Science Indicators analysis of five-year-rolling citation counts). This relates in part to growth of the institution, and the number of publications may not be a reflection of their relevance or usefulness. As such, an analysis of the most highly cited papers is useful, as shown below with a plot of the ranking based on the number of outputs with the top 1% of citations for the last 15 years.

UCL’s own analysis of publication metrics shows that UCL is now ranked 3rd globally for number of papers in the top 1% by citations, behind only Harvard and Stanford (InCites, 2017–2020). On the basis of these simple plots, it seems that UCL starts the 2021-22 year in a strong position, although the challenges we face in the future are all too obvious.

What matters most

I agree with the assertion that it is not possible or useful to reduce a multidimensional, complex body of a university to a single number. And, of course, capturing the full outputs and impact of a university – not least in terms of graduates and societal impacts – is far more complex than any one measure or ranking can do justice to.

All of the universities noted here, and many others, are truly magnificent institutions, generating brilliant graduates, making breath-taking discoveries, and contributing significant advances for society locally, nationally and globally.

The world is fortunate to have such institutions, but their key role today is to help provide the solutions to the existential global challenges that we all now face – a contribution which is not easily captured in a league table.

Whether global society reaches the 22^nd century in a stable and liveable state will greatly depend on the contribution and impact that these universities make to the world in the next few years, not the league tables that provide fleeting headlines. It is time for their excellence to deliver meaningful solutions to help us, and future generations, to survive and thrive.

A promising future

As indicated above, we should place high-quality outputs and impact as our top objective, but as mentioned above in many disciplines these require the attraction of investment. In that context, I am delighted to report that UCL research has also continued to thrive during COVID-19, with researchers winning an all-time record level of funding in the last year.

• In 2020-21, UCL researchers were awarded £573m in new research grants, the highest level ever achieved by the university. This is up £50m on the previous year’s £523m, which itself was also an all-time record.
• In the last 12 months, UCL researchers submitted 4,223 research applications. In that same period, 1,524 applications were successful, receiving an average award value of £375,895.
• The total award value was up 9.4% on 2019-20, showing that COVID-19, despite its horrendous impact on individuals and society, has had no obvious impact on UCL’s research applications or funding levels. The historical average increase in value year on year has been 4.8%, so this year's figure of 9.4% is particularly significant.

These figures highlight the continued commitment and achievement of UCL researchers, reflecting the brilliance of our community’s research proposals and the unwavering support provided by professional services colleagues in the last, very challenging, year. I recognise that the figures above, while welcome, do not reflect the unequal burden that the pandemic has presented to certain groups within our research community – I wrote last year about some of the steps taken to address this, and we will continue to better understand and respond to this issue.

Despite the turbulent conditions, the unstable funding environment, and the effects of the pandemic on us all, UCL research has continued to thrive – and indeed has been directly addressing the challenges of the pandemic. We are continuing to create research that benefits society and addresses some of its most pressing problems. Our challenge is to ensure we continue to do so in what will be a globally turbulent future.

Professor David Price, UCL Vice-Provost (Research, Innovation & Global Engagement)