Piloting Rank and Yank

UCL's new "Performance Enhancement Scheme"

Introduction | A race without a finish line | The others have performance problems, not me! | Pilot imposed | Ranking deferred | Summary | Email from Ian Jacobs to staff

In December 2010 Biomedical Faculty colleagues were emailed by Management about a new "Performance Enhancement Scheme" (PES) said to be a "pilot" to "continuously improve performance". 

Professor Ian Jacobs, Dean of FBS, said:

" "We want to develop a culture of continuous year on year performance enhancement - so that the overall performance in the Faculty demonstrably rises."

This sounds like a laudable aim. Who could possibly object?

1. Introduction

This type of 'Performance Management' scheme is not unique. Such schemes have variously been described as 'Forced Ranking', 'Forced Distribution' and (not publicly by HR) 'Rank and Yank' (RAY). Wikipedia has a good overview of this type of ranking method (with the much nicer descriptor Vitality Curve). 

The PES scheme seeks to implement the first component of a typical RAY scheme - quantification of performance - without any explicit ranking or yanking mechanism.

The key phrase - oft repeated by Dean Ian Jacobs - has far-reaching implications:

" A continuous improvement in the median performance [of staff].

We need to view the impact of the scheme year-on-year in the context of UCL competing for dwindling funds with other 'research intensive' universities, to grasp the impact of competition in an increasingly marketised sector. It becomes a charter for identifying 'failing' staff and dismissing them.

back to top

A race without a finish line: norm vs. criterion referenced schemes

Simply stated, RAY is a norm - as opposed to criterion - referenced measurement of performance.

A simple example of a criterion referenced performance measurement is the car driving test.

How does RAY differ? We measure the population under study relative to one another not against absolute criterion. We now have a distribution curve of individuals' performance.

There are many problems with this approach to performance management but we should be clear: Rank and Yank is intended to import the competition between organisations into the staff population within competing organisations and in so doing place staff under continuous pressure to increase work intensity, work longer hours and, at a University, to threaten academic freedom and shape research towards commercial goals.

The introduction of RAY is the starter pistol for a race without a finish line.

UCU opposes, on principle, the marketisation of education and health care (of which competition is a key element). Of course we can point out many other questions and objections:

• How will RAY impact team work and the free sharing of research results? Staff will quickly learn that as their performance is being measured relatively against their peers one obvious strategy to improve their own ranking is not to assist their peers in any way or even to sabotage their efforts. Observations of political manipulation, cronyism and cheating abound in organisations that have endured many iterations of rank and yank.
• Will it increase pressure on researchers to behave unethically?
• How do we select the criteria against which staff are measured?
• Can we really, even with the best of intentions, objectively measure the performance of a diverse population? The overall ratings arrived at in Rank and Yank schemes are at best an objective wrapper or laundering exercise for many subjective judgements.
• Rank and Yank schemes have a history of being a discriminator's charter.
• How will the performance of the lower ranking staff be addressed? Any supportive mechanism that UCL can use (training, rebalancing of research and teaching activities, mentoring etc.) will be replicated by competing organisations which leaves competing organisations with one option to gain a competitive advantage: fire the bottom ten percent (Imperial College went for twenty percent). This definitely raises the median performance of the remaining population! 
• Those that survive the exercise the first time round are now at risk of failure and will be highly motivated improve their standing relative to their peers (but they won't know how their peers are performing!)
• Will the performance of the highest ranking staff be rewarded? UCU's forerunner, the AUT, did a study of performance related discretionary spine-point bump awards; guess which demographic of the HE population benefited disproportionately? (answer: white, middle class, middle aged men.)
• What will be the likely psychological impact on staff? UCL are not planning to publish a league table that identifies individuals but at any point in which ranking is made explicit, each member of staff will be able to see their position relative to the overall population. This information is likely to have a strong, negative emotional impact on the relatively lower performing staff (objections about criteria and subjectivity notwithstanding).
• With very few iterations we will arrive at a population that still has a 'performance' distribution spread - and a bottom ten percent - but with no practical real-world improvement in quality.

back to top

It's the others who have the performance problems, not me!

Psychologists have studied our ability to evaluate our own abilities against those of others; this field of study has identified Attributional Bias as one of many Cognitive Biases. (For a popular exposition of 'Why clever people believe stupid things' read chapter thirteen of Ben Goldacre's book Bad Science.) Of particular interest to this discussion about RAY - especially the reaction of the UCL community to to the introduction of such a scheme - is 'Self-serving' Attributional Bias. See also Kruger, J., 'Lake Woebegon be Gone', Journal of Personality and Social Psychology (vol 77 (2) page 221-232).

" Abstract "Like the inhabitants of Garrison Keillor's (1985) fictional community of Lake Wobegon, most people appear to believe that their skills and abilities are above average. A series of studies illustrates one of the reasons why: When people compare themselves with their peers, they focus egocentrically on their own skills and insufficiently take into account the skills of the comparison group. This tendency engenders the oft-documented above-average effect in domains in which absolute skills tend to be high but produces a reliable below-average effect in domains in which absolute skills tend to be low (Studies 1 and 2). In Study 3, cognitive load exacerbated these biases, suggesting that people "anchor" on their assessment of their own abilities and insufficiently "adjust" to take into account the skills of the comparison group. These results suggest that the tendency to see oneself as above average may not be as ubiquitous as once thought."

Our tendency to rate our own skills as above average has important implications for the introduction of RAY performance management schemes: we believe we will be safe because others have the performance problems.

back to top

2. 'Pilot' imposed

UCU has formally objected to this scheme. Moreover we oppose its implementation without agreement. 

Instead of negotiating, Management have now decided to roll out what they refer to as a "pilot" in three Departments of Biomedicine.

This is the second so-called "pilot" which seems to be less about studying the scheme - there is no measure of 'success' - and more about attempting to create acceptance of the legitimacy of the procedure. 

The danger is that if it is seen as "successful" in Biomedicine it will be "rolled out" across UCL before long. 

back to top

Ranking deferred

Originally UCL proposed to explicitly rank staff relative to each other.

This would create an up-font competitive ranking of staff, a continuous pressure on staff to be seen to be the most individually productive, with the explicit threat that, should UCL need to identify staff for redundancy or capability dismissal, Management would already have drawn up a set of criteria for the purpose.

UCL has withdrawn the explicit ranking element for now. However as we have noted this element can easily be introduced at the point that UCL chooses to select staff for dismissal. 

back to top

Summary

What is wrong with the scheme?

This scheme is similar to the so-called 'academic criteria' drawn up to potentially identify staff for redundancy in Life Sciences in 2009-10. Unlike Life Sciences, which might otherwise be seen as a one-off, the PRAY scheme would embody ongoing targets, year on year.

How does the scheme measure academic performance?

The scheme is premised on a definition of 'performance' based purely on gathering data that is relatively simple to obtain but highly misleading about performance. 

It is proposed to use student feedback data to assess 'teaching performance'. However this data is qualitative and subjective, and not reliably quantitative and meaningfully useful for comparison. Staff teach different subjects and different groups of students, after all. 

Academic research involves shared output (cooperation), outputs vary over time (research life cycles), and their value is notoriously difficult to contemporaneously measure (relying on peer review). Compared with the RAE and REF (where the objections listed above also apply), the PRAY scheme is still more methodologically unsound. RAE/REF evaluations are carried out every 5 years; PRAY would be annual. 

The only saving grace of (inaccurate) individual 'performance' measures in the RAE is that results are aggregated between individuals (and so tend to a more correct overall measure). Management ignore even this elementary statistical principle in their concern to impose a performance scheme.

What are the other problems with the scheme?

Ranking can be reintroduced.

As long as the scoring component is present, the resulting data can be easily used to produce a ranking which could be used by managers to make overly subjective, unfair and possibly discriminatory comparisons between staff members. The scoring serves no purpose other than to allow comparisons to be made between staff.

The existing performance scheme, appraisal, is not 'broken'.

UCL has not yet provided a satisfactory answer to our question of what is lacking in the present appraisal scheme such that it needs to be modified in this way. There are some problems in its consistent application but not in terms of the criteria used. The new scheme will not fix this.

Dubious 'performance' measures are already being misused. 

We know from our work in representing and supporting members in 'capability' meetings and in particular from the recent experience in FLS, that performance indicators are already being misused to pressure individuals into accepting severance packages, or transfer to Teaching Fellow roles on inferior terms and conditions.

Misuse of personal data.

Although some of the relevant data are indeed already gathered, for e.g., appraisal purposes, or departmental performance statistics, and can in principle be used formatively, it is an abuse of personal data under the Data Protection Act for the same data to be used for other purposes potentially prejudicial to staff without prior agreement.

Staff-student relationships. 

Some of the performance indicators proposed are highly problematic and would rightly be seen as threatening and divisive to staff: We are not aware of the existence of any 360 degree appraisals of teaching incorporating student feedback. 

While student feedback is acceptable to education professionals for formative feedback for an individual, it is notoriously subjective, and therefore unreliable and unfair as a quantitative indicator of performance between individuals. At present, UCL uses Peer Observation of Teaching (POT) as a developmental tool, which is perfectly acceptable to UCU. Using POT for performance management however would raise many problems about subjectivity, favouritism and be hugely open to abuse.

No cost-benefit analysis. 

We have seen no cost-benefit analysis of the overall scheme, but it seems clear that this process entails additional burdens on staff time. If any piloting has so far taken place, we need to see the details of what the pilot study revealed. What were the detailed research questions? What were the results and the conclusions? Was there a control group, i.e. what was the pilot compared against? We need to see the raw data. 

If the UCL community is to invest in such a scheme we naturally expect a comparable rigour as would befit any other serious investment in time and resources.

The problem remains

Ultimately a university is not well served by a scheme which provides greater disincentives for collaboration and cooperation than exist to-date. One cannot then address a fundamental problem of competition by tacking on a "collaboration factor", when the overall scheme mitigates against it. 

For UCU to enter into meaningful negotiation on any performance management scheme therefore, all of the above points would need to be addressed. We do not, unfortunately, believe that this is the case at present. 

There are other empirical reasons why we believe a formal performance management scheme is inappropriate and will be counter-productive at UCL. Where the agreed existing academic appraisal process is operated, we find that it is applied inconsistently across the College at best, with many staff not having had appraisals for many years. Others report having the interview in very rushed or otherwise unsatisfactory circumstances as an adjunct to another activity (such as during the manager's coffee break and the like). 

What we propose therefore is a joint review of the current appraisal system, both in terms of its content and its application in practice. This would require prior consultation with UCU members to gather more systematic data on the present use (and abuse) of the scheme. 

If any malpractice or deficiencies -either from a management or staff perspective - are identified, these could then be addressed in a collegial spirit rather than an adversarial one. Any outstanding issues over the recent Professorial Pay Review could be brought into this process. The whole exercise would allow for a fresh start, and a rebuilding of the trust lost during the recent dispute over FLS redundancies.

Join your union

As it stands, the "Performance Enhancement Scheme" is a clear attack on academic freedom. In research, it creates incentives for conservative research programmes and income generation and undermines academic freedom of inquiry. In teaching it prioritises 'crowd pleasers' over difficult subjects, and further encourages the 'dumbing down' of academic life. 

The ultimate outcome of this type of scheme is to reduce previously-secure lecturing staff to the position of 'casual' (insecure) research staff, dependent on bringing in grants for a post. This move represents an important step in the casualisation of staff.

We can only defeat this type of scheme if we staff stand together collectively against the imposition of rat-race competition between staff. Colleagues need to think seriously about joining their union.

UCU is UCL's recognised trade union for academic and academic-related staff. If you are not already a member, please join TODAY. If you are a member, join the campaign!

back to top

Email from Ian Jacobs (Dean, FLS)

Re: Introduction to the pilot phase of the FBS 'Performance Enhancement Scheme' (PES) 

During the next 2 months the Faculty will be piloting (in 3 Divisions/Institutes) a new 'Performance Enhancement Scheme' (PES) which has been developed over the last year. The aim is to provide, in as objective, transparent and accurate way as possible, information about performance in research, education, knowledge transfer and enabling which can inform discussion during the academic appraisal process. PES is intended to assist the efforts of all of us to contribute to a dynamic Faculty, which is continually striving for overall performance improvement in all areas of our activity. We are unquestionably one of the world's leading biomedical centres but if we are to continue to rate with the very best internationally there is no room for complacency.

In developing PES, the Faculty Executive Team have sought extensive advice within and outside UCL. After a year of preparation and an initial small scale 'pre-pilot' we now have a scheme, which is ready for a pilot phase. No scheme of this type can be perfect but the Executive Team have done their best to develop an optimal scheme and it will be further refined on the basis of the pilot experience. We want to develop a culture of continuous year on year performance enhancement − so that the overall performance in the Faculty demonstrably rises.

Heads of Division/Institute have been kept up to date as the scheme has evolved. To ensure that all academic staff are fully informed, there will be a thorough communication process about PES including a series of open meetings, once we have piloted the scheme. In the meantime a few key points:

• PES includes activity in research, education, knowledge transfer and enabling to ensure that all components of UCL's academic mission are fully represented.

• For each academic, PES will reflect the proportion of research, education, knowledge transfer and enabling they undertake PES will also be adjusted to take account of part time vs full time academic work.

• The team developing PES has been particularly conscious of the importance of both minimising the burden that PES adds to busy working lives and avoiding duplication. For some aspects of PES almost all of the data will be downloaded centrally from existing sources with individuals having an opportunity to double check and refine.

• We want to be sure that PES works in the way intended and are therefore grateful to Deborah Gill for agreeing to lead on an evaluation process which will commence with the pilot and occur in parallel with the roll out of the scheme.

Professor Ian Jacobs

The pilot phase will occur during the next 2 months in the Institute for Women's Health, Division of Infection & Immunity and the Ear Institute. A lot more information will be provided before PES is introduced across the Faculty − so please regard this as an introductory note.

Your help in due course in making a success of this scheme will be enormously appreciated.

Best wishes,

Ian

Professor Ian Jacobs

Dean Faculty of Biomedical Sciences

back to top