Objective Structured Professional Assessments (OSPAs)

Project Summary

This web-page provides information about Educational Psychology OSPAs. The structure for these were developed in 2013-14 by a project team from the following universities: UCL (Prof Sandra Dunsmuir & Jane Lang), Southampton (Dr Sarah Wright) and Manchester (Dr Cathy Atkinson) funded by the Higher Education Academy (HEA). The aim was to develop an assessment protocol for Trainee Educational Psychologists to demonstrate their professional competence when working with young people aged 16-25 years. OSPAs have been embedded in EP training at UCL and Southampton since then and have been developed and strengthened over time. You will find information relating to the practicalities of preparing for and undertaking OSPAs as well as some useful references by clicking on the tabs below.

What is an OSPA?

An OSPA is an Objective Structured Professional Assessment similar to an OSCE (Objective Structured Clinical Examination), used to assess clinical skills in medicine. The purpose of an OSPA is to provide trainee educational psychologists with the opportunity to demonstrate the attainment of the knowledge, skills, behaviours and attitudes necessary for competent professional practice.

OSCEs were introduced into medical training in the 1970s. They have been described by the originators as a “timed examination in which medical students interact with a series of simulated patients, in stations that may involve history taking, physical examination, counselling or patient management” (Harden & Gleeson, 1979). They have become widely used within medicine and allied health professional training as well as other disciplines such as engineering. The format involves students rotating around a number of “stations” where they spend a specified amount of time carrying out a particular professional skill. At each station the student is presented with a task, which often involves responding to a short written scenario in the presence of a trained, simulated patient. Students' performance is assessed against a checklist of criteria related to the components of the task.

OSCEs are considered to increase fairness in assessment as they include a wider number of skills than written examinations, increase the number of examiners and are marked against explicit criteria which are standardised. Simulations can be used to assess procedural competence as well as knowledge.

An OSPA is...

Objective: at each station the trainee is presented with a different professional task presented as a scenario. All trainees are presented with the same scenarios and interact with an actor who has been provided with clear guidance on how to react and respond.

Structured: the assessment scheme for each station is structured. Specific competence modalities that are likely to be included are communication skills, perspective taking, information gathering and synthesis, management and professional integrity.

Professional Assessment: Each activity is a test of performance of professional skills. Trainees have to demonstrate competence in dealing with the scenarios that they are presented with.

Why use OSPAs in assessment of professional practice and performance?

Using OSPAs as a method of assessment allows careful specification of content and observation of a range of professionally authentic scenarios. The interaction between assessor and trainee is structured. An assessor is present at each station to monitor and score performance. The scoring is carried out according to pre-set criteria. Because trainees are being assessed using exactly the same scenarios and marked against clear and explicit criteria it increases the reliability of the observations and the consistency of the feedback. Assessment will be formative, defined by Cowie and Bell (1999) as “the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning”.

How are OSPAs to be carried out?

Each trainee educational psychologist engages with four standardised scenarios in turn, with an actor playing a role and with a standardised script. A different scenario is presented at each of the four stations:

1. Initial Consultation (history) 2. Assessment and explanation (data interpretation) 3. Action planning (psychological advice) 4. Communication and ethics

Scenarios are based around practice issues and tap the competencies required for practice with young people aged between 16-25 years. There is a 10 minute time limit for each scenario. An assessor (a university tutor) is present who rates the trainee's performance using a standardised marksheet. Assessors meet and review the scenarios beforehand and agree how the assessment system will operate. This process, known as calibration, is essentially a standard setting process, and is critical to the fair and consistent conduct of the assessment. All scenarios are video-taped. At the end of the 10 minute scenario, trainees are asked to make some comments on their self-assessed performance and provided with formative feedback in the form of strengths and improvement suggestions.

Reliability of OSPAs

There will be a subsequent assessor training day for local authority supervisors. Following a workshop on assessment of competence within an OSCE framework, supervisors will be invited to view a selection of OSPA videos and to provide a second rating of performance against the same criteria as the university assessors. This will provide valuable data that will be used to establish the reliability of the assessment system, across candidates, examiners and course centres.

Placement Assessment

OSPAs are designed to supplement learning and assessment on professional placements. Although placements are a costly element of professional psychology training (Gonsalvez & Milne, 2010), they are credible, straightforward, versatile and have ecological validity when judgements of competence are based on a range of observations over time. However, there is some evidence that placement supervisors demonstrate systematic halo and leniency biases (Gonsalvez & Freestone, 2007) which can reduce opportunities for trainees to receive the specificity of feedback necessary to develop professional competence. The addition of OSPAs ensures that trainees are assessed against authentic, standardised scenarios and receive consistent feedback across assessors and course centres.

References and further reading:

Cowie, B. & Bell, B. (1999). A model of formative assessment in science education. Assessment in Education, 6, 101-116.

Doran T. & O'Neill P. (2006). Core Clinical Skills for OSCEs in Medicine. Churchill Livingstone Elsevier.Gonsalvez, C. J., & Milne, D. L. (2010). Clinical supervisor training in Australia: A Review of current problems and possible solutions. Australian Psychologist, 45, 233-242.Gonsalvez, C. J., & Freestone, J. (2007). Field supervisors' assessments of trainee performance: Are they reliable and valid? Australian Psychologist, 42, 23-32.Harden, R. & Gleeson, FA. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 30 (1): 38-43.Nicol, M. & Freeth, D. (1998). Assessment of clinical skills: a new approach to an old problem. Nurse Education Today, 18, 601-609.Ranga, R. (Ed). (2005). OSCEs in Psychiatry. Royal College of Physicians.

Yap, K. Bearman, M. Thomas, N. & Hay, M. (2012). Clinical psychology students' experiences of a pilot objective structred clinical examination. Australian Psychologist, 47 165-173.

Guidance for Trainee EPs

Information and Guidance for Trainee Educational Psychologists about OSPAs

Objective Structured Professional Assessments (OSPAs) have been designed to support your learning and development of professional competence in training to be an educational psychologist. Structured simulated assessments are a tried and tested framework and will be used in a formative (not summative) way. They are designed to help you identify your strengths and areas where you would benefit from additional supervised experience. OSPAs are supplementary to existing course assessments of placement competency and performance on these will have no impact on current course assessment. You will be presented with scenarios at four different stations.

Preparing for the day:

You will receive the scenario outlines by email one working day before the OSPA Read though these There is no need to spend a lot of time preparing The resources that you have developed and accessed through participation in PBL will provide you with sufficient information to undertake the OSPA Your management of the scenario will be more natural and you will perform better if you don’t over prepare or try to adhere to a pre-determined script

Dress professionally Make sure you arrive in good time as the OSPAs operate to a very strict time schedule OSPAs can generate some anxiety, but be reassured that you have had all the preparation you need and the intention is not to catch you out but to support your learning in a meaningful and constructive way You will have exactly 10 minutes to interact with the role player (an actor) and will be given a two minute warning before the end After the OSPA is over, the assessor (a university tutor) will give you feedback in relation to strengths and improvement suggestion.

OSPA Video

To provide an idea of what it is like to participate in an OSPA, we have provided an example of an OSPA taking place. Here is the advice that trainees are given beforehand. The trainee EP in the video will have had this in mind when she undertook her online consultation.

Station 3 video: Action Planning

The aim is to agree clear actions and next steps with the client in relation to the issue presented. You should only discuss realistic and appropriate actions; do not assume unlimited resources. Establish that both yourself and the client wish to manage the same problem and there are no hidden agendas. You should allow the client to suggest solutions that they would favour and consider feasible. You should present clear and non-biased information about potential interventions and be prepared to justify these. Seek the client’s evaluation of the options you provide. If you don’t know key information, it is appropriate to let the client know that you need to find out more. At the end summarise agreed actions and check understanding. State how you will follow up and ensure the client has realistic expectations of the time frames.