How to move simulated case vivas online
Yuan-Ling Ng, UCL Eastman Dental Institute/Medical Science, shares how they overcame the challenge of assessing students’ clinical diagnostic and treatment planning skills online during Covid-19.
10 July 2020
Yuan-Ling Ng, Programme Director, Endodontology (UCL Eastman Dental Institute/Medical Science), describes how Endodontics students’ clinical diagnostic and treatment planning skills were assessed online using BlackBoard Collaborate and Moodle.
Using vivas to test clinical diagnostic and treatment planning skills
Students on Clinical Postgraduate Taught Programmes (MSc or MClinDent) are assessed on cognitive aspects of the clinical diagnosis process, treatment options appraisal, and treatment planning skills; as well as on their practical insight of the related treatment procedures.
Ordinarily, we would assess students’ clinical diagnostic and treatment planning skills using actual clinical cases, in person. We deliberately omit some critical information from the patient history and examination findings in the case summary from which the students synthesise a picture of the case, so as to assess their ability to identify missing critical data; our explorations can then focus on how they might acquire the missing information to aid their diagnosis and treatment planning.
In these extraordinary times, we were forced to rethink our approach as face-to-face assessments were prohibited during the Covid-19 lockdown; it wasn't possible to assess communication skills in the face-to-face interaction that would normally occur with patients.
We set about finding a means of online assessment of simulated clinical cases, in place of our real clinical cases - one which allowed us to assess students’ ability to critically weigh available, and relayed information so as to guide the patient to provide the relevant facts for a diagnosis.
In coming up with our approach to online assessment of clinical skills, we were mindful of several challenges we needed to overcome:
1. We needed to ensure the assessment remit and standard are fulfilled, but we also wanted to streamline the process as per face-to-face examination format:
candidate to view the simulated case - a summary of presenting conditions plus clinical and radiographic images - under invigilation;
examiners signing into the session to start the viva promptly; and
sufficient time for the invigilator to start the case viewing by the next candidate.
Solution: The functions of the Blackboard Collaborate - UCL's online conferencing and online classroom platform - met with our needs and we conducted repeated trials of our process and technology to ensure it was streamlined.
We included a “transition time” in the examination schedule so examiners and invigilators were given enough time to move between candidates.
2. We wanted to ensure the simulated case examination material can be uploaded by the administrator to all candidate sessions in advance, without leaking to the candidates.
Solution: We used the “Access Restriction” function available in Moodle to limit access by students until the appropriate time.
3. We had to ensure pre-agreed and consented video-recording could be saved securely in Moodle for internal/external examiners and quality assurance purposes, without giving access to all candidates.
Solution: The format was piloted to ensure there was sufficient bandwidth and that audio and video devices were compatible with the platform.
We used the “Access Restriction” function again by changing the access restriction from “individual student” to “examiner group” immediately after each examination.
Other platforms not integrated with Moodle, e.g. MS Teams, would require the administrator to immediately download and delete the video from the platform to prevent candidate access. This process requires some time and consideration and it may require an additional staff member to assist the exam process.
To ensure no illegal intrusion to the sessions, this approach only allows candidates to access the session via UCL Moodle, and the examiners were given the link via their UCL e-mail address.
Our solution allowed maintenance of the examination remit and standard of assessment, despite a change of format and approach, and the Blackboard collaborate meeting platform is available within Moodle, which ensures secure storage of examination materials.
This approach was a first for the UCL Eastman Dental Institute. I got the inspiration after reviewing the functionality of the online meeting platform, Blackboard Collaborate, during our Departmental Teaching Committee meetings. I gained experience by chairing virtual seminars with students via Skype for face-to-face discussion, and using the MS Teams “class notebook” function for summarising the discussion.
Timeline and resourcing
The project started in April 2020 after the Covid-19 assessment amendments were approved by UCL. We carried out the first test and mock Vivas on 30 April 2020.
We settled on the final protocol for set-up, after multiple testing rounds, on 20 May 2020 - the actual examinations were successfully completed on the 2nd June 2020.
We made refinements to the set-up to improve security and streamline the transition from candidate to candidate, as well as candidates’ viewing of case material.
I led the project with support by the Deputy Programme Director and Programme Administrator to complete initial trial.
Three internal examiners were involved in two mock Vivas per candidate (n=13) and provided feedback for refining the set-up.
The total resourcing can be estimated to 130 hours before the process was implemented for the summative assessment.
Staff and student response
The staff and students were well prepared for the actual online examinations. We were able to test the students’ clinical skills without being affected by the lack of face-to-face contact and IT issues.
We sought feedback from both staff and students with a view to enhance their experience.
The feedback was positive and constructive:
“Just wished to give feedback regarding the mock viva set up today- I thought it worked well and had no issues viewing the case. I think the format on Blackboard is very good. - Student
“Learning via the blackboard collaborative has been on par with face-to-face teaching for seminars and case-based discussions. Furthermore, the mock viva sessions were important to highlight and address any technical issues before the exam. Preparing for and undertaking the unseen clinical case viva has helped me to enhance my diagnostic skills. The process ensures that you evaluate the bigger picture before focussing on the individual tooth-specific problems. It also helps to improve decision-making skills and enhances the application of knowledge so that it is tailored to the particular case / problem at hand. Furthermore, the process of speaking out loud when practicing in a group over blackboard sessions has helped to improve confidence for the actual examination and has highlighted deficiencies which can subsequently be improved upon”. - Student, Dipita Moorjani, BDS MFDS Edin. (MClinDent Endodontics)
“Blackboard Collaborative has facilitated our learning greatly at a time when face-to-face teaching is not possible. It is an excellent platform which has enabled case-based discussions, seminars and meetings to be held without any interruption or technical difficulty. The recent oral exam went very smoothly through Blackboard Collaborative and was no different to a face-to-face viva. Two mock exams were also undertaken through this digital platform, providing the opportunity to simulate the exam conditions and to practice the viva technique, which greatly improved my confidence. Overall, I felt that this form of learning and examination has been highly beneficial! – Student, Maya Karnad BDS MJDF (Eng) (MSc Endodontics)
Staff gave feedback and suggestions via online meeting after each mock viva session. Their suggestions have included a time gap between each candidate to account for any delay in signing into the session, to permit a transition of examiners from one candidate to another.
“This project was a good example of how embracing digital education methods can make university teaching more versatile, inclusive and overcome challenges imposed by different timetables and limited availability of students and staff. The regular testing of the Blackboard Collaborate fulfilled a dual purpose: a) maintaining student engagement during this period, giving them the chance to apply their learnt knowledge to clinical scenarios and test their diagnostic and treatment planning skills; and b) allowing students and staff to become familiar with the online format and be ready to solve practical issues that may emerge. Personally, I enjoyed being part of this effort and explore the potential of digital platforms like Moodle. I think the format can be adopted well for other activities of our programmes such as the evening case-based discussions or for more personalised teaching for students that need further support. - George Milesis, Clinical Lecturer in Endodontology, Deputy Director of MSc/MClinDent Programmes in Endodontics at UCL Eastman Dental Institute
“Using Blackboard Collaborate was very positive experience overall. It was straight-forward to set-up and run, which facilitated multiple sessions to be held online. The practice sessions allowed streamlining of the process and for any IT issues to be resolved. In addition, this has permitted closer contact with the students and enabled further teaching to be completed as required. I look forward to using it more in future, as I believe it will open up even more teaching opportunities. - Elizabeth Hoy, Speciality Doctor in Endodontics
Having fully tested this approach, we intend to adopt it for online teaching and assessment next year, to overcome any geographic barriers or lack of suitable physical venues.
Yuan-Ling Ng’s top tips for developing alternative online assessments:
To mentally visualise how you wish to run the assessments to fulfil the objectives and note all the potential risks that may affect the smooth running, and security of the exam materials if they are run online.
To gain knowledge of the functionalities of the selected online platform via their “help” page and gauge whether they suit the needs for the online assessment.
Be bold to test all the functions and work with a colleague or a senior student to test the function from both an examiner and candidate perspective.
To carry out more than one trial with different groups of students and examiners with the support from administrator and colleague to ensure everybody are comfortable with the use of online assessment. The first trial aims to allow individuals to be familiar with the process, functionality of the platform and to identify any IT issues that needs to be resolved. The second trial aims to allow individuals to become comfortable in questioning and giving answers via the online video platform and also to be ready to calmly deal with any IT issues during the process without affecting performance.
To seek feedback after each trial to inform gradual refinement of protocol and development of Standard Operating Procedures in event of any IT issues during the actual examination.
Image credit: Christina Wocintechchat / Unsplash