Generative AI: Discussing proper use with students

MediaCentral Widget Placeholderhttps://mediacentral.ucl.ac.uk/Player/Hci58f84

Generative AI

Artificial Intelligence is rapidly changing the world, and large language models (such as ChatGPT) in particular have a large impact on academia. Students are exposed to very mixed messages. On one hand, academics praise the usefulness of their generative AI (GenAI) “writing partners,” while companies advertise GenAI services on popular platforms. On the other hand, students are warned that GenAI use may constitute plagiarism, that the systems are unreliable, and that they undermine students’ intellectual growth. 

I coordinate academic tutorials for first-year students of the Psychology and Language Sciences BSc and MSci programmes, and I felt we needed to go beyond UCL’s guidelines for students which are informative, but not enough to give students a clear idea on how to use GenAI in a meaningful way within their discipline. I collaborated with Dr Alice Liefgreen, a department colleague who has recently left academia. Our work started with an anonymous poll (link opens in Mentimeter) for our students in years 2-4 regarding their use of GenAI. We found that GenAI use was common, but not ubiquitous among students, and those using GenAI systems were very realistic about strengths and weaknesses. Poll results guided the design of tutorial materials. 

Need for student guidance

Alice and I shared an optimism for large language models but recognized the need for student guidance. We developed materials for a 90-minute tutorial session, briefed a group of tutorial group leaders on delivering the session, and each delivered the tutorial to a group of up to 10 students. Students then provided feedback on the session via Mentimeter.  

One important message we wanted to convey is that evaluating GenAI output takes expertise. Just as we are experts in how humans look and can detect unrealistic and strange anatomy in AI-generated images (we analysed examples in the session), experts in academic writing and specific subjects are much more likely to identify the flaws in AI-generated texts. Successful work with GenAI can save time and improve the quality of the work but requires skill. 

We spoke about how large language models are based on what word is statistically likely to follow, and their lack of cognition in the human sense. We then discussed the students’ experiences with GenAI and presented the poll results. We related the capabilities of AI to Bloom’s taxonomy and the demands of academic writing (both had been topics in previous sessions). 

The second half of the tutorial involved a practical exercise. Students had been asked to read an article in preparation for the session, and now used GenAI to answer questions about it. Questions were appropriate to different levels of Bloom’s taxonomy, starting with summaries and ending with reflections on methods and conclusions. Students evaluated the AI-generated answers and reflected on the limitations of GenAI. 

We introduced the concept of “zero-shot translation” (Mittelstadt et al., 2023; nature human behaviour), in which the user provides content to the GenAI, and the system then helps put it in the appropriate format. We suggest this as the optimal way to use these systems, consistent with what UCL classifies as Category 2 of GenAI use. 

Student feedback

Quantitative student feedback on the tutorial showed that a large group of students found it very useful and interesting, but a smaller number benefited less. We think this reflects students’ varying levels of experience with and reflection on GenAI systems prior to the tutorial, as students with lots of experience may have gained less from the session. Qualitative feedback was positive regarding the exercise, the information provided, and the positive attitude toward GenAI. There were negative comments about content which some students already knew, and one student suggested more guidance on effective GenAI prompting. We presented our work at a departmental meeting and received very positive feedback from our colleagues. 

Overall, feedback was very encouraging. I am currently refining the materials but will maintain the general structure and content. In contrast to some other topics, this session will require regular updates as GenAI systems continue to evolve.

References 

Mittelstadt, B., S. Wachter & C. Russell (2023) ‘To protect science, we must use LLMs as zero-shot translators’, Nature Human Behaviour 7. 1830-1832. 

Top Tips

GenAI is not to be avoided, but to be integrated meaningfully. Whether permitted or not in a given assessment, most students will use GenAI systems in their careers. Training on GenAI use is an important part of readying students for the future.