Duolingo English Test can predict international students’ first year academic performance

Students arriving to university with higher scores on the DET, used to assess their English language proficiency during university application, are more likely to achieve better grades in their first year at university than students with lower DET scores.

Whilst the scores can accurately predict non-Chinese students’ academic performance, the relationship was weaker for the larger cohort of Chinese students. The researchers note that this could be down to breadth of disciplines represented in their sample of Chinese students at UCL, who represented nearly three-quarters of participants in the study in terms of nationality.

The DET is a fully automated test, which means that it is both machine-delivered and scored. It was widely adopted during the COVID-19 pandemic following test-centre closures and other restrictions. Developed by Duolingo (but unrelated to their well-known language learning app), DET can be taken at any time and place for university applicants to prove their English language proficiency. The test continues to be commonly used for admissions testing, as well as the more established English proficiency tests used for that purpose such as IELTS and TOEFL iBT.

The research paper, published in Language Testing, is the first study that examines the relationship between DET scores and academic grades, in light of the steep rise in use of the test for university admissions purposes in the UK and at other universities around the world. Researchers also took into account students’ degree level, nationality and field of study in their analyses.

Lead author Dr Talia Isaacs said: “The rapid entrée of DET into the admissions testing arena meant that large-scale datasets allowing researchers to link language test scores to university grades have only recently become available. There is an urgent need for evidence supporting the high-stakes use of the test scores for university entrance purposes.

“Given its large student body and high proportion of international students, UCL was the ideal setting in which to conduct a study to begin to fill the evidence gap and answer the crucial question of whether students who score well on the test go on to perform well at university. Our study suggests that indeed the test does predict academic success, although there are some subtleties in the findings.”

The study, funded by Duolingo, involved researchers at IOE and the University of York, in collaboration with Julia Varga, Senior Admissions Officer at UCL’s Student and Registry Services.

The study team analysed correlations between English test scores and first year grades for a group of UCL students at the height of the pandemic (2020/21 academic year), including 1,881 who were admitted using the DET, compared to 2,651 first-year UCL students who had taken IELTS and 436 who took TOEFL iBT.

The researchers found that for all three language proficiency tests, higher scores predicted better academic performance. The team also found differences in academic scores based on the test taken, which varied depending on students’ proficiency level. For example, among graduate students at the most common proficiency level (C1), DET students’ grades were lower than IELTS students’ grades by 2% on average.

Results varied for other proficiency levels and tests. Such differences could be attributable to factors unrelated to the test, such as variations in admissions thresholds (the minimum scores needed to be eligible) between different programmes Another possibility is that academically weaker students may have applied to university later in the application cycle, when DET was the only test option due to Covid-19 pandemic restrictions.

Co-author Dr Ruolin Hu said: “These results reflect the unique circumstances that arose during the early stages of the COVID-19 pandemic, including sudden changes to university teaching, assessment and policy. Further research is needed to see whether our results hold in light of some recent changes to the test and also in non-pandemic situations.

“This study sets an important precedent and baseline for comparison. DET has garnered a lot of attention, the admissions testing and test preparation industry is rapidly changing, and we look forward to seeing what comes next.”

Links

• Lay summary of the research paper
• Read the paper in Language Testing
• Dr Talia Isaacs’ research profile
• Dr Ruolin Hu’s research profile
• Department of Culture, Communication and Media

Image

Credit: Kirsten Holst Photography.