A modified Angoff B screening method for collusion detection in examinations delivered on the MSCAA and Speedwell platforms

The occurrence of a pandemic led to remote un-invigilated examinations being used, particularly in the early years of medicine programmes. While students were expected to work independently, examination and social pressures can drive students to collude. Here, we describe the practical implementation of an Angoff B technique1,2, with modifications (mAngoffB), for collusion testing, supplemented by secondary measures of ‘simultaneous identical action’ using ‘Data Audit’ analysis on the MSCAA platform and Speedwell. mAngoffB is a measure of the proportion of identically wrong answers between pairs of students. Data was assessed using either Excel or R (https://www.r-project.org/). In the question papers with 100 or more MCQs, the cut-score for collusion referral was set at 6 standard deviations above the cohort mAngoffB mean; across all years, involving approximately 2000 students in 2020-21, this was 0.60±0.06 (mean±SD; n=8 examinations). This resulted in the referral of 85 students from all years (4%) through our standard academic integrity processes, with a higher proportion in the early years of the programme. Most, either spontaneously or when presented with relevant data, admitted their collusion at or on the day of their first academic integrity meeting; the rest admitted collusion during misconduct committee investigations or appeals. On the basis of confessed collusion, this process therefore has a very low false positive rate, although the false negative rate is likely to be high and difficult to determine. Timing methods are readily applied on examination platforms where delivery audit (‘clickstream’) data is available, with applications that extend beyond remote un-invigilated examinations.

References

1 Angoff WH. The development of statistical indices for detecting cheaters. J Am Stat Assoc. 1974; 69:44–9. https://doi.org/10.2307/2285498

2 Wollack JA. Simultaneous use of multiple answer copying indexes to improve detection rates. Appl Meas Educ. 2006; 19:265-88. https://doi.org/10.1207/s15324818ame1904_3