The objective structured public health examination: A study of reliability using multi-level analysis

Background: Introduced in 2006, the objective structured public health examination (OSPHE) is a part of the examination for the UK Membership of the Faculty of Public Health. Designed to simulate real work in public health, the same five generic competencies were assessed simultaneously in each of six stations in 1 h 48 min. This study estimates the examination reliability of the OSPHE. Methods: Data from all 198 candidates, 54 questions, 39 examiners over 9 test days in 2006 were analysed for reliability and sources of error using multilevel cross-classified analysis and generalisability theory. Results: A reliability co-efficient of 0.85 was estimated (95% confidence interval 0.82-0.88). Little variance was accounted for by questions or examiners. Twelve questions would increase reliability to 0.92. More than one examiner per question has negligible impact on reliability. Conclusion: Multilevel modelling estimates examination reliability which is more rigorous than Cronbach's alpha. Traditional generalisability methods based on the analysis of variance (ANOVA) are unable to do this. The reliability of this examination compares favourably with the literature despite having just six test stations and under 2 h per candidate. Other specialties could consider the use of parallel testing of the same key competencies at all stations within simulated workplace events.