The Modified 5-Year-Olds’ Index – testing reliability and the impact of training level

Abstract

Aim: To test the reliability of the Modified 5-Year-Olds’ Index using assessors with a range of experience, and to determine whether calibration improves reliability.
Design: Prospective reliability study.
Setting: Bristol Dental School, University of Bristol.
Methods: Fifty study models of non-syndromic 5-year-olds with a repaired UCLP were selected from the CCUK archives. Fifteen participants with a range of clinical experience in orthodontics were divided into three groups of equal experience. Each group was given differing amounts of information: Group 1 information sheet; Group 2 information sheet and reference models; Group 3 calibration course, information sheet and reference models. Each participant scored the 50 models using the Modified 5-Year-Olds’ Index on two occasions at least four weeks apart. ICCs calculated from a two-way random effects model were used to calculate intra-rater reliability and inter-rater reliability comparing assessors’ scoring to the experts’ consensus scores (gold standard).
Results: Group 2 (ICC 0.80 – 0.93) and Group 3 (ICC 0.80 – 0.91) demonstrated high levels of intra-rater agreement, with lower levels shown by Group 1 (0.68 – 0.93). Inter-rater agreement was high in Group 2 (ICC 0.87 – 0.93) and Group 3 (0.82 – 0.91), with Group 1 showing the lowest levels of agreement (ICC 0.69 – 0.94). The level of training of the assessors in Group 1 appears to influence reliability scores, with high intra- and inter-rater reliability scores of the consultant and post-CCST trainee at a similar level to those of the same grade in Groups 2 and 3.
Conclusions: The Modified 5-Year-Olds’ Index is a reliable method of assessing outcomes when model scoring is carried out by consultants or post-CCST trainees. Calibration in use of the Index does not improve reliability.

Date of Award	26 Nov 2020
Original language	English
Awarding Institution	University of Bristol
Supervisor	Tony Ireland (Supervisor), Sam D Leary (Supervisor), Jonathan Sandy (Supervisor) & Nicola Atack (Supervisor)