Machine learning and deep learning systems for automated measurement of ‘advanced’ theory of mind: reliability and validity in children and adolescents

R.T. Devine*, Venelin Kovatchev, Imogen Grumley Traynor, Phillip Smith, Mark Lee

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

144 Downloads (Pure)

Abstract

Understanding individual differences in theory of mind (the ability to attribute mental states to others) in middle childhood and adolescence hinges on the availability of robust and scalable measures. Open-ended response tasks yield valid indicators of theory of mind but are labor intensive and difficult to compare across studies. We examined the reliability and validity of new machine learning and deep learning neural network automated scoring systems for measuring theory of mind in children and adolescents. Two large samples of British children and adolescents aged between 7 and 13 years (Sample 1: N = 1135, Mage = 10.22 years, SD = 1.45; Sample 2: N = 1020, Mage = 10.36 years, SD = 1.27) completed the Silent Film and Strange Stories tasks. Teachers rated Sample 2 children’s social competence with peers. A single latent-factor explained variation in performance on both the Silent Film and Strange Stories task (in Sample 1 and 2) and test performance was sensitive to age-related differences and individual differences within each age group. A deep learning neural network automated scoring system trained on Sample 1 exhibited inter-rater reliability and measurement invariance with manual ratings in Sample 2. Validity of ratings from the automated scoring system was supported by unique positive associations between theory of mind and teacher-rated social competence. The results demonstrate that reliable and valid measures of theory of mind can be obtained using the new freely available deep learning neural network automated scoring system to rate open-ended text responses.
Original languageEnglish
Pages (from-to)165-177
Number of pages13
JournalPsychological Assessment
Volume35
Issue number2
DOIs
Publication statusPublished - 28 Feb 2023

Keywords

  • Theory of Mind
  • Middle Childhood
  • Adolescence
  • Machine learning
  • Deep Learning
  • Psychometric
  • Social competence

ASJC Scopus subject areas

  • Developmental and Educational Psychology
  • General Psychology
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Machine learning and deep learning systems for automated measurement of ‘advanced’ theory of mind: reliability and validity in children and adolescents'. Together they form a unique fingerprint.

Cite this