‘Phantom’ compositional effects in English school value-added measures: the consequences of random baseline measurement error

Research output: Contribution to journalArticle

Standard

Harvard

APA

Vancouver

Author

Bibtex

@article{415450f67db74b8496dee83ca147bdf5,
title = "‘Phantom’ compositional effects in English school value-added measures: the consequences of random baseline measurement error",
abstract = "A compositional effect is when pupil attainment is associated with the characteristics of their peers, over and above their own individual characteristics. Pupils at academically selective schools, for example, tend to out-perform similar-ability pupils who are educated with mixed-ability peers. Previous methodological studies however have shown that ‘phantom’ compositional effects can arise purely from measurement error. Through simulations using English National Pupil Database data, this paper demonstrates that moderate rates of measurement error in baseline scores can produce a systemic bias in value-added (‘Progress’) scores that favours higher ability pupils. This bias is large enough to explain the ostensibly superior performance of selective ‘grammar’ schools that is seen in the English data and create an appreciable bias across the school performance distribution. This paper describes how such biases can arise from ‘regression attenuation’ and discusses the more general methodological implications of the results. The paper concludes that the consequences of baseline measurement error on school value-added scores is greatly mitigated by controls for school-level prior attainment, but that the English ‘Progress’ value-added measures, as they do not control for prior attainment at school-level, are seriously biased and that the observed ‘grammar school effect’ is likely to be largely or wholly spurious.",
keywords = "Measurement error, attenuation bias, compositional effects, Progress 8, school value- added, education policyra",
author = "Thomas Perry",
year = "2019",
month = "3",
day = "4",
doi = "10.1080/02671522.2018.1424926",
language = "English",
volume = "34",
pages = "239–262",
journal = "Research Papers in Education",
issn = "0267-1522",
publisher = "Taylor & Francis",
number = "2",

}

RIS

TY - JOUR

T1 - ‘Phantom’ compositional effects in English school value-added measures

T2 - the consequences of random baseline measurement error

AU - Perry, Thomas

PY - 2019/3/4

Y1 - 2019/3/4

N2 - A compositional effect is when pupil attainment is associated with the characteristics of their peers, over and above their own individual characteristics. Pupils at academically selective schools, for example, tend to out-perform similar-ability pupils who are educated with mixed-ability peers. Previous methodological studies however have shown that ‘phantom’ compositional effects can arise purely from measurement error. Through simulations using English National Pupil Database data, this paper demonstrates that moderate rates of measurement error in baseline scores can produce a systemic bias in value-added (‘Progress’) scores that favours higher ability pupils. This bias is large enough to explain the ostensibly superior performance of selective ‘grammar’ schools that is seen in the English data and create an appreciable bias across the school performance distribution. This paper describes how such biases can arise from ‘regression attenuation’ and discusses the more general methodological implications of the results. The paper concludes that the consequences of baseline measurement error on school value-added scores is greatly mitigated by controls for school-level prior attainment, but that the English ‘Progress’ value-added measures, as they do not control for prior attainment at school-level, are seriously biased and that the observed ‘grammar school effect’ is likely to be largely or wholly spurious.

AB - A compositional effect is when pupil attainment is associated with the characteristics of their peers, over and above their own individual characteristics. Pupils at academically selective schools, for example, tend to out-perform similar-ability pupils who are educated with mixed-ability peers. Previous methodological studies however have shown that ‘phantom’ compositional effects can arise purely from measurement error. Through simulations using English National Pupil Database data, this paper demonstrates that moderate rates of measurement error in baseline scores can produce a systemic bias in value-added (‘Progress’) scores that favours higher ability pupils. This bias is large enough to explain the ostensibly superior performance of selective ‘grammar’ schools that is seen in the English data and create an appreciable bias across the school performance distribution. This paper describes how such biases can arise from ‘regression attenuation’ and discusses the more general methodological implications of the results. The paper concludes that the consequences of baseline measurement error on school value-added scores is greatly mitigated by controls for school-level prior attainment, but that the English ‘Progress’ value-added measures, as they do not control for prior attainment at school-level, are seriously biased and that the observed ‘grammar school effect’ is likely to be largely or wholly spurious.

KW - Measurement error

KW - attenuation bias

KW - compositional effects

KW - Progress 8

KW - school value- added

KW - education policyra

UR - https://doi.org/10.1080/02671522.2018.1424926

U2 - 10.1080/02671522.2018.1424926

DO - 10.1080/02671522.2018.1424926

M3 - Article

VL - 34

SP - 239

EP - 262

JO - Research Papers in Education

JF - Research Papers in Education

SN - 0267-1522

IS - 2

ER -