Quantifying error in effect size estimates in attention, executive function, and implicit learning

Accurate quantification of effect sizes has the power to motivate theory and reduce misinvestment of scientific resources by informing power calculations during study planning. However, a combination of publication bias and small sample sizes (∼ N = 25) hampers certainty in current effect size estimates. We sought to determine the extent to which sample sizes may produce errors in effect size estimates for four commonly used paradigms assessing attention, executive function, and implicit learning (attentional blink, multitasking, contextual cueing, and serial response task). We combined a large data set with a bootstrapping approach to simulate 1,000 experiments across a range of N (13-313). Beyond quantifying the effect size and statistical power that can be anticipated for each study design, we demonstrate that experiments with lower N may double or triple information loss. We also show that basing power calculations on effect sizes from similar studies yields a problematically imprecise estimate between 40% and 67% of the time, given commonly used sample sizes. Last, we show that skewness of intersubject behavioral effects may serve as a predictor of an erroneous estimate. We conclude with practical recommendations for researchers and demonstrate how our simulation approach can yield theoretical insights that are not readily achieved by other methods such as identifying the information gained from rejecting the null hypothesis and quantifying the contribution of individual variation to error in effect size estimates. (PsycInfo Database Record (c) 2024 APA, all rights reserved).