Part of the Bristol and Bath Psychology seminar series jointly hosted by both the School of Psychological Science and the Department of Psychology, respectively.
Abstract: Thanks to the wide availability of “default Bayes factors” that can be applied without the need to specify problem-specific prior distributions, Bayes factors are becoming increasingly popular. Default Bayes factors achieve this by formulating the prior distribution on a standardised effect size scale. For example, in Rouder et al.’s (2012) default Bayes factor for ANOVA designs, implemented in the BayesFactor package, the observed effect is standardised by the residual standard deviation. Whereas the idea of default Bayes factors is that the application is easy, this normalisation can have unintended consequences. For example, van Doorn et al. (in press, CBaB) showed that Rouder’s default Bayes factor is not invariant to the level of data aggregation in a mixed model setting with repeated measures. This feature can allow researchers to strategically manipulate their Bayes factor results (e.g., more aggregation can lead to higher evidence for an effect). We show that this is solely a feature of the default Bayes factor that relies on a normalised effect size. Neither the frequentist p-value nor a Bayesian approach relying on the un-normalised effect size (using either Bayes factor or other inferential approach) suffer from this problem. We conclude that in a mixed-model setting, in which usually multiple variance components are estimated, formulating standardised effect sizes measures is a non-trivial endeavour. In general, researchers should try to formulate their hypotheses on inherently meaningful units instead of relying on the deceiving convenience provided by standardised effect sizes. See: https://psyarxiv.com/kxhfu/
Bio: Henrik is a lecturer in mathematical and quantitative psychology in the Department of Experimental Psychology at UCL. He did his PhD in 2014 under the supervision of Christoph Klauer in Freiburg (Germany). After a postdoc with Klaus Oberauer in Zurich he moved to the UK; first as an assistant professor at the University of Warwick before joining UCL in 2020. His primary research interests are the development and implementation of computational methods and statistical tools for psychology and related disciplines as well developing and testing of mathematical models of higher-level cognition.