Replicability all the way up: commentary on “Replication and methodological robustness in quantitative typology” by Becker and Guzmán Naranjo

B&GN specifically operationalize replicability as the possibility of applying different statistical models to data analysis. However, their broader definition of replicability implicitly extends to all stages of analysis, from data collection to classification and labeling. Notably, in the appendix of their study, B&GN devote a specific section to guidelines for the transparent documentation of the language sample. This section covers both the sampling procedure and the explanation of the annotation choices. Yet, they themselves acknowledge that, apart from Nichols et al. (2006), there are no studies on replicability that systematically examine how different classificatory choices or the use of alternative parameters impact data analysis and annotation. This is the point on which we would like to focus. It highlights the importance of shifting attention further upstream to a prerequisite that precedes replicability and transparency in statistical model selection. While this aspect has already been implicitly addressed in many studies, the scientific community has yet to engage in a collective discussion on how best to implement it. Thus, in this commentary, we aim to address several issues that B&GN leave in the background. We believe these issues are equally important for advancing the broader goals of replicability, robustness and open science. A second point, closely related to the first, that we wish to briefly discuss in this commentary concerns the fact that in many cases of quantitative studies, what matters is not so much (or at least not only) replicability in the statistical sense described by B&GN, but rather the ability to critically examine (and, if necessary, revise) the theoretical foundations of the quantitative investigation.