Statistical Mistakes

The statistical reviewer's handbook

General – study design and analysis

Case-control study

The study is described as a case-control study. Please clarify, see Lewallen S, Courtright P. Epidemiology in Practice: Case-Control Studies. Community Eye Health. 1998;11:57–58.

Study design

The study design is not described in sufficient detail to allow a critical review of the performed statistical analysis. Please describe the relation between the included subjects/animals (n = ) and the statistically analysed samples (n = ). A clear description of this is necessary for identifying the analysis unit, confirming the statistical independence of the analysed observations and the degrees of freedom. See Parsons NR, Teare MD, Sitch AJ. Unit of analysis issues in laboratory-based research. eLife 2018;7:e32486. DOI:

Statistics section

The description of the statistical analysis is too brief. It is, for example, unclear whether or not the assumptions underlying the methods (e.g. statistically independent observations, Gaussian distribution, homogeneity of variance, etc.) were fulfilled. The ICMJE recommendation is to “Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results”.

Correlated observations

The statistical analysis and the results presentation focus on knees instead of patients. This means that bilateral (correlated) observations are included in the statistical analysis. Has the correlation been accounted for in the statistical analysis? Are the presented results reliable? See e.g. Ranstam J. Repeated measurements, bilateral observations and pseudoreplicates, why does it matter. Osteoarthritis Cartilage 2012;20:473-475.

Significant and n.s.

The analysis strategy is based on using statistical significance as a license for making claims of scientific findings. This is a strategy that leads to a considerable distortion of the scientific process (see Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician 2016 doi: 10.1080/00031305.2016.1154108). A sound analysis strategy is based on considerations regarding scientific relevance. Statistical significance is a measure of inferential uncertainty. Statistically significant findings are not necessarily scientifically relevant and statistical nonsignificance is just an indication of uncertainty, not of “no difference”.

Bonferroni correction

The statistical analysis includes Bonferroni correction. Please describe the used strategy for addressing multiplicity issues and clarify how the correction is performed with respect to the number of tested null hypothesis, how the strategy was pre-specified, and how the results from the statistical analysis were interpreted (i.e. vis-à-vis unaddressed multiplicity effects).