Statistical Mistakes

The statistical reviewer's handbook

Laborotory experiments

Study design

The study design is not described in sufficient detail to allow a critical review of the performed statistical analysis. Please describe the relation between the included subjects/animals (n = ) and the statistically analysed samples (n = ). A clear description of this is necessary for identifying the analysis unit, confirming the statistical independence of the analysed observations and the degrees of freedom. See Parsons NR, Teare MD, Sitch AJ. Unit of analysis issues in laboratory-based research. eLife 2018;7:e32486. DOI: https://doi.org/10.7554/eLife.32486.

One study hypothesis and several null hypotheses

The authors of this manuscript claim to have provided “strong evidence that…”. It is, however, unclear what the “strong evidence” consists of in terms of empirical support. A large number of p-values representing statistically significant results from significance tests of null hypotheses are presented, but the actual number of performed significance tests is neither disclosed nor is it possible to deduct, and the relation between the tested null hypotheses and the experimental design is not described. The results presentation may therefore represent major multiplicity problems with several false positive test results. As no power calculation is presented, the number of false negative tests may also be substantial. Furthermore, the statistical testing has been performed using statistical methods that requires statistically independent observations, and it is unclear if this is the case.