Statistical Mistakes

The statistical reviewer's handbook

General – study design and analysis

Case-control study

The study is described as a case-control study. Please clarify, see Lewallen S, Courtright P. Epidemiology in Practice: Case-Control Studies. Community Eye Health. 1998;11:57–58.

Description of the study design

The relation between the studied hypothesis and the statistical analyses is unclear. This information is important for evaluating the validity of the findings. Please describe the study design more clearly in terms of tested null hypotheses, sample sizes, and analysis units.

Statistics section

The description of the statistical analysis is too brief. It is, for example, unclear whether or not the assumptions underlying the methods (e.g. statistically independent observations, Gaussian distribution, homogeneity of variance, etc.) were fulfilled. The ICMJE recommendation is to “Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results”.

Correlated observations

The statistical analysis and the results presentation focus on knees instead of patients. This means that bilateral (correlated) observations are included in the statistical analysis. Has the correlation been accounted for in the statistical analysis? Are the presented results reliable? See e.g. Ranstam J. Repeated measurements, bilateral observations and pseudoreplicates, why does it matter. Osteoarthritis Cartilage 2012;20:473-475.

Normal distribution

It is stated in the statistics section that “The normal distribution of data … was confirmed with the Kolmogorov-Smirnov test”. I assume that the distributional assumption was tested using the Kolmogorov-Smirnov test. The statistical power to detect a deviation from normality depends, however, on the sample size. Given the limited sample size of the study, the word “confirm” may be an overstatement.

Significant and n.s.

The analysis strategy is based on dichotomising findings as either statistically significant or not statistically significant. This is a strategy that leads to a considerable distortion of the scientific process (see Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician 2016 doi: 10.1080/00031305.2016.1154108). A sound analysis strategy is based on considerations regarding scientific relevance. Statistical significance is a measure of inferential uncertainty. Statistically significant findings are not necessarily scientifically relevant and statistical nonsignificance is just an indication of uncertainty, not of “no difference”.

Bonferroni correction

The statistical analysis includes Bonferroni correction. Please describe the used strategy for addressing multiplicity issues and clarify how the correction is performed with respect to the number of tested null hypothesis, how the strategy was pre-specified, and how the results from the statistical analysis were interpreted (i.e. vis-à-vis unaddressed multiplicity effects).

The Bonferroni correction should be explained with respect to a) why such corrections are meaningful in an observational study without the pre-specification of an overall strategy for addressing multiplicity issues and b) a description of the multiplicity problems that are solved by the correction and of those that remain to be considered when interpreting the analysis results.