Statistical Mistakes

The statistical reviewer's handbook

General – results presentation

Finite versus infinite population

It is unclear whether the performed comparisons is a part of a finite population approach or if the authors wish to generalise the findings beyond the studies patients. In the former case, the results are relevant only for the patients included in the study. No generalisation to other patients are made; there is no room for statistical inference. While such an approach, which would be fully legitimate, seems to be supported by a results presentation referring to the studied countries in past tense, the presented results include 95% confidence intervals, and these indicate the estimation uncertainty when generalising the results to an infinite population. If the purpose is to describe a finite population, the statistical calculations may need to include a finite population correction (FPC). If the purpose instead is to generalise findings from a sample to an infinite population (including future patients), this should be reflected in how the results are presented. A clarification would be especially important if some of the presented results refer to a finite population and other results to an infinite population.

Statistical versus clinical significance

The results presentation is systematically ambiguous with regard to the word significant. Please clarify when referring to statistical significance (inferential uncertainty) and to clinical significance (practical importance). When a statement refers to statistical significance, please clarify the clinical significance of the finding (include considerations regarding estimation uncertainty). When it refers to clinical significance, please describe how this was defined and whether estimation uncertainty has been considered. Just focusing on p-values is inadequate, see Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician 2016 doi: 10.1080/00031305.2016.1154108.

No difference

Statistical non-significance is not evidence of equivalence. Claims of “no effect” or “no difference” need to include considerations regarding the estimation uncertainty of the effect or difference. Please refer to whether any clinically relevant effects are included in the relevant 95% confidence interval.

Odds ratios

The presented measure of effect is the odds ratio. However, the clinical relevance of an odds ratio (in terms of relative risk or prevalence) is not always easy to assess for the reader because this requires additional information on the baseline rate, which not always is presented, see Davies HTO. When can odds ratios mislead? BMJ 1998;316:989. What is the clinical relevance of the presented odds ratios?

Bar charts

I recommend presenting the results using dot plots instead of bar charts because a transparent results presentation is important, and in contrast to dot plots, both the number of observations and their distribution are hidden in bar charts.