Statistical Mistakes

The statistical reviewer's handbook

Results presentation

Finite versus infinite population

Please clarify whether the presented study is descriptive or inferential, i.e. if the aim is to present a case series without generalizing the findings to other patients, or if the aim indeed is to generalize the findings beyond the studied patients. The latter approach requires the presentation of inferential uncertainty, usually p-values or confidence intervals. If such measures are presented in descriptive studies, it should be explained what they represent.

It is unclear whether the performed comparisons is a part of a finite population approach or if the authors wish to generalise the findings beyond the studies patients. In the former case, the results are relevant only for the patients included in the study. No generalisation to other patients are made; there is no room for statistical inference. While such an approach, which would be fully legitimate, seems to be supported by a results presentation referring to the studied countries in past tense, the presented results include 95% confidence intervals, and these indicate the estimation uncertainty when generalising the results to an infinite population. If the purpose is to describe a finite population, the statistical calculations may need to include a finite population correction (FPC). If the purpose instead is to generalise findings from a sample to an infinite population (including future patients), this should be reflected in how the results are presented. A clarification would be especially important if some of the presented results refer to a finite population and other results to an infinite population.

Statistical versus clinical significance

The results presentation is systematically ambiguous with regard to the word significant. Please clarify when referring to statistical significance (inferential uncertainty) and to clinical significance (practical importance). When a statement refers to statistical significance, please clarify the clinical significance of the finding (include considerations regarding estimation uncertainty). When it refers to clinical significance, please describe how this was defined and whether estimation uncertainty has been considered. Just focusing on p-values is inadequate, see Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician 2016 doi: 10.1080/00031305.2016.1154108.

Please distinguish between inferential uncertainty (statistical significance) and practical relevance (clinical significance), for example in the sentence “No significant differences … could be noticed”. Furthermore, statistically significant differences are not necessarily practically relevant. Practical importance has to be shown by other measures, usually effect size estimates (with considerations regarding the estimation uncertainty). In addition, statistical non-significance is an indication of uncertainty not of “no difference” or similarity, for example in the sentence “The two cohorts were homogenous (p>0.05)”. If the similarity between the two cohorts is important, this could perhaps be evaluated by estimating the potential differences between the two cohorts with 95% confidence interval. If the intervals exclude clinically significant differences, a conclusion of similarity would be reasonable.

No difference

Statistical non-significance is not evidence of equivalence. Claims of “no effect” or “no difference” need to include considerations regarding the estimation uncertainty of the effect or difference. Please refer to whether any clinically relevant effects are included in the relevant 95% confidence interval.

Odds ratios

The presented measure of effect is the odds ratio. However, the clinical relevance of an odds ratio (in terms of relative risk or prevalence) is not always easy to assess for the reader because this requires additional information on the baseline rate, which not always is presented, see Davies HTO. When can odds ratios mislead? BMJ 1998;316:989. What is the clinical relevance of the presented odds ratios?

Bar charts

I recommend presenting the results using dot plots instead of bar charts because a transparent results presentation is important, and in contrast to dot plots, both the number of observations and their distribution are hidden in bar charts.

%d bloggers like this: