Statistical Mistakes

The statistical reviewer's handbook

Study design and statistical analysis

Case-control study

The study is described as a case-control study. Please clarify, see Lewallen S, Courtright P. Epidemiology in Practice: Case-Control Studies. Community Eye Health. 1998;11:57–58.

Description of the study design

The relation between the studied hypothesis and the statistical analyses is unclear. This information is important for evaluating the validity of the findings. Please describe the study design more clearly in terms of tested null hypotheses, sample sizes, and analysis units.

Statistics section

The description of the statistical analysis is too brief. It is, for example, unclear whether or not the assumptions underlying the methods (e.g. statistically independent observations, Gaussian distribution, homogeneity of variance, etc.) were fulfilled. The ICMJE recommendation is to “Describe statistical methods with enough detail to enable a knowledgeable reader with access to the original data to judge its appropriateness for the study and to verify the reported results”.

Odds ratios and relative risks

Statistical and clinical significance are two different things but both are important for the reader’s interpretation of the findings. The authors have adequately presented the statistical significance but not the clinical. Better estimates can be made. Several alternative statistical models exist and should be tried, see McNutt L-A, Wu C, Xue X, Hafner JP. Estimating the Relative Risk in Cohort Studies and Clinical Trials of Common Outcomes. Am J Epidemiol 2003;157:940–943 and Barros AJD, Hirakata VN. Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio. BMC Med Res Method 2003, 3:21.

Parameter estimation and prediction

The statistical analysis seems to be based on a conflation between parameter estimation and prediction. Both of these analysis approaches are based on statistical models but the purposes are different. A prediction model is optimized with respect to its predictive accuracy, which is measured in terms of sensitivity and specificity. A major problem in the development of the model is overfitting, adaption to sample-specific random variation, and the solution is validation. Parameter estimation is performed using an explanatory model, optimized with respect to the validity of the parameter estimates. The model must, therefore, be developed on the basis of known or assumed cause-effect relations. For example, confounders are included in the model to avoid confounding bias, but including a mediator or a collider induces adjustment bias. A good explanatory model does, therefore, not necessarily provide good predictions and a good prediction model does not necessarily provide unbiased parameter estimates.

Correlated observations

The statistical analysis and the presentation of the results focus on knees instead of patients. This means that bilateral (correlated) observations are included in the statistical analysis. Has the correlation been accounted for in the statistical analysis? Are the presented results reliable? See e.g. Ranstam J. Repeated measurements, bilateral observations and pseudoreplicates, why does it matter. Osteoarthritis Cartilage 2012;20:473-475.

Normal distribution

It is stated in the statistics section that “The normal distribution of data … was confirmed with the Kolmogorov-Smirnov test”. I assume that the distributional assumption was tested using the Kolmogorov-Smirnov test. The statistical power to detect a deviation from normality depends, however, on the sample size. Given the limited sample size of the study, the word “confirm” may be an overstatement.

Significant and n.s.

The analysis strategy is based on dichotomising findings as either statistically significant or not statistically significant. This is a strategy that leads to considerable distortion of the scientific process, see Wasserstein RL, Lazar NA. The ASA’s statement on p-values: context, process, and purpose. The American Statistician 2016 doi: 10.1080/00031305.2016.1154108. Statistically significant findings are not necessarily scientifically relevant and statistical nonsignificance is just an indication of uncertainty, not of equivalence.

Bonferroni correction

The statistical analysis includes Bonferroni correction. Please describe the used strategy for addressing multiplicity issues and clarify how the correction is performed with respect to the number of tested null hypotheses, how the strategy was pre-specified, and how the results from the statistical analysis were interpreted (i.e. vis-à-vis unaddressed multiplicity effects).

The Bonferroni correction should be explained with respect to a) why such corrections are meaningful in an observational study without the pre-specification of an overall strategy for addressing multiplicity issues and b) a description of the multiplicity problems that are solved by the correction and of those that remain to be considered when interpreting the analysis results.

%d bloggers like this: