In 1996 the American Psychological Association (APA) established a task force on statistical inference in order to clarify the use of statistical inference in psychological research and to suggest improvement. A review of 406 empirical research articles published before and after the release of the APA recommendations was recently published by Schatz et al. (1).
The purpose of the review was to evaluate the adherence to the statistical recommendations, and it focused on 4 specific topics: i) neglect of effect size, ii) increased risk of type-1 error (multiplicity effects), iii) inappropriate presentation of p-values (other than exact values), and iv) inappropriate use of null hypothesis testing (to show similarity).
Of the 406 articles only 93 (23%) documented effect size. However the published recommendations seemed to have improved the documentation, from 4 of 62 articles (4%) prior to the recommendations to 50 of 170 (29%) after their release.
increased risk of type-1 errors was found in 275 (68%) of the 604 articles. The recommendations had had no effect on this.
Inappropriate p-value presentation was found in 293 of the 406 articles (72%), and the recommendations had not had any effect on this.
Finally, inappropriate use of null hypothesis testing was found in 194 of 405 articles (48%). Again the APA recommendations had not had any effect.
The authors conclude that neuropsychologists involved in empirical research should be better aware the theoretical aspects of research methodology, especially regarding the limitations of hypothesis testing.
Schatz P, Jay KA, McComb J, McLaughlin JR. Misuse of statistical tests in Archives of Clinical Neurospsychology publications.2005;8:1053-1059.