Statistical testing of more than two groups are common in behavioural science, but a clear methodological motivation for the chosen approach is usually not provided. Ruxton and Beauchamp (1) surveyed 12 issues of Behavioral Ecology (volume 18) and Animal Behaviour (volume 73) with regard to such multiple comparisons and found 70 papers where more than 2 groups were tested.
In 68 of the 70 papers the statistical analysis involved homogeneity tests across all groups, using an ANOVA or similar distribution-free test. Comparison between specific groups were not performed when the null hypothesis of homogeneity across all groups could not be rejected, but followed invariably when this hypothesis was rejected. The comparisons were then done using several different tests. The authors stress the importance of distinguishing between planned and unplanned comparisons, and that it is important for researchers also to consider what constitutes a biologically interesting effect. They recommend avoiding test procedures that include all pairwise comparisons when only a small subset of them are of interest.
The authors conclude from their survey that statistical testing of comparisons among more than two groups remain common in behavioral science, but that the common practice is variable and almost always suboptimal.
- Ruxton GD, Beauchamp G. Time for some a priori thinking about post hoc testing. Behavioral Ecology 2008;19:690-693.