The Active Lives survey collects approximately 180,000 responses and, as such, there will naturally be small fluctuations when we compare figures for November 2023 – 2024 against November 2022 – 2023.
It is important to ensure that any reported changes are genuine, not just differences within the margin of error.
Therefore, significance testing should be applied when comparing figures to check whether changes observed across survey years are likely to be ‘true’ changes in the population, rather than just observed by chance.
Significance testing is important to consider in any analysis of the Active Lives dataset, particularly for queries with low response counts, where natural fluctuations between years is likely to be greater.
To run significance tests, standard errors (SE) should be generated using the complex samples module in SPSS. These are applied to t-tests using a 5% significance level to assess statistical significance as follows:
- Design effects (DE) = SE/(R*(1-R)/B)^0.5
- Effective sample size (ES) = B/(DE^2)
- Difference between estimates (Diff) = R2 – R1
- Standard error of difference between estimates (SEDiff) = ((R1*(1-R1))/ES1 + (R2*(1-R2))/ES2))^0.5
- Z value = Diff / SEDiff
- P value = 2*(1-NORM.S.DIST(ABS(z value),1))
- Significant if Z<=0.05 (at 95% level)
Where:
- B = observed sample base
- R = rate
- R1 = rate of observations 1
- R2 = rate of observation 2
- ES1 = effective sample size of observation 1
- ES2 = effective sample size of observation 2
When commenting on any Active Lives results, only differences which are statistically significant should be reported on as differences.