Experiments: Incentive experiment

Since the beginning of the Active Lives survey in 2015, respondents have received £5 for each completed survey. Inflation during this timeframe means that the buying power of the incentive has been greatly diminished.

At the same time, high levels of inflation have increased printing and postage costs greatly. As incentives make up a smaller proportion of overall costs, a smaller increase in response would be needed to make higher incentive amounts financially viable.

It was therefore deemed sensible to test the impact different incentive models would have on costs, as well as carbon footprint of the survey.

The experiment was run on additional sample, to ensure the changes would not impact the survey data; however the impact of the experiment on the data was ultimately deemed minor enough that the experiment cases were included in the Year 9 data.

This experiment was conducted across one wave in September 2024 (Wave 107). The analysis and reporting were finalised by the beginning of December.

Groups 1 to 4 followed an ‘early bird’ approach. This means that respondents who took part before a certain date were given £10, whereas everyone who took part after that date received a £5 voucher.

Two different wordings and two different cut-off dates were trialled to see what worked best.

Wording

Previous findings from other surveys found that an ‘early bird’ incentive is not guaranteed to have a positive impact on response rates.

The difference in incentive amounts before and after the cut-off date, as well as the wording of the invitation letters, can play a large role in impacting respondents’ decision to take part.

We therefore trialled two versions of the 'early bird' incentive, testing the impact of different phrasing on invitation letters.

Wording 1 placed more emphasis on the higher incentive amount, while the other emphasised the lower incentive amount.

Wording 1 was meant to convey more urgency and motivate the respondent to take part as to not ‘lose out’, whereas Wording 2 aimed to frame the higher amount as an ‘added bonus’.

Cut-off dates

Two different cut-off dates (17 and 30 September 2024) were tested, with each of them placed the day before a reminder letter (either Mailing 2 or 3 depending on the experiment group) was sent out.

The cut-off dates corresponded to the mailing dates for M2 and M3 of the survey.

Group 5 received a higher incentive value of £10 regardless of when they took part in the survey.

The experiment used a factorial design:

Due to the factorial design, a sample size of 500 was available for analysis of each experimental condition.

This provided 80% power to detect a 5.25 percentage point difference in response rate.

This was the approximate level of change needed for the tested incentive models to be financially viable.

All five experimental conditions achieved higher response rates than the main sample (18.9%). The response rates ranged from 20% (Group 4) to 24.8% (Group 1).

Wording 1 achieved a better response rate than Wording 2 (24.4% vs 20.4%) and the earlier cut-off date achieved a higher response rate than the later one (22.8% vs 22%).

For each experimental condition, the impact on carbon footprint and cost was modelled by estimating the number of letters and reminders that would need to be sent out in order to achieve the target.

All five conditions were predicted to reduce the carbon footprint, as higher response rates lead to a reduction of the sample and to a lower number of reminders being sent.

The incentive models were predicted to lower carbon footprint by between 12% (Group 4) and 25% (Groups 1,3, and 5).

However, due to the increase in incentive costs counteracting the savings in printing and postage costs, only Group 1 was predicted to lead to a decrease in cost (by 4%).

The confidence intervals associated with Group 1 indicated that the effect of this incentive model would fall somewhere between an 11% decrease in cost and a 6% increase in cost.

Due to the small sample size, the impact on survey estimates and demographic profile could not be assessed within this experiment.

Instead, it was deemed sensible to select Group 1, as the most promising option, to run a larger experiment to further investigate the impact this would have on population profiles and survey estimates of the survey.