Methods for creating weights
There were six stages to the weighting strategy:
- Calculation of an individual (within household) selection weight;
- A weight to correct for half the sample being eligible for the final mailing;
- Initial calibration to mid-year population counts for local authority and age/sex and to month counts assuming a proportionate sample;
- A second stage of calibration to the same set of measures (age/sex, local authority and month) as well as to national estimates from the Annual Population Survey;
- Trimming of the second stage of calibration; and
- A final adjustment to regional counts.
We will describe each stage of the weighting for annual weights in detail and then state the difference to produce the time series weights.
-
W1: Individual (within household) selection weight
Adults (16 or older), were calculated as W1 = M_hh / m_hh where m_hh is the number of adult participants within the household (m_hh = 1 or 2) and M_hh is the total number of adults in the household.
Read more about W1: Individual (within household) selection weightTo reduce the variance of the weights, and hence increase the efficiency of the sample, the individual selection weight was trimmed at 3.
Read less about W1: Individual (within household) selection weight -
W2: Adjustment for the final mailing strategy
The second stage of weighting corrects for half the sample being eligible for a fourth mailing. This is done by identifying the questionnaires that were completed following the fourth mailing in the sample.
Read more about W2: Adjustment for the final mailing strategyThe weight adjusted the total number received to be twice the count received for the addresses eligible for the second mailing.
This was done for each survey month for both the online only postal returns.
This weight adjusts the sample so that it is statistically equivalent to a sample for which all addresses were eligible for the fourth mailing.
As was the design for previous years of the Active Lives survey, this allows the time series to be maintained for the survey estimates.
Read less about W2: Adjustment for the final mailing strategy -
W3: Initial calibration
The next stage was to calibrate to population counts for local authority and age/sex groups, as well as to month for the two sets of time-series weights that covered the full survey year to ensure that the weighted distribution by month was proportionate.
Read more about W3: Initial calibrationThis was done separately for each region, with the combined previous weights (W1 x W2) used as the starting weights.
This weighting stage corrected for the unequal selection probabilities across the local authorities, as well as disproportionate non-response; it also put each age group into its correct proportion.
Adjusting for month also balanced the sample to adjust for seasonality.
Note that this initial calibration stage was not trimmed to ensure that the distributions by local authority, age/sex groups and month were as close to the true figures as possible.
In fact, trimming at this stage would not have been sensible as it would merely have changed the weights for the very large and very small local authorities and the first few months that had lower returns.
These were features of the design that we wanted to correct for, and so trimming the weights would be detrimental.
Read less about W3: Initial calibration -
W4: Second stage of calibration
For the second stage of calibration, national estimates from the Annual Population Survey were added to the set of control totals for:
Read more about W4: Second stage of calibrationwhite/non-white, working status by gender, household size, long-term health problem, socio-economic classification (NS-SEC), and highest educational qualification (though, for postal cases, this last was not used at full-year 5).
The calibration was done separately for each region and the starting weights were the weights from the previous initial calibration stage (W3).
Read less about W4: Second stage of calibration -
W5: Trimming of second stage of calibration
To reduce the variance of weights, this calibration stage was trimmed.
Read more about W5: Trimming of second stage of calibrationThe weights from this calibration stage (W4) were divided by the initial weights (W3) to give the adjustment factor (f_adj).
This adjustment factor was trimmed at the 5th and 95th percentiles within region to give the trimmed adjustment factor (f_trim), which when multiplied by the initial weights gave the trimmed calibration weights: W5 = W2 x f_trim.
Read less about W5: Trimming of second stage of calibration -
W6: Re-scale for region
The final stage was an adjustment so that the weighted counts by region were grossed up to the mid-year population counts.
Read more about W6: Re-scale for regionThis stage simply multiplied all the weights by a constant within each region to give the final grossed weights (W6).
Read less about W6: Re-scale for region -
Time-series weights
The time series weights were produced using a parallel approach.
Read more about Time-series weightsHowever, rather than calibrating separately by region, the calibration was done separately for each month.
We did not need to add region as a control total as it was implicitly included from having local authority as a control total – local authorities nest within region.
At the last stage, we re-scaled the weights so that the weighted distribution by month matched that of a proportionate sample.
Read less about Time-series weights