The dataset

Kaul and Wolf use Roy Morgan Research’s Single Source (Australia) survey data¹⁷, over the period January 2001 to December 2013. The subset on minors ages 14-17 years comprises about 41,438 observations, aggregated by month, with an average of 266 observations per month. Monthly smoking prevalence was produced ‘as the average of the 0-1 variable smoker that indicates whether an individual in the sample smokes’⁵. Roy Morgan Research’s data are known for the consistency of their sampling methods¹⁸, and have been used in previous research to obtain reliable estimates of smoking prevalence in Australia^18,19.

Contrary to what Kaul and Wolf indicated in their Lancet response (see above), Roy Morgan Research’s data are not publicly available. With no funds to purchase the data, we reconstructed them from Figures 1 and 2 in the paper on minors⁵, following the same method used previously¹⁴. We were able to replicate results of the authors’ weighted least square regression to within 3 decimal places (presented in Table 1 of their working paper⁵). The total reconstructed sample size (sum of the reconstructed monthly sample sizes) matches the total sample size given by the authors (our method is described in detail in the supplementary online material, which also includes the Python program to reconstruct the data and a copy of the data set).

Figure 1

Illustration of Kaul and Wolf’s inference method based on pointwise confidence intervals. If the observed prevalence for any month during the plain packaging period falls in the “effect area”, this is considered as evidence of an effect. Yellow effect area corresponds to confidence intervals at the 90% level of confidence, effect area at the 95% level of confidence.

Figure 2

Comparing standard deviation estimates used by Kaul et Wolf (o^/nt, in red) and those of the prevalence distribution B(n_t,p_t)/n_t derived from the binomial distribution, (pt(1−pt)/nt, in blue) taking p_t on the trend line, i.e. p_t = 0.1147 – 0.00037·t

Power analysis of Kaul and Wolf’s results

In their revised working paper, Kaul and Wolf carried out a formal power analysis to address ‘the concern of whether our trend analysis has any reasonable power at all against a possible plain packaging effect’⁵. They defined their inference method as consisting of three steps: 1) fitting a linear time-trend using weighted least squares regression, 2) comparing the average of the residuals prior to PP implementation to their average post-PP implementation and carrying out a formal two sample t-test to assess if the post-PP average is smaller than the pre-PP average (if the test rejected the null hypothesis, this was considered as evidence for a PP effect), and finally 3) checking whether any observed prevalence during the PP period was below the 90% pointwise monthly confidence intervals centred at the fitted trend. To help the reader visualize this third step, we call the set of values that are smaller than the fitted trend minus the 90% confidence intervals the effect area²⁰, shown in yellow in Figure 1. Kaul and Wolf interpret any observed monthly prevalence falling in the effect area as evidence of a PP effect. If, instead of 90% confidence intervals, 95% confidence intervals are used, the effect area gets smaller, as illustrated by the hatched area in Figure 1. They computed the estimated power values by Monte-Carlo simulation with ‘generated pseudo prevalence data according a model that is in agreement with the observed data but has a specified plain packaging effect’⁵. They considered plain packages effects (variable Δ) ranging from 0.25% to 1.5%, in increments of 0.25%, and assumed the effect was ‘enforced’ from December 2012 onwards, i.e. they used the immediate (or sudden) effect model described by Laverty and colleagues¹³.

For each month t, Kaul and Wolf’s algorithm generated pseudo prevalence values that were distributed according to the normal distribution N(p^t−Δ,σ^2/nt)

with means p^t−Δ and variance σ^2nt where p^t is the prevalence value corresponding to the trend line at month t, σ^2 is the estimated variance of the residuals to the fitted trend line, and n_t is the number of observations for month t. There are two issues with the choice of this distribution and its parameters. For simulation to be meaningful, the pseudo data should behave as much as possible like real data. In this case, real data follow a binomial distribution: for each month t = 1,...,156, a random sample of size n_t is assumed to be drawn from a large population in which smoking prevalence is p_t. The number of smokers in the sample is thus a random variable following the B(n_t, p_t) binomial distribution. A common rule of thumb is that the normal distribution provides a good approximation of the binomial distribution when p_{t χ}n_t > 10. However, the first issue is that, with the data at hand, there are many combinations of parameters where this criterion is not met and where, therefore, the normal distribution poorly approximates the underlying distribution. It is therefore preferable to use the binomial distribution to generate pseudo prevalence values. This is easily done in the R statistical programming language by using the rbinom function instead of rnorm function, with no computational penalty.

The second, more significant, issue is that the variance used by Kaul and Wolf, σ^2 remains constant for all p_t prevalence values, across all months t = 1,…,156, while the binomial variance is a function of p_t, i.e. , σ^t2=ntpt(1−pt), and varies each month as p_t decreases with time. Figure 2 shows that the standard deviation used by Kaul and Wolf in their power calculation is substantially higher than the standard deviation of the binomial distribution during the PP period, leading the Monte Carlo simulation to more likely generate extreme values and trigger situations that Kaul and Wolf regard as indications of a PP effect, thus artificially inflating the power values of their test.

To remedy this anomaly, we have redone Kaul and Wolf’s power calculations with random data generated with the B(n_t, p^t) binomial distribution, using the same immediate effect model as they did, as well as the gradual effect model suggested by Laverty and colleagues, ‘in which the potential impact of standardised packaging effect is gradual, increasing linearly to reach the predicted decrease at the end of a given period’¹³.

In addition to their ‘liberal’ inference method that assumes a PP effect if at least one observed monthly prevalence is below the 90% pointwise confidence interval centred on the fitted trend, Kaul and Wolf envisaged several variations of their methodology, which they called ‘more conservative approaches’⁵. The first variation they proposed was to change the confidence level from 90% to 95%, which they say is ‘more standard in applied research and would result in wider confidence intervals’. To verify whether such a more conservative method does indeed ‘reinforce’ the authors’ conclusion, as they stated, we have also run the power calculations with pointwise confidence intervals at the 95% level.

All simulations were done with 100,000 Monte Carlo repetitions.

Logistic regression analysis

The logistic regression model for binary data is a robust and more appropriate alternative to multiple linear regression for proportions²¹. As seen in the previous section, of the prevalence figures contained in the reconstructed Roy Morgan dataset, are derived from the aggregation of individual dichotomous (smoker/non-smoker) variables. We thus fitted a logistic regression model with the number of minors who smoked and the number who did not smoke each month as the outcome variable, using the indicator variables shown below for PP and for other tobacco control measures that may have confounded the potential impact of PP: Each variable comprises 156

values, one for each ordinal month, from month 1 (January 2001) to month 156 (December 2013). Indicator variables take the values 0 (‘not implemented’) or 1 (‘implemented’). The smoke.free variable also contains intermediate values reflecting the proportion of the Australian population living in States with smoke-free policies, as the measure was progressively implemented throughout the country (we refer the reader to Diethelm and Farley¹⁴ for a detailed description). The full dataset, including indicator variables, is shown in the supplementary on line material.

We ran stepwise (forward selection, backward elimination, bidirectional selection) logistic regression, selecting the model with smallest Akaike Information Criterion (AIC – a function of the number of parameters fitted and the log likelihood). All analyses were performed with the R statistical programming language.

We also performed a power analysis to check the power achieved with the logistic regression analysis, to assess the impact of the small samples sizes. This was done using the same Monte Carlo simulation described in the previous section, adjusted for logistic regression. At each iteration, a pseudo number of smokers was generated using the binomial random distribution Bn_t p^t), where the monthly prevalencep^t was as predicted by the logistic model, assuming a range of PP effects from 5% to 25% relative reduction in smoking prevalence. The power figure was then calculated as the proportion of Monte Carlo iterations exhibiting a statistically significant PP effect with p-value , when applying the logistic regression analysis described above.

Power analysis of Kaul and Wolf’s results

The results of the power analysis of Kaul and Wolf’s inference method are presented in Table 1. Column 2 shows their power calculations as presented in Table 2 of their working paper⁵. For each potential effect of plain packaging from a 0.25% to a 1.50% absolute reduction in smoking prevalence simulated, the re-estimated power figures are all substantially lower than Kaul and Wolf’s estimates. Considering, for instance, the row with a PP effect of 0.5% reduction, Kaul and Wolf attributed a power of 0.64 to their inference method to detect such an effect (column 2), while our calculations show it to be only 0.36 (column 3), assuming, as they did, a sudden PP effect, or even 0.29 (column 4) with a gradual PP effect. Corresponding power values, when Kaul and Wolf’s inference method uses 95% confidence intervals (columns 5-7), are all smaller.

Logistic regression analysis

Smoking prevalence in minors decreased over the 13-year period from a mean of 11.6% in 2001 to 5.6% in 2013, an overall 52% relative decline (Figure 3). The fitted logistic regression model (blue dotted line) showed a significant 5.5% annual reduction in smoking prevalence (95% confidence interval 4.6% to 6.4%) (Table 2). There was a modest, though not statistically significant, impact of plain packaging (Figure 3, solid red line) introduced from November 2012 with a 12.1% relative reduction in smoking prevalence, equivalent to approximately a 2-year decline in prevalence. All three stepwise regression model selection strategies (forward, backward and bidirectional) resulted in the same model with smallest AIC value. The impact of tax increases on smoking prevalence, introduction of comprehensive smoke-free policies, and graphic health warnings, were very limited and not statistically significant in this dataset.

Figure 3

Times series of observed prevalence with fitted logistic regression lines based on selected model and time trend line

Table 3 shows the power of the logistic regression model to detect (using binomially generated pseudo data) a range of PP effects ranging from 5% to 25% relative reduction, with the estimated PP effect highlighted. The power figures give the probability of finding a statistically significant effect with a p-value 0.05.

Power analysis of Kaul and Wolf’s results

Our results confirm what Laverty and colleagues¹¹ had pointed out: given the data at hand, and the small sample sizes, Kaul and Wolf’s method lacked sufficient power for detecting the likely impact of PP on smoking prevalence amongst minors in Australia during the first year after the measure was implemented. With the PP effect level that could be plausibly expected (0.5 percentage point absolute decrease of prevalence), their method actually had a much greater probability of not finding an effect than of finding one. It is therefore not surprising that they did not find any evidence of a PP effect. Our results contradict Kaul and Wolf’s statement that ‘if anything’ their analysis was ‘slightly biased in favor of finding a statistically significant (negative) effect of plain packaging on smoking prevalence of Australians aged 14 to 17 years’⁵. The best that could be said of their analysis is that it was inconclusive.

Furthermore, while emphasizing that their analysis did not discover evidence of a PP effect, Kaul and Wolf added that ‘[m]ore conservative statistical inference methods would only reinforce this conclusion’. Table 1 shows that the power figures, associated with the more ‘conservative’ 95% confidence intervals, are all lower than those associated with their ‘liberal’ 90% counterparts; contrary to their assertion, more conservative approaches are in fact less conclusive. This was to be expected as the effect area associated with 95% confidence intervals (hatched area in Figure 1) is smaller than the effect area associated with 90% confidence intervals.

Logistic regression analysis

The results of the logistic regression analysis show a clear, though not statistically significant, reduction in smoking prevalence following the introduction of plain packaging, equivalent in magnitude to approximately a 2-year relative decline in prevalence. No measurable effect of the other tobacco control measures could be detected in this dataset. This contrasts with a previous analysis¹⁴ using identical methods of trends in smoking prevalence among adults, which showed significant reductions in smoking prevalence following a tax increase and the introduction of comprehensive smoke-free policies and plain packaging. However, the dataset on minors is limited compared with that on adults – average sample size 266 compared with 4519, average smoking prevalence 8.6% compared with 21.5% – and it is of little surprise that the impact of different tobacco control measures cannot be detected given the large sampling variation. We note that the introduction of plain packaging was accompanied by a relative reduction in adult smoking prevalence of 3.7%, equivalent in magnitude to approximately a 2-year relative decline in prevalence (1.7%).

Our result concerning the effect of plain packaging on the prevalence of smoking among minors is thus consistent with our result for adults¹⁴. It is furthermore consistent with the finding of a recent study²² that provides ‘evidence that a considerable proportion of young smokers tried to quit or thought about quitting as a result of the new Australian tobacco packs in the period following their introduction’.

Power analysis of the logistic regression shows that even if the detected effect was real, there was a low probability of finding a statistically significant result at the p = 0.05 level: only 3 chances out of 10 assuming the PP effect was immediate, or even about 1 out of 10 with a gradual effect, mostly due to the low sample size of the data at hand. Thus, the lack of significance of our result was to be expected and cannot be given any further interpretation.