INTRODUCTION
Disrupting tobacco sales to minors has shown to be effective in reducing smoking among youth1. However, the existence of legislation preventing sale of tobacco to minors alone is not sufficient to achieve consistent compliance among tobacco retailers. Evidence suggests that compliance rates among tobacco retailers must be over 90% in order to have any significant effect on youth tobacco use2. Tobacco vendor training is recognized as an essential element of a comprehensive strategy to reduce youth access to tobacco2.
The US Food and Drug Administration (FDA) has published a training development protocol for tobacco retailers that is based upon published evidence3. A range of training methods were recommended to facilitate the process, including classroom or face-to-face settings by a trainer, self-study of written materials, or online training. However, no distinction was made between the anticipated effect of shallow learning versus deep learning (immersive) techniques such as simulations, mock customer interactions, gaming strategies, and formative testing.
The purpose of the study is to describe the development and evaluation of an evidence-based online e-learning program entitled Tobacco Retailer Advanced Compliance (TRAC) training. Specifically, the aims of the evaluation were: 1) perform a literature search for existing tobacco retail training protocols and programs; 2) develop training content and procedures that are based on best-practice evidence; 3) create a user-friendly online training interface to allow for effective online delivery of retail training; and 4) to pilot test the online retail training among a representative sample of retailers. The Alberta government implemented new tobacco legislation in 2014 that requires tobacco retailers to complete a mandatory training program although no required training program has been implemented to date. Nevertheless, a mandatory training program is awaiting development and implementation to satisfy the regulatory requirement. The objective of this mixed-methods study is to evaluate the acceptability, useability and knowledge impact of an evidence-based online tobacco retailer program.
METHODS
Study design and setting
Mixed research methods were used including: 1) both qualitative and quantitative methods to assess acceptability and usability of beta version of the training program, and 2) quantitative methods to access the impact on participants’ knowledge.
Quantitative methods
To test learner knowledge, each module included a final set of 15 questions randomly chosen from a list of 30. Learners were required to answer 80% of answers correctly to successfully complete the course. Three attempts were allowed to pass the final test; in the case of failing all attempts, the learner was required to retake the entire course.
One group pre-test and post-test design was used to assess the impact of TRAC. To measure the mean difference between two dependent means (matched pair) with 80% power and α=0.05, a sample of 71 participants was needed to be able to detect a one-tailed difference of 0.3 (small effect size) in final evaluation score between pre-test and post-test. The calculation assumed that participants in the pretest will correctly answer 0.5 of the questions and in post-test that number will be 0.8 or higher (passing score is 0.8).
A link to a 16-question evaluation survey was provided at the end of the TRAC training modules. Twelve questions used a 5-point Likert scale (1=strongly agree to 5=strongly disagree). The Likert scale was attached to a given statement regarding the training module design, navigation and language, information provided and impact on knowledge, and relevance to daily work.
Qualitative methods
Given the exploratory nature of the proposed study, qualitative research methods were used as they are the most appropriate to study perspectives, processes, and context4,5, and to allow for in-depth examination of people’s experiences6,7. A focus was particularly invested on data on usability and acceptability of the training modules. Through the project team collaboration with Federated Co-operatives Limited® (FCL) gas convenience stores, a list of potential participants from the company’s central Alberta stores was established. A member of our team who was trained in interviewing skills contacted potential participants who completed the TRAC training prior to conducting the interviews. After explaining the study procedures and obtaining their consent, study participants were interviewed using a semi-structured interview guide (Supplementary file Figure 1). All interviews were audio-recorded and transcribed.
Figure 1
Difference in follow-up survey responses* between store managers and clerks in the evaluation of the TRAC training program, Central Alberta, 2019
Process evaluation
Process evaluation is ‘the systematic collection of information on a program’s input, activities, outputs, as well as the program’s context and other key characteristics’8. In this ongoing process, the principles of participatory evaluation were applied to build on the contribution of everyone involved in the program, including ongoing communication with the project team and advisory committee, project document revision, and use of different data collection methods such as observations and interviews.
Outcome evaluation
According to Kirkpatrick and Kirkpatrick9, evaluation for training programs should include four levels of evaluation: 1) Reaction (how did participants feel about the learning experience?); 2) Learning (how did the training increase their knowledge from pre to post training?); 3) Behavior (what is the impact on the trainee’s behavior?); and 4) Results (what are the final results of training on the work environment?). This article will report on an evaluation designed to address levels 1 and 2 of outcomes evaluation; thus, the questions that the evaluation aims to answer are: 1) How do participants feel about the learning experience?; and 2) How does the training affect their knowledge from pre- to post-training?
Tobacco Retailer Advanced Compliance (TRAC) training modules development
Instructional design
An instructional design document was produced and reviewed by the project team and advisory committee, providing details about the course design plan, learning objectives, development process, deployment tools, operational and technical specifications, test requirements, and completion certificate. The plan was to guide learners through typical interactions with a customer to understand how to ask and verify the age of the purchaser, to determine what products are legal to sell, to identify restrictions on advertisement, and the right and responsibility of store clerks to refuse sales.
Visual script/storyboard
The content for each module was scripted to ensure that learning objectives were met. The program was visually scripted in the authoring tool describing all the text, audio, visuals, media, animations, and interactions, as well as showcasing the branching and links in the program. All scripts were approved by the project team and advisory committee.
Design mock-up
These visuals reflected the overall design, colors, and controls (non-functioning) for the program screens. They were revised and approved by the project team before proceeding to the next stage.
Design prototype
This graphic user-interface was based on the approved static screen design and included functions approved by the design team including navigation controls. The functioning screen design was the base for the template that was used to build the programs.
Template
The course navigation controls were built into a working user-interface to be integrated with the design layout.
Authoring beta version
The process included programming, interaction, and insertion of the different media elements. Authoring was done with Articulate Storyline 360®. The beta version, consisting of the program and associated materials, was reviewed and tested.
Quality assurance
Each component created was reviewed by the Project Manager. Full media testing of all audio and video files, internal and external links was done in the runtime environment. SCORM packages are tested for compliance with the ADL SCORM compliance suite. SCORM is an industry standard software platform for online learning management systems.
The project was based on an appropriate needs assessment conducted by the project team and on extensive knowledge regarding current legislation and practices to improve retailer compliance with current youth access legislation.
A script for each module was developed and underwent several revision cycles by the project team and evaluator. Final scripts were reviewed and approved by the advisory committee. Similar steps were followed for various elements of each module, including test questions lists, designs, colors, characters, animations, and interactivity options.
When the module development was completed, the training was initially tested for integrity and ease of use several times by the project team and evaluator. After the initial testing, a weblink to the module was sent to the advisory committee to provide further analysis and feedback. The committee’s response was overall quite positive with no requests for major alterations.
Data collection and analysis
A memo was sent to 12 FCL gas convenience stores in Edmonton and central Alberta requesting staff participation in the program evaluation. Each store employed an average of 10–12 staff, and several reminders were sent to employees regarding the deadline for participation.
To simplify the process, the training program design vendor (Yardstick®) combined the pretest questions, the training, and the post-test questions in one account with one common weblink sent to participants. The courses were hosted on RapidLMS® which is Yardstick’s proprietary learning management system and each user was required to set up an account and to utilize individually encoded enrollment keys to access the course.
Quantitative
Chi-squared testing was used to compare categorical variables, while Student’s t-tests and paired tests were used for continuous variables.
Qualitative
We started this study with ‘explorer’s eyes’ without being driven by a prior hypothesis10. Inductive content analysis techniques were used to analyze the interviews to identify themes both within and across groups7. Collected data were reviewed to find the emerging themes or patterns. The analysis looked for how well the project was doing, what was working, what should be done differently and what difference it was making. After independent review, team members worked together to arrive at a final tabulation of potential themes and identified issues with the training module (beta version) to be modified into the final version, following any required revisions. Categories were identified and grouped together into themes using Atlas t-software.
RESULTS
Quantitative
Useability
A total of 128 participants were enrolled in the evaluation study, of which 90 were store clerks (CLKs) and 38 were store managers (MGRs). Statements that focused on design and appearance of the modules demonstrated that a majority of participants considered the modules as appealing, interesting, and successfully held their attention. Nevertheless, a higher percentage of MGRs (over 90% for three questions) were positive (agree/strongly agree) compared to CLKs (about 50% for the three questions). There was a significant difference between MGRs and CLKs in the mean Likert scale score for statements regarding module appearance [4.4 (0.65) vs 3.3 (1.04), t=3.57, p<0.001, respectively]. Similar results were reported for the training being interesting [4.3 (0.61) vs 3.4 (1.1), t=2.64, p=0.006] and for its ability to hold attention [4.4 (0.84) vs 3.2(1.3), t=2.81, p=0.004] (Figure 1, and Tables 1 and 2).
Table 1
Difference in follow-up survey response scores* between store managers and clerks in the evaluation of the TRAC training program, Central Alberta, 2019
Table 2
Percent (%) difference in follow-up survey responses between store managers and clerks in the evaluation of the TRAC training program, Central Alberta, 2019
A majority of participants agreed or strongly agreed that the modules were easy to navigate, to follow, and to understand. A higher percentage of MGRs (around 80–85%) were positive (agree/strongly agree with the statement) compared with CLKs (about 60–70%) for the three questions. However, the difference between MGRs and CLKs was only significant for their evaluation of how easy it had been to follow the module [4.5 (0.65) vs 3.8 (1.11), t=2.22, p=0.02, respectively]. No significant difference was seen in the evaluation of how easy it was to navigate [3.9 (1.5) vs 3.6 (1.3), t=0.47, p=0.5] and understand the information [4.1 (1.03) vs 3.5 (1.4), t=1.5, p=0.07] in the training modules (Figure 1, and Tables 1 and 2).
The lowest percentage of positive responses among MGRs (50%) and CLKs (36–40%) was for the training impact on knowledge regarding health effects and social impacts of tobacco use. The mean score on the Likert scale was the lowest for this section, with no significant difference between MGRs and CLKs [3.5 (1.1) vs 3.1 (1.2), t=0.91, p=0.18], and for the knowledge regarding social impact of tobacco use [3.4 (0.75) vs 2.96 (1.3), t=1.08, p=0.14]. However, when asked about the impact of training on knowledge regarding laws on tobacco sales, the percentages almost doubled for both MGRs (94%) and CLKs (64%). Similarly, the mean score was the highest for MGRs and second highest for CLKs [4.6 (0.63) vs 3.5 (1.2), t=3.44, p<0.001] (Figure 1, and Tables 1 and 2).
When asked about the relevance of training for the learners’ daily work, 100% of MGRs agreed/strongly agreed that the training was useful for their job, and 93% agreed/strongly agreed that the training helped them decide when to refuse to sell tobacco. In comparison, 60% of CLKs agreed/strongly agreed for both questions. The mean score was significantly higher for MGRs compared to CLKs for usefulness of the training to their daily job [4.4 (0.50) vs 3.6 (1.1), t=2.5, p=0.009], and for decision making regarding selling tobacco [4.07 (0.48) vs 3.5 (1.1), t=1.9, p=0.03] (Figure 1, and Tables 1 and 2).
About 86% of MGRs and 60% of CLKs who underwent the training would recommend the training to others. On a 5-point scale, where 5 means strongly agree, the MGR mean score was 4.3 (1.2) while the CLK mean score was 3.4 (1.3). The difference was significant (t=2.23, p=0.02) (Figure 1, and Tables 1 and 2).
Knowledge
There were 53 CLKs who completed CLK training, 13 MGRs who completed CLK training, and 22 MGRs who completed MGR training (Figure 2 and Table 3).
Table 3
Pre-training and post-training test scores by store clerks (CLKs) and store managers (MGRs) regarding the TRAC training program, Central Alberta, 2019
Figure 2
Pre-training and post-training test score results for the TRAC training program among store clerks (CLKs) and managers (MGRs), Central Alberta, 2019
CLKs who completed CLK training (n=53)
The mean (SD) score for the post-test was 75.5 (11.1) compared to 59.1 (12.8) for pretest score. Only about 5% of participants in the pretest achieved a passing score of 80, while more than 25% of participants in the post-test reached the passing score. Paired t-tests showed a significant difference between mean pre-test and post-test scores (t=8.6378, p<0.001). There was no significant difference between the mean pretest score of training completers and non-completers [59.1 (12.8) vs 60.6 (11.1), t= -0.5732, ns].
MGRs who completed CLK training (n=13)
In the post-test, the mean (SD) score was 77.9 (10.0) compared to the mean pretest score of 66.5 (12.8). About 10% of MGRs in the pretest achieved the passing score of 80, while more than 50% of participants in the post-test reached the passing score. Paired t-tests show a significant difference between mean pre-test and post-test scores (t=3.74, p<0.05).
MGRs who completed MGR training (n=22)
The post-test mean (SD) score was 73.1 (12.3) compared to 51.5 (11.1) for the pretest score. None of the MGRs in the pretest achieved a passing score of 80, while more than 50% of participants in the post-test reached the passing score. Paired t-tests show a significant difference between the mean pre-test and post-test score (t=7.6446, p<0.001).
Training completers versus non-completers
There was no significant difference between the mean pretest score for CLK training completers and non-completers [59.1 (12.8) vs 60.6 (11.1), t= -0.5732, ns] and for MGR training completers and non-completers [51.5 (11.1) vs 51.5 (9.56), t= -0.1318, ns].
Qualitative
A small sample of store managers who completed the training (n=10) were interviewed. Several themes emerged through data analysis.
User experience
A few participants reported having trouble with deploying Internet browsers to conduct the training. For example, a participant had trouble navigating with Internet Explorer, but not with Google Chrome, and suggested that navigation would be hard for individuals who are not familiar with the technology. However, users were instructed to use the latest version of Google Chrome to deploy the modules. Another found it hard to navigate when using his iPhone in landscape mode. One participant did not particularly like the navigation experience, referring to a tobacco industry retailer training program as an alternative. Nevertheless, a majority of participants found that training was easy to navigate and user friendly:
‘It was pretty easy to navigate, I thought it will be a little bit more difficult but was pretty user friendly, which was kind of nice, for change. It was clear and concise; I had no issue listening or reading anything. It was straight forward and was not overly complicated.’ (Participant 1)
Reading, viewing and listening through the module were also considered easy by participants, with no reported negative comments.
In general, participants enjoyed the interactivity in TRAC, stating that they felt the interactive component made the training more engaging and practical:
‘It was really good for the employees, so they know that you are not free to ID people but you do not want to insult them. Right, you do not want to insult the 35-year-old one or whatever and ask them for ID and they get all mad. But it is good because it assured you that number one you have that responsibility and number two you have the authority to do it.’ (Participant 10)
Information provided
Almost all participants found the information provided to be useful, clear, and precise. This was especially true for the information regarding legal responsibilities, compliance with the law, and how to validate government issued identification cards. One participant found the information provided to be overwhelming and confusing, while information regarding the social and health-related impacts of tobacco use was seen as unnecessary by some participants. These participants stated that purchasing tobacco is a personal decision and that the retailer’s job is to sell and not educate consumers on their lifestyle choices. Others viewed the information as helpful, suggesting that people learn from numbers and statistics:
‘I’m positive that probably 90% of my staff would not know that information if they do not take the training. I think it is super important that they are in there, and the way that they laid out was nice too.’ (Participant 1)
Impact on knowledge and daily work
Participants believed that information provided had a positive impact on their knowledge, as most employees were not aware of these details, especially legal obligations and ID validation tips:
‘I have to pick the right one for this, it is kind of get me in. It is not just telling me this but I have actually to do it myself, I liked this. I found that it helps a lot of people learn if they do it themselves.’ (Participant 8)
‘I liked the part about what to look for in the government ID. I was not aware of all of these. It was great to see.’ (Participant 3)
Suggestions
Many participants provided constructive suggestions for the training, including enthusiastically recommending the training to be mandatory for all employees for everyone’s benefit. Some suggested including more pictures and diagrams to improve the education experience. One suggested that the correct answers should be provided to participants immediately after any evaluation questions, to help prepare them for the final summative test. Multiple suggestions were made regarding improving usability across various platforms and operating systems:
‘They give you a lot of resources at the end, you can go to look what type of IDs are acceptable and what to look for. There are some types of IDs that we rarely see. We acutely printed some of these resources and created a document for our cashiers to go to in case someone brings an ID that they never seen before.’ (Participant 7)
How to motivate participation
Several suggestions were provided by participants to help motivate employees to get involved, such as making the training mandatory, offering the training free of charge or paid for by employers, and keeping the training simple. It was also suggested that a participation incentive be provided, such as a certificate at the end of the training. Others emphasized the role of employer engagement as a crucial factor in motivating employees to participate in the training:
‘I think the best way for us to have more people take it is to make it mandatory, because if it is not mandatory people are not going to be forced to take it and people would not take it.’ (Participant 9)
Positives
Interactivity was repeatedly cited as a factor for the positive reaction to the training. Others believed that the simplicity of the reading and clarity of information was a major factor in their positive assessment. Some participants expressed their appreciation for the option to complete the training at their own pace with the ability to review.
Negatives
While most participants did not see negative issues in their experience with training, some cited navigation problems as negatively impacting their experience. Others reported that some of the provided information was not perceived as important such as the health effects of tobacco use and found it to be unnecessary.
DISCUSSION
The TRAC training was found to be acceptable and effective in increasing individual test scores, improving knowledge, and in increasing the percentage of participants reaching the passing score. These results satisfy elements 1 and 2 of the Kirkpatrick and Kirkpatrick9 training program evaluation guidelines regarding user acceptability and knowledge gained. Some challenges were identified with learner navigation of the user interface especially among (older) store managers who were perhaps not as familiar with the technology as were younger clerks. Managers and clerks both expressed some dissatisfaction with the health information included in the training, but their test scores improved significantly after completing this section. Online immersive and interactive training provides an efficient, effective, easily accessible and affordable method of providing training to thousands of retailers.
The TRAC training program was designed to align with the FDA Tobacco Retailer Training Guidance (2018) in keeping with best-practices and published evidence. A literature review was conducted to validate these guidelines within the Canadian context. Canada has implemented restrictions on tobacco retailing that have not been adopted in the US, such as a ban on the retail display of tobacco and a ban on pharmacy sales (with the exception of one province). No comparable online training or e-learning programs which combined best practices in tobacco retailer training, e-learning and adult education was identified in our literature search. Several online training programs for liquor and cannabis retailers that appeared to meet similar criteria were identified; however, we could identify only one such program that was independently certified by a respected training authority. Most of these programs were developed and/or implemented by provincial and state liquor and/or cannabis control authorities for the use of provincial- and state-licensed vendors.
A combination of licensing, enforcement, education, and promotion restrictions at the point-of-sale and the deployment of a well-resourced compliance program to prevent sales to minors is the best-practice approach to tobacco retail regulation3,11. Our literature review included an analysis of compliance factors that determine the likelihood of legal violations such as asking for valid identification, properly validating identification12, refusing to make illegal sales, posting required signage, and providing staff training. The lessons in the training modules were focused on addressing major compliance factors through interactions, simulations, role-playing and other deep learning techniques.
TRAC combines best practices in tobacco retailer training, e-learning and adult education. Fink’s theory of significant learning provides six categories of deep leaning: foundational knowledge, application, integration, human dimension, caring and learning how to learn, and stressing on the interactive nature of those categories which engage participants on different levels creating a lasting an applicable knowledge13-15. Deep learning encompasses different learner centered techniques to encourage participants to use their thinking skills which promote deeper understanding of the subject15. These principals and techniques were implemented in addition to education systems16 in several domains including healthcare17-19, business20,21, and others.
Limitations
The program evaluation focused on the acceptability of the modules among retailers and the impact of the modules in increasing retailer knowledge. The project timelines and evaluation budget did not allow for an evaluation of long-term knowledge retention, retail compliance rates and any potential impact on youth tobacco use.
Further evaluation of long-term outcomes of TRAC training on employee behavior and its impact on reducing tobacco sales to minors is highly recommended to determine the full impact of the training on retail behavior and compliance.
This pilot program should be scaled-up for provincial and statewide implementation and evaluation and it should be accompanied by a certification program for improved learner outcomes that meets independent and rigorous standards for effective e-learning programs.
CONCLUSIONS
The end goal of the TRAC training platform was to support the government of Alberta in fulfilling its commitment to implement tobacco legislation that requires retailers to complete a mandatory training program. The evaluation revealed that participants found the training to be easy to navigate, engaging, informative, and beneficial to their daily work. The participants provided some suggestions to improve the experience such as improving navigation and providing answers to questions immediately after the formative tests.
The major findings of this study are: 1) Online training (e-learning) represents an easily accessible and cost-effective approach for providing thousands of retailers with tobacco regulatory compliance education; 2) Best practices in adult instruction, retailer training and e-learning, including the principles of deep learning, should be applied in the development of online retailer training programs; 3) This pilot program can be scaled up for provincial/statewide implementation and evaluation and it can be accompanied by a certification program to meet independent training standards; 4) A certification program could help to provide the financial resources needed to finance and maintain the training program through modest annual training and certification fees charged to retailers; 5) The training program should be revised and updated regularly, or when a significant change has been made to tobacco laws and regulations; and 6) Further evaluation is recommended to measure the long-term outcomes of TRAC training on employee behavior and its potential influence on reducing tobacco sales to minors and increasing retailer compliance with applicable tobacco legislation.
The TRAC training modules provide an immersive, realistic, and interactive deep-learning experience for tobacco retailers. The modules have produced significant results in improving retailers’ knowledge of tobacco legislation, rationale for compliance and practical steps to maintain compliance.
