INTRODUCTION
Tobacco use is the leading preventable cause of death in the world. Waterpipe tobacco smoking (WTS) is increasing worldwide1, 2, particularly among youth. Accumulating evidence suggests that waterpipe smoking is associated with a number of adverse health outcomes3–5. In Lebanon, the high prevalence among increasingly younger age groups is alarming and calls for immediate intervention. The Global Youth Tobacco Survey (GYTS) conducted in 2011 with a nationally representative sample of youth (13-15 years old) in grades 7-9, indicated that 34.8% of students were current waterpipe smokers with no differences by gender6.
Given the high school enrollment rate at the elementary level (reaching 98% and 81% respectively for age groups 6-11 years and 12-17 years)7, schools seem like an ideal place for intervention. Many school-based tobacco prevention programs have been implemented and evaluated. A recent review by Thomas et al. analyzed 49 randomized controlled trials (RCT) (including over 140,000 school children) aiming to prevent children who had never smoked from becoming smokers. At greater than one year post intervention, intervention effects were significant with an average 12% reduction in smoking initiation compared to the control groups. However, no overall effect was detected at one year or less. The combined social competence and social influences interventions were found to be more effective than other programs. Results of other studies, however, have indicated that school-based programs to prevent and control tobacco use have limited effects in the absence of other community and policy interventions9, 10. In fact, prior studies have acknowledged the role of a multitude of contextual factors in the effectiveness of smoking prevention and control efforts among youth11, 12, highlighting the need for a comprehensive approach13. These ambivalent results come after almost 50 years of tobacco prevention communication, advocacy, and policy regulation in the U.S. and other developing countries.
Tobacco control programs, however, have mostly focused on cigarettes8. To date, knowledge about effectiveness of policy measures to curb the waterpipe epidemic are still nascent.
In Lebanon, a ministerial decree banned smoking in school as far back as 1993. However, enforcement of this policy is variable, amid the overall pro smoking social norms that persist despite passage of a more recent law, Law 174 in 2011 which includes comprehensive bans14, 15. Education about the consequences of tobacco use is a required part of the National curriculum for all schools in Lebanon, but is not implemented comprehensively16.
In countries with nascent tobacco control efforts and strong pro-tobacco social norms (such as in Lebanon), and with an emerging use of the waterpipe, school-based interventions to correct misperceptions and promote skills to resist peer pressure to use may prove effective as one of the approaches contributing to controlling the spread of tobacco use.
With the above in mind, a school-based intervention was developed to prevent or delay initiation of waterpipe use among 6th and 7th graders in Lebanon.
Program evaluation is critical in identifying the extent that a program is effective in achieving its objectives17,18. Confidence in attributing outcomes to the intervention itself and not to extraneous environmental factors is contingent on a thorough process evaluation16, 18–20. Process evaluation is a tool that is used to document whether a specific intervention is implemented as planned, and therefore can guide the understanding of the relationship between the intervention activities and the intervention outcomes18. The aim of this manuscript is to describe the methods and findings of the process evaluation of the school-based intervention.
METHODS
This paper reports on results of a process evaluation of a school-based waterpipe intervention to prevent or delay use in Lebanon by 6th and 7th graders. Process evaluation measures what was delivered (as compared to the intervention plan) and how it was delivered21. A framework for designing and implementing
process interventions in public health has been proposed18, 22 and includes the following basic elements: fidelity – which refers to the extent to which an intervention is delivered as planned22, 23, dose delivered, dose received, reach, satisfaction, and context. Mixed methodology - both quantitative and qualitative - was used to gather data about these elements. The intervention, the school sample, and process evaluation tools are described further below.
Intervention
The intervention consisted of ten sessions focused on preventing or delaying initiation of waterpipe use (Table 1) carried out over 5 months (January - May 2012) in school and during school hours, using a participatory - rather than didactic - approach (such as games, videos, role play, etc.). In order to document the intervention and ensure consistency in delivery between facilitators, each session was described in detail in an intervention manual. The description included objectives and activities, time breakdown, and resources needed for each activity. Session length was limited by the school schedule, each on average taking up 50 minutes. The sessions that intended to enhance skills for the prevention of waterpipe use were designed to be applicable to other forms of tobacco use and other health risk behaviors. In the last session, students and teachers pledged, individually and as a class, to not using any tobacco product for a one year period renewable.
Table 1
Description of intervention sessions
| Session number | Content |
|---|---|
| Sessions 1-4 | Knowledge (health consequences of tobacco use: cigarettes, waterpipe and other forms; second-hand smoke) |
| Session 5 | Media critical analysis focusing on waterpipe tobacco smoking as a topic |
| Sessions 6-7 | Decision making skills/self-efficacy to learn how to make decisions and to identify alternative entertaining activities rather than waterpipe smoking |
| Sessions 8-9 | Refusal skills and social promise* to be confident to refuse waterpipe smoking whenever offered. |
| Session 10 | Presentation of student projects, pledges, and end of program celebration |
School sample
The intervention was implemented in 14 schools (7 public, 7 private) selected randomly from a roster of schools with 6th and 7th grades provided by the Ministry of Education and Higher Education (MEHE). In these intervention schools, 40 classes (17 were 6th grade classes, and 23 were 7th grade classes) participated in the intervention.
Six facilitators were assigned to implement the intervention, and four observers to monitor the process evaluation.
The number of schools each facilitator was responsible for varied, ranging from 2-5 schools. Once the MEHE and principal approvals were obtained, parental written consent and child written assent to participate were requested. The study protocol for implementation and evaluation of this intervention - including the intervention manual and the process evaluation design and forms - were approved by the Institutional Review Board (IRB) at the American University of Beirut.
Process Evaluation Methodology
The plan for process evaluation was guided by the literature18 in addition to experience gained from an earlier project led by authors of this manuscript24. The evaluation plan incorporated data collection tools to measure fidelity, dose delivered, dose received, satisfaction, context, and reach.
Evaluation tools included: session observation forms B and B, a satisfaction form C, attendance log sheets form D, a focus group discussion interview guide, and daily journal entries by facilitators (Table 2). All the data collection forms were pretested during the piloting phase of the intervention and adjustments were made as necessary.
Table 2
Plan of process evaluation
Observation Form A: this form was completed by the facilitator for each session implemented. It tracked number of children who actively participated in the sessions. For a single session, the active participation rate was calculated as ratio of number of actively participating students to the total number of students who attended that session. Then, the average active participation rate was reported. This form also included an overall rating of the dynamics of the session (positive and active/ positive but not active/ not much enthusiasm/ negative). The above two measures were used to indicate dose received.
This form also measured the extent (0-25, 26-50, 51-75 and 76-100%) to which the objectives of the sessions were implemented as planned; an indication of dose delivered.
Form A was also used to measure fidelity. For each session, this form included a table that documented whether each activity was or was not implemented as outlined in the manual. The summative responses across all sessions were used to calculate percentage of activities that were implemented as planned, those that were implemented differently, and those that were not implemented.
Observation Form B: this form was completed by the observer. The project protocol specified that 10% of the sessions were to be observed. Observed sessions were scheduled to ensure that data was collected from all schools, all facilitators, and all sessions. Form B included the same questions as those in Form A. In addition, it asked the observer about the facilitator’s influence on the session (Appropriate and positive/ neutral/ inappropriate and negative) and suggestions to enhance the role of the facilitator.
Satisfaction Form C: this form measured the level of satisfaction of the children with the intervention. At the end of each session, students were asked to draw a smile (happy), a straight line (neutral), or a frown (sad) on already prepared cardboards circle faces expressing how much they liked the sessions. Once the children completed the evaluation, their responses were tallied for each session – resulting in a satisfaction form for each session - and overall.
Attendance log sheet Form D: Facilitators’ tracked attendance of every child at each session as a measure of reach.
Focus group discussions: Focus group discussions were conducted at the conclusion of the intervention to obtain qualitative feedback from students about their satisfaction with the intervention and perceived dose received. The focus group discussions were moderated by research team members who were not involved in intervention delivery. They were all females, graduate students in Public Health and trained on how to conduct focus group discussions. We chose not to record the focus group discussions and a notetaker was assigned to record all discussions that took place. Students were informed about the reason behind doing the focus group discussions and were specifically asked to: list and describe the activities of the intervention that they enjoyed and benefitted from most, to indicate with whom and how they shared the knowledge they gained; and to give feedback on what they would like to see done differently if the intervention was conducted again.
Focus group discussions were conducted in 8 out of the 14 intervention schools, using purposive sampling: at least one in each of the governorates, public and private schools, and grades 6 and 7. After obtaining the principal’s approval, one section/school was chosen to recruit focus group discussion participants. Due to large class sizes around 15 students were randomly selected from each class to participate in the focus group discussion. For 5 of the 8 focus group discussions, a total of 31 students participated (of which 17 are females). Number of participants in the other 3 focus group discussions is not recorded, though it is known that on the day of focus group discussions, the numbers attending the sessions in the 3 classes were 20, 18, and 16, and about 10 to 12 from each class were invited to participate. The focus group discussion lasted for about 30-45 minutes. Focus group discussions were conducted at schools either after the end of the last intervention session or after the post-test. To maintain confidentiality, school staff were asked not to attend. Parental permission and child assent were obtained as part of the original informed consent process. Focus group discussions were conducted during the period of April 7 - May 23, 2012.
Journal entries: A classroom management program (Moodle) was used to develop a forum for sharing and documenting daily experiences in implementation, challenges faced, and successful practices. Facilitators completed the journal entries on the same day they implemented each session. Facilitators were able to access each other’s entries with the goal of sharing experiences and discussion of potential corrective actions needed.
Data Analysis
The forms were completed on a daily basis by each facilitator, and on the same day of observation for each observer. All forms were handed in to the project coordinator to review for completion and follow up on missing data. All statistical analyses were done using SPSS (Version 21). Descriptive statistics including proportions were computed. As for the qualitative data, the focus group discussions’ notes were analyzed by two members of the team based on the interview guide. Journal entries were analyzed for recurring themes to identify contextual barriers that influenced implementation.
RESULTS
Process evaluation results are summarized in Table 3.
Table 3
Results of the process evaluation
| Satisfaction | ||
|---|---|---|
| % of happy faces per session N (%) | Knowledge-based sessions a | Skill-based sessions b |
| Lowest thru 75% | 26 (19.8) | 10 (6.7) |
| 76% thru 95% | 59 (45) | 62 (41.6) |
| 96% thru highest | 46 (35.1) | 77 (51.7) |
| Facilitator c | Observer d | |
| Dose Delivered | ||
| Objectives reported to be implemented as planned, extent reaching, N (%) | ||
| 0-25% | 14 (4.4) | |
| 26-50% | 34 (10.7) | 3 (7.5) |
| 51-75% | 71 (22.3) | 11 (27.5) |
| 76 to 100% | 199 (62.6) | 26 (65.0) |
| Dose received | ||
| Percent of sessions rated as, N (%) | ||
| positive and active | 247 (77.9) | 33 (78.6) |
| positive but not active | 15 (4.7) | 4 (9.5) |
| not much enthusiasm | 18 (5.7) | 5 (11.9) |
| Negative | 37 (11.7) | 0 (0) |
| Average active participation rate, % | 76.6 | 92.6 |
Reach
Parents of 484 students (22.6%) refused their child’s participation. Of the 1658 students whose parents gave their consent, 38 students (2.3%) refused to participate. In intervention schools, 322 sessions were given to a total of 40 classes. Out of all the students who completed the baseline survey (n = 844), 1.5% didn’t attend any session, 5.1% attended less than 50% of the sessions, 6.2% attended between 50 and 74% of the sessions, 21.6% attended between 75 and 89% of the sessions, and 65.6% attended at least 90% of the sessions.
Fidelity
Over all the sections, the facilitators reported 72.3% of the activities to be implemented fully, 16.3% of the activities to be implemented differently than originally planned in the manual and 11.4% not to be implemented (either due to time or to difficulty). For those implemented differently, the main change was in specific activities used to achieve the objective.
Dose Delivered
The percentage of sessions in which over 75% of the objectives were implemented as planned was 62.6% as reported by facilitators, and 64.1% as reported by observers.
Dose received
The percentage of sessions in which children’s participation was rated as ‘positive and active’ was 77.9% by facilitators and 78.6% by observers. In support of this, average active participation rate was reported as 76.6% by facilitators and 92.6% by observers; indicating that the majority of children were involved in the sessions. Facilitators and observers reported percentages did not differ significantly from each other (data not shown).
Satisfaction
A total of 322 sessions were conducted in which 5314 students participated; 280 session satisfaction forms (87%) were submitted in which 5108 (96%) students provided their feedback. These session satisfaction forms reported on the satisfaction of individual students within those sessions. In all sessions combined, 4571 children (89.5%) chose a happy face to express their rating of the session. Furthermore, students were more satisfied with skill-based sessions than knowledge-based sessions: in the skill-based sessions, 91.7% of children chose a happy face. In the knowledge-based sessions 86.7% of children chose a happy face.
Focus group discussion findings
Feedback on intervention content
Focus group discussions results indicated that students were most likely to recall information about the constituents found in cigarettes and waterpipe, the various health effects and diseases caused by tobacco use, methods to prevent peer pressure among friends, effects of passive smoking on health, media influence and misleading advertising of tobacco products. Students also mentioned the various skill building exercises such as the traffic light exercise which taught them how to practice decision making and refusal skills. Students liked the mode of delivery of the intervention especially activities that involved direct interaction; i.e. no smoking pledge, ice breaker activities, experiments, games, exploring the insides of a waterpipe hose, and most of all the role plays, sketches, and videos visuals aids.
Feedback on intervention delivery
Students also mentioned appreciating small details such as the name tags used throughout the program, and the non-smoking pins and certificates of participation distributed on the last day. They valued that their opinion was solicited through the smiley faces. They also appreciated the facilitators’ role, mostly referring to their friendly demeanor and participatory approach in delivering the intervention.
Intervention compared to other programs
When comparing this program to others delivered at their schools, students said that this program has benefits to their health, included a variety of activities, and implemented a non-didactic approach to knowledge and skill delivery which they appreciated. They compared it to other programs, specifically related to tobacco use that did not use these approaches.
Importance of program
When asked about the importance of such an intervention at their age, students insisted that this intervention should target those aged between 10-15 years in order to provide knowledge and skills and prevent tobacco use before it is too late when students will have already fallen into the trap of tobacco use. Others mentioned that they are in their teen years ‘entering life and making decisions regarding their future’ and as such it is a critical period.
“It (the program) is a very important experience for people, and healthy. You need to carry on the program and never discontinue. It has to reach the entire world.” (Grade 7 student).
Personal experience
Students indicated that they had tried to make use of their acquired skills and knowledge with their parents, siblings, relatives, and friends. Some of their personal experiences were shared: some were successful in persuading their parents to decrease their smoking or to smoke only in open spaces, while others expressed their frustration in their inability to convince others to quit, and some were also proud that they were able to refuse offers of smoking cigarette/waterpipe from friends based on skills learned in the intervention.
“Someone suggested that I try the waterpipe, but I refused and told him about its harmful effects that we’ve learned. As I returned home, I informed my sister about this, and she was proud of it.” (Grade 7 student)
Suggestions for improving the intervention
Some participants suggested broadening the intervention to cover other topics of interest such as drugs and alcohol, while others wanted to increase the number and frequency of sessions, proposing that the program be incorporated permanently into the school curriculum or alternatively outside school grounds to allow for more flexibility in delivery. Students expressed interest in conducting social awareness campaigns at school, creating anti-smoking advertisements, and distributing educational brochures on the streets of their villages and towns and in other schools. Expanding the intervention to have a social media component was also felt to have impact on a larger audience. Students recommended that this program involve younger age groups, other schools in Lebanon, and particularly parents.
Findings from daily journal entries
The journals documented a variety of contextual factors that affected implementation. These were categorized into factors at the country level, at the school level, and related to social norms around waterpipe smoking. At the country level, during the time of the intervention, political unrest made it difficult to reach particular schools, or schools being closed for a period of time. This posed a challenge for reach, as well as dose received, as waterpipe smoking as a topic sometimes seemed trivial to discuss amidst daily living struggles and concerns of students.
In addition, schools were spread all over Lebanon and some took up to 2 ½ hours to reach. Transportation was costly, resulting in the need to pool resources for facilitators to share rides. This made scheduling with the schools for intervention delivery cumbersome. In areas that were the furthest from the capital city, Arabic dialects are also different, and issues with comprehension were faced, also influencing dose received.
At the school level, the nature of school based interventions requires that they be implemented in the scholastic year. The academic year in which we implemented the intervention (2012) witnessed an unprecedented number of teacher strikes as part of a country wide syndicate request to improve compensation standards, which led to disruption of curriculum delivery. This made it difficult to request sessions to deliver the intervention within an already cramped curriculum. Sometimes, principals did not abide with the schedule and hence facilitators would arrive at the school only to be told to return another day. Teachers resented that we took up class time for ‘important’ subjects (math, sciences, language) so they often gave up the sessions which they consider less important (art, music, physical education) but however those were considered the most fun by students. All these issues interfered with dose delivered. Facilitators were also faced with psychosocial problems among students and their families, including drug use, extreme poverty, and others that affected dose received.
The schools also often had structural barriers to adequate implementation such as lack of heating, lack of electricity, small classroom sizes, and lack of classroom resources (LCD, laptops …). The team compensated for this through innovation. As one example, a video depicting the expansion of second hand smoke (SHS) between one side of a room and another could not be shown due to lack of an LCD and electricity. Facilitators sprayed perfume on one side of the room and asked students to determine when it reached the other side, to mimic the effect of SHS. Generally, participating schools still teach in the traditional way: instruction is uni-directional, memorization is the key ingredient, and strict adherence to whatever the teacher/principal/etc. says is the requirement. This made more participatory modes of learning difficult for the students to engage with and often created chaos in the classroom. This affected both dose delivered and received.
Finally, the wider social acceptance of waterpipe smoking particularly among teachers and parents likely compromised dose received. Students told us of school trips where they smoked waterpipes. Despite a policy prohibiting smoking in schools and a national comprehensive ban of indoor smoking issued in 2011, many schools allow smoking on their premises. This makes an attempt to message around tobacco difficult.
The above contextual challenges resulted in delays in implementation which affected fidelity. Facilitators had to reorganize sessions to deliver the same content with less time. The original plan was to implement one session per visit. The team ended up delivering 322 sessions in total, with a range of 6-10 visits per school, often combining sessions. In addition, these challenges resulted in the need for more human resources than originally planned and budgeted. To compensate for this shortcoming, MPH student volunteers were recruited and trained to support the facilitators.
DISCUSSION
This paper reports the results of a process evaluation of a waterpipe prevention intervention implemented among 6th and 7th graders in Lebanon. To our knowledge, this is the first waterpipe specific intervention to report on process evaluation globally. Process evaluation data - and particularly the Moodle journal entries - were analyzed concurrent with intervention implementation allowing adjustments to be made during delivery of the intervention. Process evaluation is critical in enhancing validity of intervention effect pathways by providing knowledge of what components were related to success, providing feedback on the quality of the intervention, identifying strengths and weaknesses, and documenting implementation22. Process evaluation ideally precedes any evaluation of effectiveness and paves the way for a more in depth understanding of the pathways to impact18.
The results of this process evaluation suggest a high fidelity intervention with high participant satisfaction, enhancing the probability that the intervention will be effective. Overall, findings reflected that students valued the program. Students were particularly satisfied with skill-building rather than knowledge-focused sessions and with the non-didactic nature of intervention delivery.
The biggest challenge to implementation was the context. As described by Saunders et al., the context includes ‘aspects of the physical, social, and political environment and how they impact implementation18. Less than two thirds of the sessions were implemented as originally planned. Facilitators reviewed and combined activities to ensure that the objectives of each session were preserved. The school-based contextual challenges that we faced - such as traditional teaching methods, and the family issues that students were dealing with - also suggest the need for a multi-disciplinary academic and field team for school-based interventions in countries/regions where schools may not be equipped to meet the learning and socio-emotional needs of students. Even ‘health-specific’ interventions need teams that include a psychosocial counselor to guide action related to issues faced by students and their families to which the intervention team may be exposed; an education specialist to guide on educational techniques and tools given the normative teaching methodologies; and ‘local’ facilitators where possible to ensure language dialects are most appropriate; among others. The environmental issues suggest the need for a more holistic intervention that tackles aspects of the socio-normative environment of tobacco use. The socio-political issues faced indicate the need for utmost flexibility in timelines for implementation which may affect robustness of original research designs.
The quantitative and qualitative results of this process evaluation resulted in ‘version 2’ of the intervention manual, which includes 8 – rather than 10 – sessions. This change reflects the feedback of students, as well as the reality of uncertainties on the ground. More importantly, contextual issues brought up a variety of questions for critical discussion. These include but are not limited to:
Is a school-based intervention for students the most relevant in a context where community support for a negative behavior is high? In a context of low human and financial resources, should public health researchers, practitioners, and advocates focus their efforts on higher levels (policy) of the ecological model rather than lower levels (individual change)25?
How do public health practitioners who value participatory consensus-building non-didactic approaches work in systems where top down traditional approaches prevail? How do students who are used to such traditional instructional methods interact with a short term change in approach? And does this change in approach, since it is short term, do more good or more harm?
In context of high uncertainty and/or of extreme disadvantage, should interventions be focused on social, structural, and political determinants of health rather than on a particular health outcome? Irrefutable evidence now points to the power of social determinants to change health status overall26, 27. Is any other approach just a band-aid solution?
What are the benefits for young persons of exposure to a different approach in ‘education’; to content and skills building around a particular health issue; to interaction with university academics, research assistants, and students - that may go beyond the specific sessions and provide access to a set of skills that empower, a window to a different world and different visions and options for the future? And do these benefits somewhat neutralize the concerns noted above?
More dialogue and analysis are needed in the public health community around these types of questions. The experience of this intervention confirms the critical importance of context in program implementation21. We argue that context related challenges affecting implementation are the norm rather than the exception in complex social interventions. Just as debate surrounding whether RCT is the most appropriate to evaluate impact of complex intervention28,29, we suggest - in lines with others16,18,22 - that frameworks that sterilize process evaluation to doses received, delivered, and satisfaction miss out on critical information regarding the contextual situation that provides rich fodder for learning about implementation challenges.
Strengths and limitations
This manuscript reports the results of a robust and comprehensive assessment of the extent of implementation of a school-based intervention to prevent and delay waterpipe smoking in Lebanon. Providing evidence of implementation is rare in our region, despite its critical importance in linking impact to specific interventions. The limitations are a function of the exact context which we describe as intervening in ideal intervention implementation. On the ground, our ability to assess reach, dose delivered, dose received, fidelity and satisfaction was affected by conflict, school personnel attitudes that prevented sessions from being given as planned and participant autonomy in choosing not to fill out satisfaction smileys.
In response, we improvised early on and developed the journal entries which served to document daily challenges and limitations to implementing the intervention as planned. Despite limitations, we believe that we have been able to extensively document the intervention with rigor and provide a model – in line with current scholarship - to guide further implementation research.
Conclusions and Lessons Learned
Implementation challenges and process evaluation of this intervention has been a rich source of many valuable insights that can feed into the agenda of other evaluators and planners. Experience from this intervention has shown that - despite strong social norms promotive of waterpipe tobacco smoking - interventions in schools can be somewhat effective to control use but are not sufficient. Also, in addition to thematic content, school-based intervention programs may need to consider and adapt to traditional modes of learning; and in low resources contexts, be prepared to respond to various student encountered needs by bringing together a multidisciplinary team including education specialists, psychological counselors, and others.
