Australian otolaryngology, head and neck surgery trainee education for contemporary airway management of obstructive sleep apnoea: a pilot randomised controlled study

Introduction

Obstructive sleep apnoea (OSA) has recently been estimated to affect one billion people worldwide (1). Untreated moderate-severe OSA can have significant consequences on health, and is associated with increased risk of insulin resistance, dyslipidaemia, cardiovascular disease and premature death (2). The traditional view that all patients should be treated with Continuous Positive Airway Pressure (CPAP) alone, has been challenged by evidence of high prevalence (up to 50%) of CPAP and other device failure (3), as well as rigorously executed trials supporting salvage surgery options (4,5). Assessing a patient’s candidacy for benefit of sleep apnoea surgery requires refined clinical assessment and judgement, identification of surgically correctable sites of airway obstruction via static and dynamic examination, and nuanced surgical decision making (6).

Soft tissue surgical management of OSA is performed by Otolaryngology, Head and Neck Surgeons (OHNS) [also known as Ear, Nose & Throat (ENT) Surgeons. Up to one-third of patients with clinical features of OSA are referred to OHNS for primary diagnosis and treatment (7), and many clinicians have equipoise as to how to manage those already diagnosed and unable to tolerate CPAP (8).

In Australia, OHNS trainees spend five years in formal, accredited training posts under the guidance of the Australian Society of Otolaryngology Head and Neck Surgeons (ASOHNS) and the Royal Australasian College of Surgeons (RACS). There is currently no formal curriculum for training or assessment of competency of OHNS in the management of complex OSA patients. Australian OHNS trainee confidence, knowledge and exposure to complex multi-level OSA surgery is lacking (6). Similar findings have been described in OHNS Residency programs within the United States, with low confidence of trainees in performing multilevel surgery (9,10), a highly disparate educational experience regarding sleep medicine/sleep surgery across different training sites (11) and lack of dedicated specialist sleep surgeons involved in training (12). Lack of exposure to sufficient complex OSA surgery case load has been identified as a major weakness in training within a recently published international survey (13). These concerning findings have led to a “call to arms” aiming to increase the quality of, and access to, sleep surgery training for OHNS trainees (14). There have been randomised-controlled studies looking and different teaching modalities in medical education (15), however, there is a paucity of such studies in sleep surgery literature.

This study was a randomized controlled trial evaluating the effect of Australian OHNS trainee exposure to education materials compared with no exposure, on sleep surgery specific examination performance (multiple choice and short written answer).

Methods

Ethics approval and participant consent

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study protocol and methods were reviewed and approved by the Health and Medical Human Research Ethics Committee at the University of Wollongong (Reference: 2019/ETH13470). Written informed consent was obtained from each participant.

Study participants

In the Australian OHNS training program there are 70 active accredited trainees (at the time of the study). ASOHNS administers the training. With the assistance of ASOHNS all active Australian OHNS accredited trainees were invited via email (containing project description and rationale) to participate in the study.

Within the Australian hospital system are dedicated OHNS positions for junior medical staff with aspirations of pursuing a career in OHNS. These “unaccredited” positions are competitive to secure and generally last for 12 months within an OHNS unit. Some unaccredited trainees work for several years in these positions before being accepted into the formal training program. Using hospital contacts in Australia, 45 unaccredited trainees were invited to participate.

There is one formal Sleep Surgery Fellowship in Australia. This expert training is dedicated to the field of sleep surgery and undertaken following completion of the OHNS FRACS program. Fellows spend 12 months in the program in Wollongong, New South Wales. To date, there are five OHNS in Australia who have completed this Sleep Surgery Fellowship program (including the primary author). The four others were invited to participate in the study as an additional comparator group.

Study design

After study participants’ consent was confirmed, participants were randomly allocated using Random.Org© software and assigned to sleep surgery educational material exposure or no exposure to those materials.

Each participant was then randomly assigned a unique “participant identification number” initially concealed to the primary investigator. This list was then provided to the primary investigator for distribution of materials.

Each participant was contacted via their provided primary email address to inform them of which group they had been assigned. At this stage, those randomized to the exposure group were provided an online link to the educational material. A period of 2 weeks was set as the exposure time prior to the online examination.

Following the exposure period, each participant was then asked to complete an online examination, consisting of:

• 40 multiple choice questions (MCQ) (40 min);
• 1 short answer (20 min).

The de-identified results were then collated and analysed in relevant groups by the primary author. The multiple-choice exam was scored within the ExamBuilder© program. The written exam was scored by international sleep surgery expert (SGM), being blinded to participant identity.

Intervention—Sleep Surgery Educational material

Those participants randomized to educational material exposure were emailed access to a secure, password protected, cloud-based data folder consisting of the following material (designed by the primary author and International SGM):

• 18 key sleep surgery scientific publications (see Appendix 1);
• Sleep Surgery Textbook (16);
• Access to the International Surgical Sleep Society Online Lecture Series (https://surgicalsleep.org);
• Sleep Surgery Dissection Manual (available from corresponding author).

Those randomized to the “no exposure” group were advised avoidance of sleep surgery specific study/reading outside their normal day-to-day learnings in the workplace and were excluded from access to the exposure education material.

There was a 2-week period of material exposure time before sitting of the online examination.

Examinations

Each participant was asked to complete an online examination with a 1-hour time limit.

The examination consisted of 2 components:

• 40 multiple choice questions: exam generated using ExamBuilder© program. Each question contained 4 possible answers (A, B, C, D). Questions compiled by primary author and sleep surgery expert (NH/SGM) to a level expected of OHNS trainees.
  Each participant was emailed an individual link to the exam, and scores stored within the program until all participants had completed (exam was open for completion for 72 hours; 6 pm Friday 6^th March 2020 until 6 pm Monday 9^th of March 2020).
• 1 Short Answer Question: designed for completion in 20 min. Completed as a Microsoft Word© document, participants were asked to include their participant identification number and return completed question to the primary author.
  The de-identified answers were then forwarded to an International SGM for marking. The SGM was blinded to participant identity and level of training. The participant ID numbers were randomly allocated by the primary author on participant registration.

Statistical analysis

All statistical analysis was undertaken using IBM SPSS Statistics Version 25 (Endwark, NY, USA).

Descriptive data are presented using mean and standard deviations. Differences between exposure and control group means were tested using independent samples t-tests, P values below 0.05 were regarded as statistically significant. Subgroup descriptive statistics (level of accreditation) are presented using mean and standard deviations. Mean test scores for sleep fellowship trained OHNS are presented as a separate exploratory comparator.

Sample size for this study was relatively small, and limited by the amount of OHNS trainees in Australia. A larger cohort could be recruited in subsequent studies involving multiple centres on international scale.

Results

One hundred and fifteen eligible participants were approached for recruitment. Sixty-nine were unable to participate or had previous exposure to relevant teaching materials. Forty-six entered the randomization procedure, of these 24 were allocated to exposure and 22 to control. Eight participants in the exposure group were lost to follow up and 5 in the control group. Thus, there were 16 and 17 participants who attempted the examination in the exposure and control groups respectively. In both the exposure and control group 100% of participants completed the online multiple-choice examination. In the exposure group 14 (87.5%) participants completed the written exam, 13 (76.5%) completed it in the control group (Figure 1).

Figure 1 Flow diagram.

The mean multiple-choice exam test score (expressed as % correct) was 52.63±10.54 in the exposure group and 56.00±12.66 in the control group [mean difference 1.3±1.6 (3.3%), P=0.41]. The mean written exam test score (expressed as % correct) was 58.57±14.89 in the exposure group and 49.42±15.82 in the control group [mean difference 1.8±1.2 (9.0%), P=0.14] (Table 1, Figure 2).

Figure 2 Test score distributions (mean with standard deviation).

The cohort comprised 17 accredited OHNS trainees and 16 non-accredited OHNS trainees. The mean multiple-choice exam test score (expressed as % correct) was 55.29±12.78 in accredited trainees and 53.38±10.58 in non-accredited trainees [mean difference 0.8±1.6 (2.0%), P=0.64]. The mean written exam test score (expressed as % correct) was 60.96 ±11.84 in accredited trainees and 47.86±16.67 in non-accredited trainees [mean difference 2.6±1.1 (13.0%), P=0.03] (Table 2, Figure 3). The mean test score in a separate exploratory group of 2 sleep fellowship trained OHNS was 80.50% in the multiple-choice exam and 76.25% in the written exam (Figure 3).

Figure 3 Subgroup test score distributions (mean with standard deviation).

Discussion

This study’s methodology is unique in both the sleep surgery and general OHNS literature. It assessed the effect of educational material exposure on trainee performance. Whilst this study was not adequately powered to confirm statistical significance of the observed effect of educational material exposure on test performance, several aspects are worthy of discussion. The study might act as a framework for further investigations into the most effective way to educate OHNS trainees and ultimately optimize patient outcomes.

In general, trainees (both accredited and non-accredited) performed below expectations in the examinations. Within the ASOHNS curriculum, there is no outline or expected minimum level of knowledge relating to sleep surgery for OHNS trainees. However, the authors believe the questions fairly represent knowledge that should be baseline for surgeons undertaking practice that will involve assessing and treating OSA patients. For all groups analysed in both the multiple choice and written exam, 52.8% and 58% of questions respectively were answered correctly. This low baseline score could be used as a comparator for the effectiveness of future OHNS sleep surgery education within the training program.

The sub-group analysis showed no significant difference between scores in the accredited and non-accredited groups for the multiple-choice exam. Given accredited trainees are part of a structured and formal educational program administered by ASOHNS—it is expected they would outperform the unaccredited trainees. There was however, an observed superior performance in the written exam by the accredited trainees. The lack of difference noted between the accredited and non-accredited trainees in the multiple-choice exam may reflect some of the shortcomings described within educational literature in using multiple-choice exams. This form of examination is known to examine more factual/“recall” type knowledge, and not focus on cognitive skills or problem solving (17). Whilst it is acknowledged no exam format is perfect, it may be that written answer style examinations are better at differentiating higher order knowledge (18). This observation was noted in our study, with the written examination generating a greater mean spread of scores amongst candidates.

Despite the stated shortcomings of testing, the lack of difference between accredited and non-accredited trainees could be a source of concern and further sleep surgery education required within the Australian OHNS program to ensure trainees are adequately trained in sleep surgery prior to undertaking independent practice.

In keeping with the concerns raised by Lam et al. (11) in relation to US training, Australian OHNS trainees have highly disparate experiences with sleep surgery depending on which hospital rotations they are allocated. Some trainees will undertake a term (normally 6–12 months) at a centre where they work under consultant(s) who have an interest, and are trained in contemporary upper airway surgery for OSA and snoring. These trainees will be involved in the comprehensive work-up, and surgical management of these patients. Other trainees, will not gain such exposure throughout the duration of their training. This heterogeneity of training is not easy to address, as not all trainees are able to rotate through all hospitals. The importance of a formal curriculum for sleep surgery targeting not only trainees, but consultant surgeons also, is highlighted.

Two participants qualified in OHNS with further formal post-fellowship training in sleep surgery completed the examination. These surgeons were not exposed to the educational material prior to undertaking the examination. In both the multiple choice and written exam performance by this group was superior to the OHNS trainee groups. Given the small sample size, few statistically robust conclusions can be made, however, there appears to be superiority of knowledge in these candidates (which would be expected given their level of training). Surgeons that have completed the general OHNS training program may not have the experience and/or confidence to address multiple levels of airway anatomy in order to treat severe OSA and complex cases. Kezirian et al. previously reported a reluctance to perform surgery for OSA amongst American general OHNS surgeons based on limited training (19). There is evidence showing a combination of procedures (modified uvulopalatopharyngoplasty and coblation channelling of the tongue) in OSA patients who have failed medical management is efficacious and safe (4). Further research is required to clearly identify patients and combinations of upper airway reconstructive procedures that can be competently performed by surgeons without sub-specialty training in sleep surgery.

This study has limitations. By the nature of OHNS training in Australia, the study has a relatively small number of participants and is not adequately powered to confirm statistical significance for the performance differences between exposure groups. Within the study, it was not possible to analyse the time each trainee in the “exposure” group spent reading the material. It was also not possible to guarantee participants in the “no exposure” group did not do some reading/study prior to the exams. There was also a cohort of participants who completed the MCQ exam, but did not submit the written component—this meant each exam type was analysed independently, where the pre-study aim was to combine the scores for an overall mark.

Short term educational intervention may not be sufficient training in complex and nuanced decision making required for OSA patients. Formal post-fellowship training remains the current best method. Further research is required to identify the best ways possible to educate OHNS trainees in sleep surgery.

Acknowledgments

Funding: The study was supported by Illawarra Health and Medical Research Institute Grant Funding 2019.

Footnote

Data Sharing Statement: Available at https://ajo.amegroups.com/article/view/10.21037/ajo-21-23/dss

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://ajo.amegroups.com/article/view/10.21037/ajo-21-23/coif). This study received Illawarra Health and Medical Research Institute Grant Funding 2019. SGM serves as an unpaid editorial board member of Australian Journal of Otolaryngology from January 2021 to December 2022. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study protocol and methods were reviewed and approved by the Health and Medical Human Research Ethics Committee at the University of Wollongong (Reference: 2019/ETH13470). Written informed consent was obtained from each participant.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Benjafield AV, Ayas NT, Eastwood PR, et al. Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis. Lancet Respir Med 2019;7:687-98. [Crossref]
2. Strollo PJ Jr, Soose RJ, Maurer JT, et al. Upper-airway stimulation for obstructive sleep apnea. N Engl J Med 2014;370:139-49. [Crossref]
3. Weaver TE, Grunstein RR. Adherence to continuous positive airway pressure therapy: the challenge to effective treatment. Proc Am Thorac Soc 2008;5:173-8. [Crossref]
4. MacKay S, Carney AS, Catcheside PG, et al. Effect of Multilevel Upper Airway Surgery vs Medical Management on the Apnea-Hypopnea Index and Patient-Reported Daytime Sleepiness Among Patients With Moderate or Severe Obstructive Sleep Apnea: The SAMS Randomized Clinical Trial. JAMA 2020;324:1168-79. [Crossref]
5. Browaldh N, Nerfeldt P, Lysdahl M, et al. SKUP3 randomised controlled trial: polysomnographic results after uvulopalatopharyngoplasty in selected patients with obstructive sleep apnoea. Thorax 2013;68:846-53. [Crossref]
6. Mackay S, Holmes S, Jones A. Australian training for contemporary airway management of obstructive sleep apnoea in ENT surgery: current status and future recommendations. J Laryngol Otol 2015;129:S8-15. [Crossref]
7. Kezirian EJ, Maselli J, Vittinghoff E, et al. Obstructive sleep apnea surgery practice patterns in the United States: 2000 to 2006. Otolaryngol Head Neck Surg 2010;143:441-7. [Crossref]
8. Field CJ, Robinson S, Mackay S, et al. Clinical equipoise in sleep surgery: investigating clinical trial targets. Otolaryngol Head Neck Surg 2011;145:347-53. [Crossref]
9. Sim MW, Stanley JJ. Trends in otolaryngology residency training in the surgical treatment of obstructive sleep apnea. Laryngoscope 2014;124:579-82. [Crossref]
10. Shen T, Shimahara E, Cheng J, et al. Sleep medicine clinical and surgical training during otolaryngology residency: a national survey of otolaryngology residency programs. Otolaryngol Head Neck Surg 2011;145:1043-8. [Crossref]
11. Lam AS, Wise SK, Dedhia RC. Practice Patterns of Sleep Otolaryngologists at Training Institutions in the United States. Otolaryngol Head Neck Surg 2017;156:1025-31. [Crossref]
12. Gouveia CJ, Kern RC, Liu SY, et al. The state of academic sleep surgery: A survey of United States residency and fellowship programs. Laryngoscope 2017;127:2423-8. [Crossref]
13. Cammaroto G, Bianchi G, Zhang H, et al. Sleep medicine in otolaryngology units: an international survey. Sleep Breath 2021;25:2141-52. [Crossref]
14. Dedhia RC, Yaremchuk KL. The uncertain fate of otolaryngology in sleep medicine. Otolaryngol Head Neck Surg 2015;152:381-2. [Crossref]
15. Ilic D, Nordin RB, Glasziou P, et al. A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ 2015;15:39. [Crossref]
16. Friedman M, Jacobowitz O. Sleep Apnea and Snoring: Surgical and Non-Surgical Therapy (2nd Edition): Elsevier, 2019.
17. Azer SA. Assessment in a problem-based learning course: twelve tips for constructing multiple choice questions that test students' cognitive skills. Biochemistry and Molecular Biology Education 2009;31:428-34. [Crossref]
18. Brown H, Abeywickrama P. editors. Language assessment: Principles and classroom practices. 3rd ed. White Plains, New York: Pearson Longman; 2019.
19. Kezirian EJ, Hussey HM, Brietzke SE, et al. Hypopharyngeal surgery in obstructive sleep apnea: practice patterns, perceptions, and attitudes. Otolaryngol Head Neck Surg 2012;147:964-71. [Crossref]