Factors impacting the efficiency of triaging ENT (ear, nose, throat) patients in the outpatient setting

Introduction

In the Australian public hospital system, free healthcare is available to Australian citizens and permanent residents (1), and as such, triaging outpatient referrals from general practice is a vital component to prioritising patients according to clinical urgency while maintaining fair access to this finite resource. At Eastern Health, servicing the eastern metropolitan area of Melbourne, triaging of otorhinolaryngology patients is performed by senior ENT (ear, nose, throat) surgeons, requiring approximately 3 hours per week (the time equivalent of an ENT consultant clinical session, or one of the eight scheduled operating lists available to the ENT unit per week). The most time-consuming step in the triage process is the process of rejecting a referral as the senior ENT surgeon is required to provide an explanation for their decision and advice as to how to further advance a referral [usually recommending a general practitioner (GP) provide further information or organise a further investigation prior to re-referral] which is sent as correspondence to the referring doctor. Improved efficiency would result in increased ENT consultant availability for dedicated clinical work and/or registrar training. To aid efficiency, a state wide referral template was introduced on 1 November 2019 and fully implemented on 1 July 2020 (see Appendix 1) (2). The minimum information required for referrals to be accepted for specific ENT conditions is outlined on the Statewide Referral Criteria for Specialist Clinics website run by the Victorian government (https://src.health.vic.gov.au/specialities). This study aims to determine if the introduction of the new state wide referral template for Eastern Health has improved the referral rejection rate and to inform other potential avenues to improve triage efficiency such as the use of artificial intelligence (AI).

The emergence of AI as a possible tool to improve triage efficiency has shown promise in experimental models for Emergency Medicine (3). Natural language processing AI has also shown promise in experimental models in referral pathways for other surgical specialties (4). The possible adoption of AI in various aspects of otorhinolaryngology has also been explored (5) but its use in triaging ENT outpatient referrals has not been established. A staged approach to the potential implementation of AI would be prudent, rather than a wholesale AI automation of the triage process. Therefore, there is a need, in the first instance, to address the greatest area of inefficiency in the triage process, which is referral rejection. Identifying the most common reasons for referral rejection will lay the foundation for AI algorithm development to automate this process, which would include the identification of referrals that do not meet the criteria to be accepted and to advise the referring GP with respect to the need to provide further clinical information/investigations in order to advance a referral. We present this article in accordance with the STROBE reporting checklist (available at https://www.theajo.com/article/view/10.21037/ajo-23-61/rc).

Methods

This study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Eastern Health Human Research Ethics Committee (Reference LR23-067-103601). Because of the retrospective nature of the research, the requirements for informed consent were waived.

This study was a retrospective cohort study of all patients who were referred to the Eastern Health ENT unit, whose referrals were rejected from 1 July 2018 to 30 June 2022. For each patient, data was collected on the reason for referral (ENT condition) according to ENT domain namely, “otology”, “rhinology”, “laryngology”, “head and neck” and “other”, and the reason for referral rejection (reason for rejection).

Using the outpatient referral database at Eastern Health, the reasons for referral rejection were analysed and characterized into different groups, “missing information” and “duplicate referrals”. The “missing information” group was further subcategorized into “missing clinical information”, “missing imaging” [i.e., computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), swallow studies], “missing pathology” (i.e., missing histopathology or cytology for head and neck lesions with suspicious features on imaging), “missing audiogram”, “missing sleep studies”, and “other” groups, which included all referrals that did not include these items, where required by the triaging criteria or where the triaging consultant viewed them to be necessary. At Eastern Health, referrals that included these items, irrespective of recency of the information were accepted by the triaging surgeon. Basic descriptive statistics were employed to identify the most common types of ENT conditions whose referrals were rejected and the most common reasons for referral rejection. A comparison of the number of rejected ENT referrals due to duplications or inadequate patient information as a percentage of total ENT referrals received prior to and since the introduction of the state wide GP referral template was assessed to measure the impact of this referral tool.

Results

The total number of referrals received by Eastern Health declined during every year of the study period from 5,944 to 3,709 (see Table 1). In the two 12 months periods leading up to the full implementation of the state wide referral template, the percentage of rejected referrals due to missing information or duplicates was reduced from 9.6% to 6.8%. However, in the two 12 months periods following the full implementation of the template, the percentage of rejected referrals increased to 8.1% and then 10.1%. In each of the years, missing information was the most common reason for referral rejection, followed by duplicate referrals.

In each of the years, otology conditions (40.2–58.2%) comprised the most common ENT domain represented in rejected referrals, followed by rhinology conditions (18.3–25.4%) (see Table 2).

The most common items of missing information were audiograms (35.3–52.6%), imaging (28.7–44.9%), and missing reason for referral (8.6–14.0%) (see Table 3). The most common missing imaging type was CT scans (56.8–69.9%) (see Table 4).

A minority of referrals (6.5–15.5%) that were initially rejected due to missing information were eventually accepted as outpatients (see Table 5). This proportion was higher in the first 2 years of the study period compared with the second 2 years.

Discussion

The introduction of the state wide referral template does not appear to have resulted in improvements in ENT outpatient triage efficiency at Eastern Health as measured by the rate of referral rejections. Approximately 10% of all referrals were rejected in this study due to missing information or duplicate referrals. Coronavirus disease (COVID) lockdowns in Melbourne, which occurred around the same time as the introduction of the state wide referral template, may have been a confounding factor for this effect, for example, patient access to audiologists or imaging services, or willingness to undergo these investigations may have been reduced due to lockdowns. However, it should be noted that the referral rejection rate increased even further in the final 12 months of the study period, even once COVID lockdowns had passed and COVID vaccinations had become freely available.

Interestingly, the total number of outpatient ENT referrals received declined after the introduction of the statewide referral template, which could suggest that the statewide referral template may act as a disincentive for GPs to refer patients to public hospital ENT services. COVID lockdowns may also have contributed to this noted effect.

This study identified the main types of missing information to be audiograms, imaging and sleep studies, combining for approximately 80% of the missing information group of rejected referrals both prior to and after the introduction of the state wide referral template. Sleep studies, in particular, may have been difficult to access due to limited availability and long wait times to access this service in general, compounded by COVID lockdowns. Also, in Australia, there are often private costs incurred by patients for some imaging modalities. For example, the Medicare rebate for MRIs is limited to only a few specific conditions when requested by a GP. Nevertheless, modifications to the referral template, such as more clearly stating the need to include audiograms for the appropriate otology conditions, or CT scans for specific ENT conditions such as chronic rhinosinusitis, may achieve improved efficiencies, but as previously noted, the information requirements are already widely available on the Statewide Referral Criteria for Specialist Clinics website. Reasons for why there remains a notable proportion of referrals that do not comply with these requirements can be postulated and may include the possibility that referrers are generally unaware of these criteria or find them challenging to interpret. It is possible that some referrers may prefer to act according to direct recommendations for conditions that are unfamiliar to them, and may prefer to be told what investigations to perform, rather than risk being seen as performing an “unnecessary investigation” or investigations associated with potential harm (e.g., radiation associated with CT scans). Further research into referrer behaviour would be an interesting research avenue that could also help contribute to strategies aimed at improving referral triage efficiencies. Furthermore, in order to progress the referral, the referrer needs to check a box in the statewide referral template confirming that all the required information has been included in accordance with the requirements outlined on the Victorian Government website (https://src.health.vic.gov.au/specialities), however, currently the box can be checked without the provision of the required information and/or investigations and there is no mechanism to verify that the referrer has actually included them in their referrals prior to the referrals being received. A system that does not allow a referral to be submitted without the addition of required investigations for particular conditions could be explored, and this study demonstrates a number of possible initial targets (e.g., audiograms for otology conditions, sleep studies for obstructive sleep apnoea, appropriate imaging for specific conditions such as CT scans for specific rhinology conditions such as sinusitis). Such a system would also likely require AI to recognise and identify that the required investigations have not been included as attachments to the referral prior to being submitted and there may be medicolegal issues which arise from using a program run by an ENT service external to the referring GP practice preventing a referral from being submitted prior to it even being received.

This study also demonstrates that another potential strategy to explore is to implement targeted GP education with a campaign concentrated on informing GPs about the need for the specific required investigations to be included for the ENT conditions whose referrals are most commonly rejected.

Duplicate referrals comprised a significant proportion of rejected referrals and may have included referrals that were a result of long wait times, or referrals that were made multiple times by different doctors, or referrals that were sent multiple times by the same doctor as an administrative error. Repeat referrals informing an escalation in patient symptoms or urgency to be seen which were accepted by the triaging ENT surgeon, were not included as a rejected duplicate referral because a new action in the form of triage category escalation would have occurred. Specific data was not collected on the reasons for rejected duplicate referrals which was beyond the scope of this study, as patients with rejected duplicate referrals would have already been triaged and accepted to the outpatient waitlist.

The emergence of AI technologies with natural language processing may also have potential to improve efficiency of the triage process by filtering incomplete referrals or duplicate referrals in the first instance and automating responses to GPs in real-time by instructing them to provide further information or organise investigations prior to re-referral, without the need for the senior ENT surgeon to perform these tasks, which involves providing written correspondence to the GP to provide advice regarding the requirement for further information/investigation prior to being accepted as an outpatient (which may be up to a week or more after the initial referral). This study demonstrates a potential, and at least partial theoretical role for AI in the triage process in an ENT outpatient setting by providing the main targets to identify referrals that should be rejected (i.e., duplicate referrals, missing audiograms, missing imaging, incomplete reason for referral). As rejected referrals are the most time-consuming in triage processing, while being by definition nonurgent, they are a good first target for clinical studies which assess the safety and effectiveness of AI algorithms, testing them against consensus ENT consultant decision-making. This could result in the majority of rejected referrals being managed by AI thus improving triage efficiency. This itself would provide a foundation upon which future steps towards greater automation and efficiency of ENT outpatient referral triaging via AI can be explored.

Finally, only a minority of rejected referrals were eventually accepted as outpatients after initial rejection due to missing information/investigations, which may suggest that the majority of these initially “under-investigated” patients may not require specialist ENT assessment in the first place. It is possible that a proportion of these patients’ subsequent investigations did not demonstrate ENT pathology, or their symptoms may have resolved. It is also possible that they may have sought specialist ENT assessment from alternative health services or from the private sector, where a cost to the patient is usually incurred, and where the requirements for accepting a referral may be less strict. A higher proportion of initially rejected referrals were eventually accepted as outpatients prior to the full implementation of the referral template, however, it is unclear why this may be the case. One reason may be that the earlier a referral was rejected, the more likely it would eventually be accepted as there was more time to organise for missing information/investigations to be completed.

Conclusions

This study demonstrates that the introduction of a statewide referral template did not improve triage efficiency at an ENT service in Victoria. The study also identifies the common reasons for ENT outpatient referral rejection and potential targets for AI algorithm development in automating the process of rejecting referrals due to missing information/investigations and communicating with GPs regarding further steps required in order to progress a patient’s referral. Improving triage efficiency will potentially increase the amount of time consultant ENT surgeons can allocate towards direct patient care or registrar teaching.

Acknowledgments

Funding: None.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://www.theajo.com/article/view/10.21037/ajo-23-61/rc

Data Sharing Statement: Available at https://www.theajo.com/article/view/10.21037/ajo-23-61/dss

Peer Review File: Available at https://www.theajo.com/article/view/10.21037/ajo-23-61/prf

Conflicts of Interest: Both authors have completed the ICMJE uniform disclosure form (available at https://www.theajo.com/article/view/10.21037/ajo-23-61/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Eastern Health Human Research Ethics Committee (Reference LR23-067-103601). Because of the retrospective nature of the research, the requirements for informed consent were waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Willis E, Parry Y. The Australian health care system. In: Willis E, Reynolds L, Keleher H. editors. Understanding the Australian Health Care System. Chatswood, NSW, Australia: Elsevier; 2016:232-77.
2. Statewide Referral Criteria. Available online: https://src.health.vic.gov.au
3. Delshad S, Dontaraju VS, Chengat V. Artificial Intelligence-Based Application Provides Accurate Medical Triage Advice When Compared to Consensus Decisions of Healthcare Providers. Cureus 2021;13:e16956. [Crossref] [PubMed]
4. Krebs B, Nataraj A, McCabe E, et al. Developing a triage predictive model for access to a spinal surgeon using clinical variables and natural language processing of radiology reports. Eur Spine J 2023; [Crossref] [PubMed]
5. Bur AM, Shew M, New J. Artificial Intelligence for the Otolaryngologist: A State of the Art Review. Otolaryngol Head Neck Surg 2019;160:603-11. [Crossref] [PubMed]