Trends in the quality & quantity of otorhinolaryngology research in Australia: 1989–2018

Introduction

Otorhinolaryngology [ear, nose & throat (ENT)] surgery first gained recognition as a single surgical specialty in the early 20^th century when the separate fields of otology and laryngology amalgamated (1). Since that time, the specialty has seen considerable scientific and clinical advancement due largely to the previous and ongoing research of the professions academics and clinicians. Nowadays, over 2 million articles are published annually in biomedical research journals worldwide (2).

The contribution of all surgical specialists to the academic landscape, however, has previously been questioned. The Lancet a likened clinical surgical research to a “comic opera” in 1990, lamenting the inherent poor quality of their published evidence. Instead, surgeons of the day were seen to prioritise technical proficiency and management, while quality assurance and analysis of treatment efficacy was shrouded under the veil of “audit” (3). In the decades to follow, however, there has been a large cultural shift to squash this previously archaic ethos, with a significant increase in both the quality and quantity of surgical research (4). The Royal Australasian College of Surgeons (RACS) embodies this transition to an academic inclusive philosophy, defining one if its core competencies as “scholarship and the teacher” (5). Trainee’s on the Surgical Education and Training (SET) programme are required to author their own research projects as part of their training requirements.

Indeed, evidence-based medicine (EBM) is now seen as the “gold standard” by which university degrees and training programs form the foundations of their teaching, and more recently by which public bodies make financial decisions about service provision (6).

The challenge today exists in being able to sift through and critically appraise the sheer quantity of publications to apply a hierarchical classification to its quality (i.e., levels of evidence). This stratification allows surgeons to appropriately translate research into individual practise, all in the intent to improve patient care and outcomes. Otorhinolaryngology, albeit in its relative infancy, has been shown to compare favourably to other surgical specialties in its influence to academic advancement both in the number as well as the quality of their published research worldwide (7). What isn’t known, however, is the extent to which the Australian ENT community contributes to this published literature. The aim of this bibliometric analysis was to evaluate Australian otorhinolaryngological research over the last 30 years and to detect trends in both the quantity and quality of its output.

Methods

The 24 highest ranked otorhinolaryngology journals worldwide were identified. This was by means of their “impact factor” (IF); a measurement representing the frequency at which an article in a journal has been cited in any particular year, and thus, indirectly, its relative importance amongst other peer reviewed journals (8). For each article, the journal IF was chosen for the year prior to publication as this represents the standing of the journal at the time of article submission. Journals not wholly dedicated to otorhinolaryngology surgery (e.g., audiological, dental/maxillofacial, sleep) were excluded.

A literature search was then conducted for the years between 1989–present [2018] using Ovid MEDLINE/PubMed. Search term included the name of each respective “journal”, “AND”, “Australia” in the “affiliation” field. The result of this search produced all published articles with an Australian affiliation (including hospital, health institution or university) dating back to 1989.

Excluded from this analysis were articles deemed to be “non-clinical”. This comprised of those associated with laboratory and/or animal-based research, editorials, letters to the editor, cost analysis, post mortem/cadaveric studies, historical articles and those related to cost effectiveness. Any study related to research with direct implications on “patient care” were included (including surgical simulation) as were International multicentre studies where the Australian affiliation was not the first author. Literature reviews were included in data collection as “clinical”, however were appropriately differentiated from systematic reviews.

Screening of all titles and abstracts was conducted by the first author and data was input into an Excel based spreadsheet. This method of single author data collection was adopted to avoid inherent trans-author conflict in interpretation, therefore ensuring consistency in data input. The following information was included in the collection spreadsheet: article title, journal name, number of authors, year of publication, type of publication, level of evidence (LOE), whether it was clinical or nonclinical, subspecialty, journal IF, whether it was multinational, and the number of institutions involved. The IF for the year prior to publication was assigned to each article as this represented the reputation of the journal at the year of submission.

Each abstract was read and analysed by the first author and a number corresponding to its LOE was assigned in reference to the “Oxford Centre of Evidence-Based Medicine” (Table 1) (9). Evidence levels ranged from 1 (highest quality), which represented systematic reviews and meta-analysis of randomised controlled trials (RCTs) & RCTs respectively, to 5 (lowest quality), corresponding to “expert opinion”. If the information provided in the abstract was scarce and did not allow for adequate analysis, the whole article was sourced and LOE determined. At regular intervals, articles with a contentious or unclear LOE were discussed and clarified with the supervising author. Clinical articles such as case reports and literature reviews that did not pertain to a specific LOE were given an “NA”, however, also included in data collection.

Once all data entry was complete, statistical analysis was achieved with the SPSS programme (10).

Results

A total of 1,384 articles were reviewed from the literature search for the 30 years between 1989–2018. This figure was narrowed to 1,095 articles that satisfied the inclusion criteria specified above. There has been an exponential increase in the quantity of clinical research throughout the study period from 42 Australian articles published in the 5 years between 1989–1993, to 453 Australian articles between 2014–present (P<0.001) (Figure 1). This trend was mirrored by the quantity of authors per paper, increasing from a mean of 3.2 between 1989–1998, to 5.3 between 2009–2018 (P<0.001).

Figure 1 Annual number of published articles: 1989–2018 (figure updated).

The 5 most common journals with the highest quantity of published Australian based research included the Journal of Laryngology and Otology (n=197), Laryngoscope (n=147), Head & Neck (n=148), International Journal of Pediatric Otorhinolaryngology (n=134), and Otolaryngology Head & Neck Surgery (n=107); collectively accounting for 66.9% of all publications (Table 2).

Head and neck constituted the greatest majority of subspecialty research (30%), closely followed by rhinology (24%) and otology (23%). Airway/laryngology, base of skull/neuro-otology and general ENT topics rounded off the remaining percentage (Figure 2). Paediatric & indigenous based publications were inclusive within subspecialty classifications.

Figure 2 Publications by subspecialty. BOS, base of skull.

In the 30 years between 1989–2018, 45 (4.1%) articles constituted level 1 evidence as per the “Oxford Centre of Evidence-Based Medicine” (9). This included 36 RCTs (3.3%), as well as eight systematic reviews where RCTs formed the majority of included articles for review. Level 2 evidence accounted for 121 (11.1%) manuscripts and was inclusive of mainly systematic reviews of cohort studies (with internal control group), as well as individual prospective cohort studies themselves. The remaining 85.6% of Australian published literature was between levels 3–5 as well as those articles deemed clinical, however, not appropriately fulfilling criteria to earn a LOE, termed “NA”. Level 4 evidence (prospective or retrospective case series), accounted for the largest quota of all collected study designs, counted at 535 articles (48.9%).

Figure 3 demonstrates the trends in the published literature by LOE in the study period. There was a significant increase in published high quality research with 77.8% of all level 1 research reaching print in the years between 2009–2018. The remaining 22.2% of level 1 research was published in the 10 years prior, 1999–2008, whilst no such quality evidence was produced before 1999. A similar trend was recognised in publications of level 2 studies as well, 78.5% of which were published between 2009–2018, while only 18.2%, and 3.3% were published in the prior two, 10-year periods.

Figure 3 Number of articles by level of evidence per decade (figure updated).

This increase in quality of evidence closely parallels the similar growth in pure quantity of published research as stated prior. Of more relevance, however, is that there has been a relative increase in the percentage of levels 1 and 2 evidence compared to the amount of research published per 10-year period. Level 1 quality research represented 3.9% of publications for the period between 1999–2008, compared to 4.9% for the next and most recent 10 years between 2009–2018 (P=0.022). There has been no such statistically significant for levels 4 & 5 evidence in these same time periods.

Multi-institutional and multinational collaboration in ENT research is also on the rise. Prior to 2009, there was no more than seven publications in any given year whose authors and/or institutions were from different countries. This collaboration, however, has exponentially risen, most notably in the last 5 years, with a mean number of 41.0 multinational articles published per year, 43.9% of which with an Australian author/institution as the primary author. This increasing in multinational publications both with and without an Australian institution as primary author were both statistically significant, P<0.001.

In the last 3 decades, certain authors have made a profound contribution to the body of Australian otorhinolaryngology literature. Of significant note include Peter-John Wormald; and Richard J. Harvey; who have contributed towards 168 and 88 publications respectively from the included Journals within this study. Other authors publishing extensively in the analysed journals include Raymond Sacks (n=67); Alkis James Psaltis (n=46); Stephen O’Leary (n=27); Bruce Benjamin (n=27); & Ben Panizza (n=23). These contributions are furthermore reflected in the institutions from which they come; with a substantial portion of the published articles coming from a combination of St Vincent’s Hospital, Sydney; The University of NSW; Macquarie University, NSW; as well as The University of Adelaide & the Royal Adelaide Hospital.

Journal IF is the most commonly used means to quantitatively appraise and rank the bibliometric impact of published articles (11). IFs were available for all 24 included journals, however, their individual availability for years before 2008 is scarce & variable. The mean IF of journals for all Australian publications in 2009 was 1.23, compared to 1.96 for 2018, that being a change of 0.73. The mean growth of IF for the included journals, irrespective of Australian publications, however, was 0.42 over this same time period. Accounting for the inherent organic growth of journals over this time, this indicates Australian research is being published in higher ranked journals.

Discussion

The philosophies of EBM date back to the early 1900s, however, the term was first popularised by Guyatt et al. in JAMA in 1992, and has since developed into a predominant architype for clinical practise (12). Otorhinolaryngology has joined this chorus of EBM-based practise and has experienced steady growth in its clinical research, with the power of its study designs slowly increasing (7,13). The findings of our study are aligned with these documented worldwide trends, identifying that 79.0% of all Australian ENT research is clinical in nature, similar to that documented by Wasserman et al., at 75% & Bentsianov et al. at 72%. Of note is that these two articles studied these trends retrospectively over a 10-year period ending in 2003 & cross-sectionally in 1999 respectively. As our study included all articles up to 2018 (a further 15+ years data), it can be deduced that clinical based research is on the rise (7,14). Our study has additionally appreciated a steep surge in the quantity of published clinical literature, as the 10-year period between 2009–2018 yielded over six times more publications compared to the 10 years between 1989–1999 (P<0.001).

Although on paper, these trends appear to be favourable and representative of a growing research culture within otorhinolaryngology, a litany of critics have previously lamented on the paucity of the quality within this rapidly expanding body of research from the profession. Maran et al. found that observational descriptive studies made up 39–50% of all ENT articles. These included single case reports, anecdotal and uncontrolled publications, all of which constitute low quality research. His study, which published in 1997, concluded that ENT did not fit within the sphere of evidence-based practise (15). Furthermore, Gates in 1999, added weight to this perception, affirming that high-powered research in the form of clinical trials were seen as a significant downfall of ENT, especially compared to other medical based specialties (16).

Many years ago, The Laryngoscope began including the LOE (levels 1–5) for all of their published research articles, primarily to provide clarity to their readers, but also as a tool to motivate authors and investigators to improve their study designs and produce higher quality research (17). We have translated this initiative from The Laryngoscope into our study design to create an accurate and honest assessment of Australian ENT research. Over the 30-year study period, we identified a statistically meaningful increase in the publication of higher levels of evidence, with 18.2% of all published Australian ENT literature in the last 10 years representing levels 1 & 2 evidence combined. This represents a statistically significant increase when compared to the two previous 10-year periods, which documented 12.5% and 3.4% of combined levels 1 & 2 evidence respectively (P<0.001). Despite this increase, over 75% of current research still equates to level 4 evidence or worse. These Australian figures compare quite similarly to ENT research from other western countries. In the last 30 years, 3.3% of Australian ENT publications have been RCTs, which equates to level 1b literature. In comparison, Maran et al. documents 1.2% of all American and 4% of UK ENT literature were RCTs (level 1b) by design (15). Australia’s favourable academic representation in this global forum can be partly contributed to the establishment of the Garnett Passe and Rodney Williams Memorial (GPRWM) Foundation in 1986. The trust became operative in 1993 and to date has been one of the largest bequeathments made to otorhinolaryngology, head and neck surgery in Australia (18). The foundations aim is to promote, maintain and improve medical knowledge and education, and the highest standards of excellence in otolaryngology, head and neck surgery. In doing so, the foundation has to date invested over $65 million in support to Australasian scientific and clinical researchers. This significant financial backing in our field has been further compounded by the RACS more recent commitment to academic and research inclusive training, as the “Scholarship and Teacher” is now defined by the college as one of its nine professional competencies (5). The last 10 years has seen the development of many academic focused courses and themes within Australian based surgical training (19). Nowadays, there exists many a provision of surgical research grants from both RACS and subspecialty societies, most notably the Garnett Passe in ENT, providing the necessary financial support for trainees and fellows wishing to conduct either part or full-time research (20).

It is difficult to compare the amount of high-quality research of Australian ENT and ENT in general to other medical specialties. Michaud et al. claimed that over 64% of primary therapeutic clinical decisions in three general medical services were supported by evidence from randomised controlled or head to head trials, i.e., high-quality research (21). This emphasises a huge gap between the proportion of similar clinical decisions extrapolated from similar quality ENT based research. Given proper thought, however, it would seem naïve to expect such comparable trends from the ENT fraternity. The subspecialty nature of otorhinolaryngology means patient volumes and disease prevalence in our studies are inherently smaller compared to these larger medical specialties (22). To therefore generate the number of participants needed in both study and control groups to produce a similar high-powered study design would be extremely onerous and often not feasible. Furthermore, the nature of surgery as a whole creates both ethical and financial difficulties in developing many prospective and/or randomised studies that medical specialties may not face (7). Nonetheless, the quality of otorhinolaryngology publications has been shown to be analogous to other surgical subspecialties such as orthopaedics, ophthalmology, and neurosurgery (21).

This is not to say that otorhinolaryngology has a ceiling to its potential growth in developing higher powered and quality research going forward. There is undoubtedly a groundswell of increasing collaborative surgical research, with significant growths in the number of not only inter departmental and multi-institutional collaboration, but multinational collaboration as well (23). The last 10 years of Australian based ENT research has had an average of just over three institutions involved per published article, more than double that for the period between 1989–1999. Furthermore, our study shows that for the last 5 years, Australian authors/institutions have been affiliated with an average of 41 multinational articles per year, 43.9% of which had an Australian as first author. Therefore, in an attempt to negate or counteract the aforementioned limitations to high powered ENT research, such collaboration is crucial for the future.

A strength of this study is the breadth of both the number of journals included and the amount of time in years data was collected over and thus analysed. Many similar studies often chose the 4 or 5 most highly ranked or influential otorhinolaryngology journals to investigate, many of which only collected data over 1 specific year at regular time intervals as a snapshot to extrapolate trends (7,15). We therefore believe the findings of this paper more conclusively represent the true trends of otorhinolaryngology research in Australia. Furthermore, our method of using a sole investigator to screen all collected abstracts (in close collaboration with the supervising author) created the consistency necessary to ensure the utmost accuracy in determining many of the subjective factors of data collection, mainly in deciphering the levels of evidence. In the process of screening many of these abstracts, it was clear that several authors would oversell the quality of their study design by using particular terminology. Many abstracts would define the design of their study as a “cohort study”, however, merely referencing a cohort of patients which should be more appropriately termed a case series. True cohort studies refer to an observational, prospective epidemiological study, whereby multiple variables are assessed (including a control or unexposed group) and relative risks are calculated (17). This “Freudian slip”, is often the key differentiating point between a level 2 and a level 4 study.

Despite including the large majority of international affiliated otorhinolaryngology journals, a potential downfall to this study is the exclusion of general/all-encompassing Australian surgical journals (e.g., ANZ Journal of Surgery). Furthermore, the highest quality research from any field is often published in equivalent high-quality general medical journals (e.g., BMJ, JNE), which were similarly omitted. Naturally, many relevant publications and potentially articles with very high quality of evidence could have been missed, thus possibly distorting the true trends in otorhinolaryngology research in Australia.

Conclusions

This retrospective epidemiological study has proven that the quantity and quality of Australian based otorhinolaryngology research in the last 30 years is on the rise. There is a statistically significant increase in the proportion of research with higher levels of evidence. Although the evidence base for otorhinolaryngology still trails the likes of larger medical specialties, it remains comparable to many other surgical subspecialties. The ongoing impetus of RACS and the Australian Society of Otolaryngology Head and Neck Surgery (ASOHNS) towards a strong focus on academic surgery and research, as well as the outstanding contribution of the GPRWM foundation have leaded to an evolution in otorhinolaryngology research in Australia. Several Australian researchers have embraced multi-institutional and multinational collaboration and in doing so are producing literature of the highest calibre, ensuring that Australian institutions are making highly significant contributions to the future of otorhinolaryngology both nationally and globally.

Acknowledgments

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/ajo.2019.03.02). AC serves as an unpaid editorial board member of Australian Journal of Otolaryngology. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Otorhinolaryngology Weir N. Postgrad Med J 2000;76:65-9.
2. Rosenfeld RM, Bluestone CD, editors. Evidence-based otitis media. Hamilton, Ontario: BC Decker, 1999.
3. Horton R. Surgical research or comic opera: questions, but few answers. Lancet 1996;347:984-5.
4. Brooke BS, Nathan H, Pawlik TM. Trends in the Quality of Highly Cited Surgical Research Over the Past 20 Years. Ann Surg 2009;249:162-7.
5. Gough IR. New paradigms in surgical education. Surg Pract 2012;16:2-5.
6. Review of Health and Medical Research in Australia. Australian Government, Department of Health. (cited 31/10/18). Available online: http://www.health.gov.au/internet/main/publishing.nsf/content/review+of+health+and+medical+research+terms+of+reference
7. Wasserman JM, Wynn R, Bash TS, et al. Levels of evidence in otorhinolaryngology journals. Otolaryngol Head Neck Surg 2006;134:717-23.
8. Joseph KS. Quality of impact factors of general medical journals. BMJ 2003;326:283.
9. Phillips B, Ball C, Sackett D, et al. Levels of evidence and grades of recommendation. Oxford Centre for Evidence-based Medicine. (cited 25/9/18). Available online: https://www.cebm.net/levels_of_evidence.asp
10. SPSS Inc. SPSS 11.5 Brief Guide. Upper Saddle River, NJ: Prentice Hall Inc., 2002.
11. Garfield E. The history and meaning of the journal impact factor. JAMA 2006;295:90-3.
12. Evidence-Based Medicine Working Group. Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA 1992;268:2420-5.
13. Rosenfeld RM. Clinical research in otolaryngology journals. Arch Otolaryngol Head Neck Surg 1991;117:164-70.
14. Bentsianov BL, Boruk M, Rosenfeld RM. Evidence-based medicine in otolaryngology journals. Otolaryngol Head Neck Surg 2002;126:371-6.
15. Maran AG, Molony NC, Armstrong MWJ, et al. Is there an evidence base for the practice of ENT surgery? Clin Otolaryngol 1997;22:152-7.
16. Gates GA. So where’s the evidence? Otolaryngol Head Neck Surg 1999;120:619-20.
17. Stewart MG. Reporting levels of evidence. Laryngoscope 2014;124:358.
18. The Garnett Passe and Rodney Williams Memorial Foundation. (cited 30/9/18). Available online: https://gprwmf.org.au
19. Hanney RM, Hill AG, LeMaire SA, et al. Developing academic surgeons: a focus of a new course. ANZ J Surg 2009;79:872-5.
20. Wells CI, Robertson JP, O'Grady G, et al. Trends in publication of general surgical research in New Zealand, 1996-2015. ANZ J Surg 2017;87:76-9.
21. Michaud G, McGowan JL, Van der Jagt R, et al. Are therapeutic decisions supported by evidence from health care research? Arch Intern Med 1998;158:1665-8.
22. Bhattacharyya N. Peer review: studying the major otolaryngology journals. Laryngoscope 1999;109:640-4.
23. Goldsack JC, Sonnad SS. Changing trends in surgical research: an analysis of 30 years of collaborative practices. JAMA Surg 2014;149:873-4.