Advertisement

Readability of online monkeypox patient education materials: Improved recognition of health literacy is needed for dissemination of infectious disease information

  • Author Footnotes
    1 Joint-first authors. Both authors contributed equally to the preparation of the manuscript.
    Jessica C. Frost
    Footnotes
    1 Joint-first authors. Both authors contributed equally to the preparation of the manuscript.
    Affiliations
    Buckinghamshire Healthcare NHS Trust, Buckinghamshire, UK
    Search for articles by this author
  • Author Footnotes
    1 Joint-first authors. Both authors contributed equally to the preparation of the manuscript.
    Alexander J. Baldwin
    Correspondence
    Corresponding author.
    Footnotes
    1 Joint-first authors. Both authors contributed equally to the preparation of the manuscript.
    Affiliations
    Buckinghamshire Healthcare NHS Trust, Buckinghamshire, UK
    Search for articles by this author
  • Author Footnotes
    1 Joint-first authors. Both authors contributed equally to the preparation of the manuscript.
Published:December 21, 2022DOI:https://doi.org/10.1016/j.idh.2022.11.002

      Highlights

      • The majority of individuals now choose to access health information online.
      • Low health literacy has a strong correlation to ill health.
      • Health information should be written at the level of American sixth grade.
      • In this study, no webpages met this target readability level.
      • Readability must be considered when publishing online patient education materials.

      Abstract

      Background

      Health literacy is key to navigating the current global epidemic of misinformation and inaccuracy relating to healthcare. The American Medical Association (AMA) suggests health information should be written at the level of American sixth grade. With the monkeypox outbreak being declared a Public Health Emergency of International Concern (PHEIC) in July 2022, we sought to assess the readability of online patient education materials (PEMs) relating to monkeypox to see if they are at the target level of readability.

      Methods

      A search was conducted on Google.com using the search term ‘Monkeypox’. The top 50 English language webpages with patient education materials (PEMs) relating to monkeypox were compiled and categorised by country of publication and URL domain. Readability was assessed using five readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and, Simple Measure of Gobbledygook Index (SMOG). Unpaired t-test for URL domain, and one-way ANOVA for country were performed to determine influence on readability.

      Results

      Three of the five tools (FRES, GFI, CLI) identified no webpages that met the target readability score. The FKGL and SMOG tools identified one (2%) and two (4%) webpages respectively that met the target level. County and URL domain demonstrated no influence on readability.

      Conclusion

      Online PEMs relating to monkeypox are written above the recommended reading level. Based on the previously established effect of health literacy, this is likely exacerbating health inequalities. This study highlights the need for readability to be considered when publishing online PEMs.

      Keywords

      Introduction

      Monkeypox is a viral zoonosis that presents with symptoms similar to those previously seen in smallpox patients. Routes of transmission include human-to-human via direct contact with infectious skin or mucocutaneous lesions, respiratory droplets or indirect contact from contaminated objects or materials. Whilst not considered a sexually transmitted disease, the majority of cases have been reported in men who have sex with men (MSM), especially those engaging in high-risk sex with multiple partners. The virus has been endemic in West and Central Africa for decades. However, between January 1, 2022, and July 20, 2022, around 14,500 probable and laboratory confirmed cases were reported to the World Health Organisation (WHO) from seventy-two countries across all six WHO regions. On July 23, 2022, the monkeypox outbreak was declared a Public Health Emergency of International Concern (PHEIC), a status previously achieved by COVID-19, polio and both the Ebola and Zika virus [
      • World Health Organization (WHO)
      Second meeting of the international health regulations (2005) (IHR) emergency committee regarding the multi-country outbreak of monkeypox.
      ]. The WHO has not currently advised mass vaccinations, but they have recommended post-exposure preventive vaccination (PEPV) for contacts of cases, ideally within four days of first exposure, and primary preventive vaccination (PPV) for individuals at high risk of exposure. This group includes MSM or other persons with multiple sex partners, health workers at risk, laboratory personnel working with orthopoxviruses, clinical laboratory staff performing diagnostic testing for monkeypox [
      • World Health Organization (WHO)
      Vaccines and immunization for monkeypox: interim guidance, 24 August 2022.
      ].
      Research suggests that the majority of individuals now choose to access health information online, especially with regard to sexual health given the accessibility and anonymity the internet provides [
      • Bujnowska-Fedak M.M.
      • Waligóra J.
      • Mastalerz-Migas A.
      The internet as a source of health information and services.
      ]. There is currently no quality standard for online information and it is evident from previous research that many online patient education material (PEMs) are written above the recommended reading age [
      • Meleo-Erwin Z.
      • Basch C.H.
      • Fera J.
      • Garcia P.
      Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
      ]. The multitude of complexities and uncertainties associated with the monkeypox outbreak, including atypical presentations and biphasic appearance of lesions, has also undoubtedly contributed to the fuelling of misinformation being shared on a global scale about this public health event.
      Health literacy is key to navigating the current global epidemic of misinformation and inaccuracy, and is defined by The World Health Organisation (WHO) as ‘the personal characteristics and social resources needed for individuals and services to make decisions about health.’ [
      World Health Organization (WHO) Regional Office for South-East Asia
      Health literacy toolkit for low- and middle-income countries : a series of information sheets to empower communities and strengthen health systems.
      ] The ability to use general literacy and numeracy skills in relation to health-related resources can also be broken down into five core components: obtention, comprehension, appraisal, communication and application of information [
      • Sørensen K.
      • Van den Broucke S.
      • Fullam J.
      • Doyle G.
      • Pelikan J.
      • Slonska Z.
      • et al.
      Health literacy and public health: a systematic review and integration of definitions and models.
      ]. Difficulties with any of these stages can signify low health literacy, which has been linked to significant adverse health outcomes stemming from limited understanding of health and disease, including reduced engagement with healthcare providers, lower medication compliance and adherence, increased hospitalisations and increased mortality [
      • Jayasinghe U.W.
      • Harris M.F.
      • Parker S.M.
      • Litt J.
      • van Driel M.
      • Mazza D.
      • et al.
      The impact of health literacy and life style risk factors on health-related quality of life of Australian patients.
      ]. A report by National Voices in 2017 declared health literacy ‘the strongest correlation to ill health – stronger than education level, deprivation, age or ethnicity’ [
      National Voices: People and Communities Board
      A new relationship with people and communities: actions for delivering chapter 2 of the NHS five year forward view.
      ]. Readability is defined as the ‘ease of understanding or comprehension due to the style of writing’ and is frequently measured in the context of American ‘grade levels’ [
      • Ley P.
      • Florio T.
      The use of readability formulas in health care.
      ,
      • Shedlosky-Shoemaker R.
      • Sturm A.C.
      • Saleem M.
      • Kelly K.M.
      Tools for assessing readability and quality of health-related Web sites.
      ]. Readability formulas thus offer a means of ensuring a resource is suitable for its intended audience, with the aim of improving patient education and self-efficacy with regard to health.
      Globally, one in ten adults lack ‘the most basic information processing skills considered necessary to succeed in today's world’, whilst almost 50% of adults in Europe are deemed to have ‘inadequate’ or ‘problematic’ health literacy [
      Organisation for Economic Co-operation and Development (OECD)
      OECD skills outlook 2013: first results from the survey of adult skills.
      ,
      HLS-EU Consortium
      Comparative report of health literacy in eight EU member states: the European Health Literacy Survey HLS-EU.
      ]. A publication from Public Health England and the Institute of Health Equity in 2015 demonstrated that in the United Kingdom (UK) 43% of adults lack adequate literacy skills, and 61% lack adequate numeracy skills, to routinely process health information [
      Public Health England & UCL Institute of Health Equity
      Local action on health inequalities.
      ]. In the United Stated of America (USA), this rises drastically to 88% of adults not meeting the ‘proficient’ target [
      • National Center for Education Statistics
      The health literacy of America's adults: results from the 2003 national assessment of adult literacy.
      ]. Whilst there is no consensus on the recommended level of reading difficulty for health information, Health Education England (HEE) advises that patient information should be written at a level comprehensible to an 11 year old in order to facilitate uptake [

      NHS Health Education England. Health literacy ‘how to’ guide. https://library.nhs.uk/wp-content/uploads/sites/4/2020/08/Health-literacy-how-to-guide.pdf. [Accessed 11 August 2022].

      ]. This is equivalent to a US sixth grade level or less, which is also the level at which the American Medical Association (AMA) suggests health information should be written [
      • Weiss B.D.
      Health literacy: a manual for clinicians.
      ].
      Given the trend in current literature, we hypothesised that online PEMs about the monkeypox outbreak are written above the recommended level appropriate for the general public, which may be contributing to public misinformation and adverse health outcomes as a result.

      Methods

      Online PEMs were identified using the Google.com search engine. This search engine was selected because, according to the popular online web traffic analysis website StatCounter, Google searches comprised 92% of the online search market share at the time that this study was conducted [
      • GlobalStats Statscounter
      Search engine market share worldwide.
      ]. The search was carried out using the search term ‘Monkeypox’ on July 6, 2022. To eliminate any skewed or biased results based on previous search history or internet activity, the Internet browser was cleared of all search history, cache, cookies, and other user data.
      The sample set for analysis was generated by compiling the first 50 English language webpages that contained patient orientated information aimed at the general public. Webpages were excluded for the following reasons: information pages written for healthcare professionals, studies from peer-reviewed journals, news articles, personal experiences, or, webpages that contained exclusively audio-visual material. PEMs that were solely audio-visual in nature were excluded because these could not undergo readability analysis as proposed in this study's methodology. By enforcing these exclusion criteria, we included only those webpages aimed at providing health education to the general population. To allow for categorisation, basic data on the webpages' characteristics were captured prior to analysis of readability. These data included: country of origin and URL domain name (i.e.gov,. com,.org, etc.).
      Text from each webpage was entered into a Microsoft Word document and edited based on previously established protocols for readability assessment studies [
      • Kruse J.
      • Toledo P.
      • Belton T.B.
      • Testani E.J.
      • Evans C.T.
      • Grobman W.A.
      • et al.
      Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
      ,
      • Kasabwala K.
      • Agarwal N.
      • Hansberry D.R.
      • Baredes S.
      • Eloy J.A.
      Readability assessment of patient education materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation.
      ]. Components of the text unrelated to patient education were removed, as these may influence readability score. This included the removal of disclaimers, advertisements, webpage navigation, website URLs, copyright information, acknowledgements, author information, citation and references. Supplemental editing of non-textual elements and punctuation was performed including paragraph breaks, colons, and bullet points that may cause the readability tool to over- or underestimate the difficulty of a text [
      • Friedman D.B.
      • Hoffman-Goetz L.
      A systematic review of readability and comprehension instruments used for print and web-based cancer information.
      ].
      Readability scores were generated for the edited text using an online readability software [

      Readability Formulas. Text readability consensus calculator. https://readabilityformulas.com/freetests/six-readability-formulas.php. [Accessed 6 August 2022].

      ]. Text was assessed using five validated readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and, Simple Measure of Gobbledygook Index (SMOG). Each tool calculates readability by applying a mathematical formula to a passage of text (Supplementary Table 1). The formulas each consider different factors when producing an ease of readability score. These factors include a combination of mean number of syllables per word, mean number of words per sentence, numbers of sentences and numbers of complex words (polysyllabic words) [
      • Ley P.
      • Florio T.
      The use of readability formulas in health care.
      ,
      • Shedlosky-Shoemaker R.
      • Sturm A.C.
      • Saleem M.
      • Kelly K.M.
      Tools for assessing readability and quality of health-related Web sites.
      ]. The FRES gives a score of 0–100, with higher score being deemed easier to read, whilst the other tools (FKGL, GFI, CLI, and, SMOG) give a reading grade level. The interpretation and target score for each tool is described in Table 1.
      Table 1Interpretation of each readability score.
      Readability ToolInterpretationTarget score
      FRES90-100: Very easy (Grade 5)

      80-90: Easy (Grade 6)

      70-80: Fairly easy (Grade 7)

      60-70: Standard (Grade 8–9)

      50-60: Fairly difficult (Grade 10–12)

      30-50: Difficult (College)

      0-30: Very difficult (College graduate)
      ≥80
      FKGLYears of education required to understand most of the text≤6.9
      GFIYears of formal education that a person requires in order to easily understand the text on the first reading≤6.9
      CLIYears of education required to understand most of the text6
      SMOGReading grade level necessary for a person to have reached in order to fully understand text≤6.9
      FRES = Flesch Reading Ease, FKGL = Flesch-Kincaid Grade Level, GFI = Gunning Fog Index, CLI = Coleman-Liau Index, and SMOG = Simple Measure of Gobbledygook.
      The methodology used in this study is consistent with other readability studies within the field of infectious diseases, public health, sexual health and other medical and surgical specialties [
      • Meleo-Erwin Z.
      • Basch C.H.
      • Fera J.
      • Garcia P.
      Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
      ,
      • Kruse J.
      • Toledo P.
      • Belton T.B.
      • Testani E.J.
      • Evans C.T.
      • Grobman W.A.
      • et al.
      Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
      ,
      • Kasabwala K.
      • Agarwal N.
      • Hansberry D.R.
      • Baredes S.
      • Eloy J.A.
      Readability assessment of patient education materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation.
      ,
      • Corcoran N.
      • Ahmad F.
      The readability and suitability of sexual health promotion leaflets.
      ,
      • Gao B.
      • Shamrock A.G.
      • Gulbrandsen T.R.
      • O'Reilly O.C.
      • Duchman K.R.
      • Westermann R.W.
      • et al.
      Can patients read, understand, and act on online resources for anterior cruciate Ligament surgery?.
      ].

      Data analysis

      Webpage characteristics were described using rounded frequencies (per cent). Overall, and sub-group (based on webpage characteristics) FRES, FKGL, CLI, SMOG, GFI and median grade scores were described as mean values ± standard deviation, having been found to be normally distributed using the Shapiro–Wilk test of normality. The median grade score was calculated from the tools that compute the result as a grade score (FKGL, CLI, SMOG, GFI). The median grade score was calculated as the grade scores for each individual webpage were not normally distributed. Unpaired t-test was performed to determine if a government domain extension (.gov) compared to a non-government domain extension (.org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.) influenced readability. One-way analysis of variance (ANOVA) was performed to determine if country of publication (United States of America, United Kingdom, Canada or Other, which included Australia, Ireland, Netherlands, Singapore, Switzerland) had an effect on readability. All analyses were performed in R (v4.2.1).

      Results

      Fifty webpages were analysed (Supplementary Table 2). When separated by webpage characteristics (Table 2), 29 (58%) of these were.gov webpages, and 21 (42%) had other URL domain names, which included:.org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us. When divided by country of origin, 34 (68%) webpages originated from the United States of America (USA), 5 (10%) originated from the United Kingdom (UK), 4 (8%) originated from Canada, with the remaining 7 (14%) originating from a variety of other countries, including Australia, Ireland, Netherlands, Singapore, Switzerland.
      Table 2Webpage characteristics.
      Webpage characteristicFrequency (%)
      URL domain name
       .gov29(58)
       Other
      org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.
      21(42)
      Country of publication
       USA34(68)
       UK5(10)
       Canada4(8)
       Other
      International, Australia, Ireland, Netherlands, Singapore, Switzerland.
      7(14)
      USA = United States of America, UK = United Kingdom.
      a org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.
      b International, Australia, Ireland, Netherlands, Singapore, Switzerland.
      Distribution of webpages by difficulty level can be found in Table 3. Using FRES, none of the webpages met the target ease of reading score of ≥80. For the tools that gave a score representing a school grade, or years of education, the GFI and CLI tools found no webpages with a readability score at the target level of sixth grade. The FKGL and SMOG tools identified one (2%) and two (4%) webpages respectively that met the target level of sixth grade, with both the FKGL and SMOG identifying the same webpage in one instance.
      Table 3Distribution of webpages by difficulty level and mean readability scores.
      Readability score by toolFrequency (%)Mean Score (SD)
      FRES54.4(8.5)
       Very Easy (90–100)0(0)
       Easy (80–90)0(0)
       Fairly easy (70–80)0(0)
       Standard (60–70)13(26)
       Fairly difficult (50–60)20(40)
       Difficult (30–50)17(34)
       Very difficult (0–30)0(0)
      FKGL9.6(1.6)
       Grade 6 or below1(2)
       Grade 78(16)
       Grade 8-918(36)
       Grade 10-1222(44)
       College/College graduate1(2)
      GFI12.4(1.8)
       Grade 6 or below0(0)
       Grade 70(0)
       Grade 8-96(12)
       Grade 10-1226(52)
       College/College graduate18(36)
      CLI11.1(1.3)
       Grade 6 or below0(0)
       Grade 70(0)
       Grade 8-95(10)
       Grade 10-1238(76)
       College/College graduate7(14)
      SMOG9.4(1.3)
       Grade 6 or below2(4)
       Grade 76(12)
       Grade 8-925(50)
       Grade 10-1217(34)
       College/College graduate0(0)
      Median Grade Score
      Calculated from median of FKGL, GFI, CLI and SMOG.
      10.4(1.3)
       Grade 6 or below0(0)
       Grade 71(2)
       Grade 8-917(34)
       Grade 10-1230(60)
       College/College graduate2(4)
      FRES = Flesch Reading Ease, FKGL = Flesch-Kincaid Grade Level, GFI = Gunning Fog Index, CLI Coleman-Liau Index, and SMOG = Simple Measure of Gobbledygook.
      a Calculated from median of FKGL, GFI, CLI and SMOG.
      For all webpages, the FRES ranged from 34.6 (difficult, or College level) to 69.8 (standard difficulty, or grade 8 to 9), with a mean score of 54.4 (SD = 8.5). FKGL ranged from 7 to 12.8, with a mean score of 9.6 (SD = 1.6). GFI ranged from 8.9 to 16.6, with a mean score of 12.4 (SD = 1.8). CLI ranged from 9 to 14, with a mean score of 11.1 (SD = 1.3). SMOG ranged from 6.6 to 12.4, with a mean score of 9.4 (SD = 1.3). Separating the webpages by ‘.gov’ sources and ‘other URL domain names’ or by county of publication showed no influence on readability (Table 4, Table 5).
      Table 4Mean readability by County of origin.
      Readability testUSA (SD)UK (SD)Canada (SD)Other
      International, Australia, Ireland, Netherlands, Singapore, Switzerland.
      (SD)
      P value
      p-value derived using ANOVA testing.
      FRES54.8(7.7)58.4(12.5)55.1(2.1)50.9(8.9)0.5112
      FKGL9.6(1.6)8.8(2.3)9.9(0.7)10.3(1.5)0.6407
      GFI12.3(1.7)11.7(2.4)12.5(1.1)13.0(1.8)0.4658
      CLI11.0(1.1)10.4(1.7)10.5(0.6)12.1(1.6)0.0671
      SMOG9.3(1.2)8.8(1.9)9.5(0.7)9.9(1.2)0.5251
      Median grade
      Calculated from median of FKGL, GFI, CLI and SMOG.
      10.3(1.2)9.7(1.9)10.3(0.3)11.2(1.4)0.2613
      USA = United States of America, UK = United Kingdom.
      a International, Australia, Ireland, Netherlands, Singapore, Switzerland.
      b p-value derived using ANOVA testing.
      c Calculated from median of FKGL, GFI, CLI and SMOG.
      Table 5Mean readability by URL domain name.
      Readability test.gov (SD)Other
      org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.
      (SD)
      P value
      p-value derived using unpaired t-test.
      FRES52.9(7.5)56.5(9.4)0.1377
      FKGL9.9(1.5)9.2(1.7)0.1172
      GFI12.7(1.7)11.9(1.9)0.0971
      CLI11.3(1.3)10.8(1.2)0.1398
      SMOG9.6(1.2)9.0(1.4)0.1423
      Median grade
      Calculated from median of FKGL, GFI, CLI and SMOG.
      10.6(1.2)10.0(1.4)0.1248
      a org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.
      b p-value derived using unpaired t-test.
      c Calculated from median of FKGL, GFI, CLI and SMOG.

      Discussion

      Health literacy is an important subject across all specialities, including both infectious disease and sexual health. Whilst not a sexually transmitted infection, PEMs created about monkeypox rely on patients having an ‘ability to understand sexual health information and application of that information’ to be able to make informed choices with regard to sex and safe sex practices, also defined by the WHO as sexual health literacy [
      • Rakhshaee Z.
      • Maasoumi R.
      • Nedjat S.
      • Khakbazan Z.
      Sexual health literacy, a strategy for the challenges of sexual life of infertile women: a qualitative study.
      ]. Electronic health literacy is also crucial in our increasingly technologized world, and refers to the processing and understanding of health information from electronic sources. In contrast to those with high electronic health literates, research suggests people with low electronic health literacy levels are significantly more likely to believe in misinformation and also to experience information overload from online sources [
      • Kim H.K.
      • Tandoc Jr., E.C.
      Consequences of online misinformation on COVID-19: two potential pathways and disparity by eHealth literacy.
      ]. Although the internet remains the top source of health information for the vast majority of people as it is both accessible and offers anonymity, it is apparent from other readability studies in both infectious disease and sexual health that the majority of PEMs available online are pitched at audiences above the recommended reading level [
      • Meleo-Erwin Z.
      • Basch C.H.
      • Fera J.
      • Garcia P.
      Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
      ,
      • Kruse J.
      • Toledo P.
      • Belton T.B.
      • Testani E.J.
      • Evans C.T.
      • Grobman W.A.
      • et al.
      Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
      ,
      • Corcoran N.
      • Ahmad F.
      The readability and suitability of sexual health promotion leaflets.
      ]. Our results are consistent with these findings, demonstrating that none of the fifty total sampled PEMs regarding the monkeypox outbreak identified via Google search were of an appropriate readability for the general public, based on the results of all five validated readability tools, and thus are likely to be suboptimal as educative resources.
      The COVID-19 pandemic previously underscored the scale of the ‘infodemic’ we are currently facing as a result of both increased information consumption, especially via a digital means, and global health illiteracy [
      • Banerjee D.
      • Meena K.S.
      COVID-19 as an "infodemic" in public health: critical role of the social media.
      ]. An infodemic is defined by the WHO as an ‘overabundance of information - some accurate and some not - that occurs during an epidemic’, which can hugely hinder the evidence-based approach to managing a public health crisis, and make it difficult for the general public to make informed decisions [

      World Health Organization (WHO). Infodemic management. https://www.who.int/teams/risk-communication/infodemic-management. [Accessed 11 August 2022].

      ]. The COVID-19 pandemic is even more topical when we consider the heightened need for individuals to be able to respond to health information at speed. Health advice relating to monkeypox has been evolving almost daily, which requires the general public to be able to rapidly acquire and apply health information. Low health literacy rates can impair quicker processing times and, combined with online information perhaps unintentionally better suited to a higher readability than recommended, may be further complicating people's ability to comprehend and utilise this information to make informed health decisions.

      Limitations

      There are several limitations of this study. Firstly, as noted in other readability studies, information changes rapidly online and search results can change day on day [
      • Meleo-Erwin Z.
      • Basch C.H.
      • Fera J.
      • Garcia P.
      Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
      ]. Due to the cross-sectional design of a study such as this, online information is only captured at a snapshot in time. It is also important to consider that readability does not alone give a complete assessment of how understandable a webpage may be. Other factors such as visuals, both pictographic and audio-visual, and design elements, including font size and the amount of white space, have also been shown to influence comprehension of patient education information [
      • Doak L.G.
      • Doak C.C.
      • Meade C.D.
      Strategies to improve cancer education materials.
      ]. None of the readability formulas used were designed to consider the effect of these design elements. Future studies of PEMs created for monkeypox should examine ‘understandability’, likely using the Patient Education Materials Assessment Tool (PEMAT); ‘quality’, likely using with the DISCERN instrument; and, ‘accountability’, likely using the Journal of the American Medical Association (JAMA) benchmarks of accountability, which are ‘authorship’, ‘attribution’, ‘disclosure’, and ‘currency’. This study was also limited to material written in English and did not include pamphlets or other downloadable content. Future research should assess readability of downloadable content, including pamphlets, and readability of non-English material, especially given the international nature of this public health issue. This study also did not include an analysis of which polysyllabic words were used most frequently. An analysis such as this could reveal the vocabulary contributing to the more difficult readability scores.
      Another limitation of this study was that the sample was limited to the first 50 webpages returned by the search term. This sample size is, however, in keeping with the design of other readability studies and in the case of this study appears to have given representative findings, given the homogeneity of our results. Furthermore, individuals most commonly access the top webpages returned from any given search, therefore our sample is representative of the webpages that the typical lay individual would access.
      Despite the limitations of readability tools, they remain an efficient way of testing ease of readability and their use should be encouraged so as to aid those designing PEMs to carefully select their words and content in a way that is mindful of the literacy levels in the general population. Future studies of online patient information regarding monkeypox might also assess quality of the content. Nevertheless, the findings of this study have started to fill a gap in the available literature.

      Conclusion

      Ultimately, this study has shown that PEMs relating to monkeypox are written above the recommended reading level. Based on the previously established effect of health literacy, this is likely exacerbating health inequalities. This study also highlights the need for readability to be taken into account when publishing online resources to ensure simpler information is made available on this topic. Improving uptake and understanding of health information is not only the responsibility of individuals, but also that of governments, health systems and organisations, to ensure that accurate, appropriate and digestible information is made accessible for diverse audiences. Readability can be improved easily by using shorter words, shorter sentences and avoiding complex or medical vernacular. There are many tool-kits available online to simplify complicated scientific information into better designed communication materials [
      • United States (US)
      Department of health and human services & centers for disease control and prevention (CDC).
      ].
      A more symbiotic and holistic approach to address health literacy may reduce disparities in health-associated quality of life and improve health outcomes and treatment-seeking. Addressing the gap in health literacy may also tackle misinformed stigma of certain groups and the perpetuation of incorrect information online [
      • United States (US)
      Department of health and human services & centers for disease control and prevention (CDC).
      ]. Further research, however, is needed to fully understand how best to articulate consistent and clear messages across multiple populations within a ‘rapidly changing scientific and politically charged environment’, especially within a sexual health context [
      • McDaid L.
      • Flowers P.
      • Ferlatte O.
      • Young I.
      • Patterson S.
      • Gilbert M.
      Sexual health literacy among gay, bisexual and other men who have sex with men: a conceptual framework for future research.
      ].

      Ethics

      This study used information freely available in the public domain and did not include human subjects and therefore did not require ethical approval.

      Authorship statement

      Jessica C. Frost and Alexander J. Baldwin are joint-first authors of this study. They both performed conceptualization, methodology, investigation, data curation, formal analysis and writing - original draft, review and editing. Both authors read and approved the final manuscript.

      Financial disclosure statement

      None to declare.

      Funding

      None to declare.

      Provenance and peer review

      Not commissioned; externally peer reviewed.

      Conflict of Interest

      None.

      Appendix A. Supplementary data

      The following are the Supplementary data to this article:

      References

        • World Health Organization (WHO)
        Second meeting of the international health regulations (2005) (IHR) emergency committee regarding the multi-country outbreak of monkeypox.
        2022 (Accessed 11 August 2022)
        • World Health Organization (WHO)
        Vaccines and immunization for monkeypox: interim guidance, 24 August 2022.
        (Accessed 01 November 2022)
        • Bujnowska-Fedak M.M.
        • Waligóra J.
        • Mastalerz-Migas A.
        The internet as a source of health information and services.
        Adv Exp Med Biol. 2019; 1211: 1-16
        • Meleo-Erwin Z.
        • Basch C.H.
        • Fera J.
        • Garcia P.
        Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
        Infect Dis Health. 2020; 25: 277-282
        • World Health Organization (WHO) Regional Office for South-East Asia
        Health literacy toolkit for low- and middle-income countries : a series of information sheets to empower communities and strengthen health systems.
        WHO Regional Office for South-East Asia, New Delhi2015
        • Sørensen K.
        • Van den Broucke S.
        • Fullam J.
        • Doyle G.
        • Pelikan J.
        • Slonska Z.
        • et al.
        Health literacy and public health: a systematic review and integration of definitions and models.
        BMC Public Health. 2012; 12: 80
        • Jayasinghe U.W.
        • Harris M.F.
        • Parker S.M.
        • Litt J.
        • van Driel M.
        • Mazza D.
        • et al.
        The impact of health literacy and life style risk factors on health-related quality of life of Australian patients.
        Health Qual Life Outcomes. 2016; 14: 68
        • National Voices: People and Communities Board
        A new relationship with people and communities: actions for delivering chapter 2 of the NHS five year forward view.
        (Accessed 11 August 2022)
        • Ley P.
        • Florio T.
        The use of readability formulas in health care.
        Psychol Health Med. 1996; 1: 7-28
        • Shedlosky-Shoemaker R.
        • Sturm A.C.
        • Saleem M.
        • Kelly K.M.
        Tools for assessing readability and quality of health-related Web sites.
        J Genet Counsel. 2009; 18: 49-59
        • Organisation for Economic Co-operation and Development (OECD)
        OECD skills outlook 2013: first results from the survey of adult skills.
        OECD Publishing, Paris2013
        • HLS-EU Consortium
        Comparative report of health literacy in eight EU member states: the European Health Literacy Survey HLS-EU.
        2012 (Accessed 11 August 2022)
        • Public Health England & UCL Institute of Health Equity
        Local action on health inequalities.
        in: Improving health literacy to reduce health inequalities. Public Health England, London2015
        • National Center for Education Statistics
        The health literacy of America's adults: results from the 2003 national assessment of adult literacy.
        National Center for Education Statistics, Washington2006
      1. NHS Health Education England. Health literacy ‘how to’ guide. https://library.nhs.uk/wp-content/uploads/sites/4/2020/08/Health-literacy-how-to-guide.pdf. [Accessed 11 August 2022].

        • Weiss B.D.
        Health literacy: a manual for clinicians.
        American Medical Association Foundation and American Medical Association, Chicago2003
        • GlobalStats Statscounter
        Search engine market share worldwide.
        (Accessed 6 August 2022)
        • Kruse J.
        • Toledo P.
        • Belton T.B.
        • Testani E.J.
        • Evans C.T.
        • Grobman W.A.
        • et al.
        Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
        Am J Infect Control. 2021; 49: 690-693
        • Kasabwala K.
        • Agarwal N.
        • Hansberry D.R.
        • Baredes S.
        • Eloy J.A.
        Readability assessment of patient education materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation.
        Otolaryngol Head Neck Surg. 2012; 147: 466-471
        • Friedman D.B.
        • Hoffman-Goetz L.
        A systematic review of readability and comprehension instruments used for print and web-based cancer information.
        Health Educ Behav. 2006; 33: 352-373
      2. Readability Formulas. Text readability consensus calculator. https://readabilityformulas.com/freetests/six-readability-formulas.php. [Accessed 6 August 2022].

        • Corcoran N.
        • Ahmad F.
        The readability and suitability of sexual health promotion leaflets.
        Patient Educ Counsel. 2016; 99: 284-286
        • Gao B.
        • Shamrock A.G.
        • Gulbrandsen T.R.
        • O'Reilly O.C.
        • Duchman K.R.
        • Westermann R.W.
        • et al.
        Can patients read, understand, and act on online resources for anterior cruciate Ligament surgery?.
        Orthop J Sports Med. 2022; 1023259671221089977
        • Rakhshaee Z.
        • Maasoumi R.
        • Nedjat S.
        • Khakbazan Z.
        Sexual health literacy, a strategy for the challenges of sexual life of infertile women: a qualitative study.
        Galen Med J. 2020; 9e1862
        • Kim H.K.
        • Tandoc Jr., E.C.
        Consequences of online misinformation on COVID-19: two potential pathways and disparity by eHealth literacy.
        Front Psychol. 2022; 13783909
        • Banerjee D.
        • Meena K.S.
        COVID-19 as an "infodemic" in public health: critical role of the social media.
        Front Public Health. 2021; 9610623
      3. World Health Organization (WHO). Infodemic management. https://www.who.int/teams/risk-communication/infodemic-management. [Accessed 11 August 2022].

        • Doak L.G.
        • Doak C.C.
        • Meade C.D.
        Strategies to improve cancer education materials.
        Oncol Nurs Forum. 1996; 23: 1305-1312
        • United States (US)
        Department of health and human services & centers for disease control and prevention (CDC).
        in: Simply Put: a guide for creating easy-to-understand material DC. 3rd ed. Centers for Disease Control and Prevention, Atlanta2009
        • McDaid L.
        • Flowers P.
        • Ferlatte O.
        • Young I.
        • Patterson S.
        • Gilbert M.
        Sexual health literacy among gay, bisexual and other men who have sex with men: a conceptual framework for future research.
        Cult Health Sex. 2021; 23: 207-223