Abstract
Background
Health literacy is key to navigating the current global epidemic of misinformation and inaccuracy relating to healthcare. The American Medical Association (AMA) suggests health information should be written at the level of American sixth grade. With the monkeypox outbreak being declared a Public Health Emergency of International Concern (PHEIC) in July 2022, we sought to assess the readability of online patient education materials (PEMs) relating to monkeypox to see if they are at the target level of readability.
Methods
A search was conducted on Google.com using the search term ‘Monkeypox’. The top 50 English language webpages with patient education materials (PEMs) relating to monkeypox were compiled and categorised by country of publication and URL domain. Readability was assessed using five readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and, Simple Measure of Gobbledygook Index (SMOG). Unpaired t-test for URL domain, and one-way ANOVA for country were performed to determine influence on readability.
Results
Three of the five tools (FRES, GFI, CLI) identified no webpages that met the target readability score. The FKGL and SMOG tools identified one (2%) and two (4%) webpages respectively that met the target level. County and URL domain demonstrated no influence on readability.
Conclusion
Online PEMs relating to monkeypox are written above the recommended reading level. Based on the previously established effect of health literacy, this is likely exacerbating health inequalities. This study highlights the need for readability to be considered when publishing online PEMs.
Introduction
Monkeypox is a viral zoonosis that presents with symptoms similar to those previously seen in smallpox patients. Routes of transmission include human-to-human via direct contact with infectious skin or mucocutaneous lesions, respiratory droplets or indirect contact from contaminated objects or materials. Whilst not considered a sexually transmitted disease, the majority of cases have been reported in men who have sex with men (MSM), especially those engaging in high-risk sex with multiple partners. The virus has been endemic in West and Central Africa for decades. However, between January 1, 2022, and July 20, 2022, around 14,500 probable and laboratory confirmed cases were reported to the World Health Organisation (WHO) from seventy-two countries across all six WHO regions. On July 23, 2022, the monkeypox outbreak was declared a Public Health Emergency of International Concern (PHEIC), a status previously achieved by COVID-19, polio and both the Ebola and Zika virus [
[1]- World Health Organization (WHO)
Second meeting of the international health regulations (2005) (IHR) emergency committee regarding the multi-country outbreak of monkeypox.
]. The WHO has not currently advised mass vaccinations, but they have recommended post-exposure preventive vaccination (PEPV) for contacts of cases, ideally within four days of first exposure, and primary preventive vaccination (PPV) for individuals at high risk of exposure. This group includes MSM or other persons with multiple sex partners, health workers at risk, laboratory personnel working with orthopoxviruses, clinical laboratory staff performing diagnostic testing for monkeypox [
[2]- World Health Organization (WHO)
Vaccines and immunization for monkeypox: interim guidance, 24 August 2022.
].
Research suggests that the majority of individuals now choose to access health information online, especially with regard to sexual health given the accessibility and anonymity the internet provides [
[3]- Bujnowska-Fedak M.M.
- Waligóra J.
- Mastalerz-Migas A.
The internet as a source of health information and services.
]. There is currently no quality standard for online information and it is evident from previous research that many online patient education material (PEMs) are written above the recommended reading age [
[4]- Meleo-Erwin Z.
- Basch C.H.
- Fera J.
- Garcia P.
Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
]. The multitude of complexities and uncertainties associated with the monkeypox outbreak, including atypical presentations and biphasic appearance of lesions, has also undoubtedly contributed to the fuelling of misinformation being shared on a global scale about this public health event.
Health literacy is key to navigating the current global epidemic of misinformation and inaccuracy, and is defined by The World Health Organisation (WHO) as ‘the personal characteristics and social resources needed for individuals and services to make decisions about health.’ [
[5]World Health Organization (WHO) Regional Office for South-East Asia
Health literacy toolkit for low- and middle-income countries : a series of information sheets to empower communities and strengthen health systems.
] The ability to use general literacy and numeracy skills in relation to health-related resources can also be broken down into five core components: obtention, comprehension, appraisal, communication and application of information [
[6]- Sørensen K.
- Van den Broucke S.
- Fullam J.
- Doyle G.
- Pelikan J.
- Slonska Z.
- et al.
Health literacy and public health: a systematic review and integration of definitions and models.
]. Difficulties with any of these stages can signify low health literacy, which has been linked to significant adverse health outcomes stemming from limited understanding of health and disease, including reduced engagement with healthcare providers, lower medication compliance and adherence, increased hospitalisations and increased mortality [
[7]- Jayasinghe U.W.
- Harris M.F.
- Parker S.M.
- Litt J.
- van Driel M.
- Mazza D.
- et al.
The impact of health literacy and life style risk factors on health-related quality of life of Australian patients.
]. A report by National Voices in 2017 declared health literacy ‘the strongest correlation to ill health – stronger than education level, deprivation, age or ethnicity’ [
[8]National Voices: People and Communities Board
A new relationship with people and communities: actions for delivering chapter 2 of the NHS five year forward view.
]. Readability is defined as the ‘ease of understanding or comprehension due to the style of writing’ and is frequently measured in the context of American ‘grade levels’ [
[9]The use of readability formulas in health care.
,
[10]- Shedlosky-Shoemaker R.
- Sturm A.C.
- Saleem M.
- Kelly K.M.
Tools for assessing readability and quality of health-related Web sites.
]. Readability formulas thus offer a means of ensuring a resource is suitable for its intended audience, with the aim of improving patient education and self-efficacy with regard to health.
Globally, one in ten adults lack ‘the most basic information processing skills considered necessary to succeed in today's world’, whilst almost 50% of adults in Europe are deemed to have ‘inadequate’ or ‘problematic’ health literacy [
[11]Organisation for Economic Co-operation and Development (OECD)
OECD skills outlook 2013: first results from the survey of adult skills.
,
[12]HLS-EU Consortium
Comparative report of health literacy in eight EU member states: the European Health Literacy Survey HLS-EU.
]. A publication from Public Health England and the Institute of Health Equity in 2015 demonstrated that in the United Kingdom (UK) 43% of adults lack adequate literacy skills, and 61% lack adequate numeracy skills, to routinely process health information [
[13]Public Health England & UCL Institute of Health Equity
Local action on health inequalities.
]. In the United Stated of America (USA), this rises drastically to 88% of adults not meeting the ‘proficient’ target [
[14]- National Center for Education Statistics
The health literacy of America's adults: results from the 2003 national assessment of adult literacy.
]. Whilst there is no consensus on the recommended level of reading difficulty for health information, Health Education England (HEE) advises that patient information should be written at a level comprehensible to an 11 year old in order to facilitate uptake [
]. This is equivalent to a US sixth grade level or less, which is also the level at which the American Medical Association (AMA) suggests health information should be written [
[16]Health literacy: a manual for clinicians.
].
Given the trend in current literature, we hypothesised that online PEMs about the monkeypox outbreak are written above the recommended level appropriate for the general public, which may be contributing to public misinformation and adverse health outcomes as a result.
Methods
Online PEMs were identified using the Google.com search engine. This search engine was selected because, according to the popular online web traffic analysis website StatCounter, Google searches comprised 92% of the online search market share at the time that this study was conducted [
[17]Search engine market share worldwide.
]. The search was carried out using the search term ‘Monkeypox’ on July 6, 2022. To eliminate any skewed or biased results based on previous search history or internet activity, the Internet browser was cleared of all search history, cache, cookies, and other user data.
The sample set for analysis was generated by compiling the first 50 English language webpages that contained patient orientated information aimed at the general public. Webpages were excluded for the following reasons: information pages written for healthcare professionals, studies from peer-reviewed journals, news articles, personal experiences, or, webpages that contained exclusively audio-visual material. PEMs that were solely audio-visual in nature were excluded because these could not undergo readability analysis as proposed in this study's methodology. By enforcing these exclusion criteria, we included only those webpages aimed at providing health education to the general population. To allow for categorisation, basic data on the webpages' characteristics were captured prior to analysis of readability. These data included: country of origin and URL domain name (i.e.gov,. com,.org, etc.).
Text from each webpage was entered into a Microsoft Word document and edited based on previously established protocols for readability assessment studies [
[18]- Kruse J.
- Toledo P.
- Belton T.B.
- Testani E.J.
- Evans C.T.
- Grobman W.A.
- et al.
Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
,
[19]- Kasabwala K.
- Agarwal N.
- Hansberry D.R.
- Baredes S.
- Eloy J.A.
Readability assessment of patient education materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation.
]. Components of the text unrelated to patient education were removed, as these may influence readability score. This included the removal of disclaimers, advertisements, webpage navigation, website URLs, copyright information, acknowledgements, author information, citation and references. Supplemental editing of non-textual elements and punctuation was performed including paragraph breaks, colons, and bullet points that may cause the readability tool to over- or underestimate the difficulty of a text [
[20]- Friedman D.B.
- Hoffman-Goetz L.
A systematic review of readability and comprehension instruments used for print and web-based cancer information.
].
Readability scores were generated for the edited text using an online readability software [
]. Text was assessed using five validated readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and, Simple Measure of Gobbledygook Index (SMOG). Each tool calculates readability by applying a mathematical formula to a passage of text (
Supplementary Table 1). The formulas each consider different factors when producing an ease of readability score. These factors include a combination of mean number of syllables per word, mean number of words per sentence, numbers of sentences and numbers of complex words (polysyllabic words) [
[9]The use of readability formulas in health care.
,
[10]- Shedlosky-Shoemaker R.
- Sturm A.C.
- Saleem M.
- Kelly K.M.
Tools for assessing readability and quality of health-related Web sites.
]. The FRES gives a score of 0–100, with higher score being deemed easier to read, whilst the other tools (FKGL, GFI, CLI, and, SMOG) give a reading grade level. The interpretation and target score for each tool is described in
Table 1.
Table 1Interpretation of each readability score.
FRES = Flesch Reading Ease, FKGL = Flesch-Kincaid Grade Level, GFI = Gunning Fog Index, CLI = Coleman-Liau Index, and SMOG = Simple Measure of Gobbledygook.
The methodology used in this study is consistent with other readability studies within the field of infectious diseases, public health, sexual health and other medical and surgical specialties [
[4]- Meleo-Erwin Z.
- Basch C.H.
- Fera J.
- Garcia P.
Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
,
[18]- Kruse J.
- Toledo P.
- Belton T.B.
- Testani E.J.
- Evans C.T.
- Grobman W.A.
- et al.
Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
,
[19]- Kasabwala K.
- Agarwal N.
- Hansberry D.R.
- Baredes S.
- Eloy J.A.
Readability assessment of patient education materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation.
,
[22]The readability and suitability of sexual health promotion leaflets.
,
[23]- Gao B.
- Shamrock A.G.
- Gulbrandsen T.R.
- O'Reilly O.C.
- Duchman K.R.
- Westermann R.W.
- et al.
Can patients read, understand, and act on online resources for anterior cruciate Ligament surgery?.
].
Data analysis
Webpage characteristics were described using rounded frequencies (per cent). Overall, and sub-group (based on webpage characteristics) FRES, FKGL, CLI, SMOG, GFI and median grade scores were described as mean values ± standard deviation, having been found to be normally distributed using the Shapiro–Wilk test of normality. The median grade score was calculated from the tools that compute the result as a grade score (FKGL, CLI, SMOG, GFI). The median grade score was calculated as the grade scores for each individual webpage were not normally distributed. Unpaired t-test was performed to determine if a government domain extension (.gov) compared to a non-government domain extension (.org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us.) influenced readability. One-way analysis of variance (ANOVA) was performed to determine if country of publication (United States of America, United Kingdom, Canada or Other, which included Australia, Ireland, Netherlands, Singapore, Switzerland) had an effect on readability. All analyses were performed in R (v4.2.1).
Results
Fifty webpages were analysed (
Supplementary Table 2). When separated by webpage characteristics (
Table 2), 29 (58%) of these were.gov webpages, and 21 (42%) had other URL domain names, which included:.org,.com,.ca,.uk,.ac,.ch,.ie,.int,.nhs,.nl,.scot,.us. When divided by country of origin, 34 (68%) webpages originated from the United States of America (USA), 5 (10%) originated from the United Kingdom (UK), 4 (8%) originated from Canada, with the remaining 7 (14%) originating from a variety of other countries, including Australia, Ireland, Netherlands, Singapore, Switzerland.
Table 2Webpage characteristics.
USA = United States of America, UK = United Kingdom.
Distribution of webpages by difficulty level can be found in
Table 3. Using FRES, none of the webpages met the target ease of reading score of ≥80. For the tools that gave a score representing a school grade, or years of education, the GFI and CLI tools found no webpages with a readability score at the target level of sixth grade. The FKGL and SMOG tools identified one (2%) and two (4%) webpages respectively that met the target level of sixth grade, with both the FKGL and SMOG identifying the same webpage in one instance.
Table 3Distribution of webpages by difficulty level and mean readability scores.
FRES = Flesch Reading Ease, FKGL = Flesch-Kincaid Grade Level, GFI = Gunning Fog Index, CLI Coleman-Liau Index, and SMOG = Simple Measure of Gobbledygook.
For all webpages, the FRES ranged from 34.6 (difficult, or College level) to 69.8 (standard difficulty, or grade 8 to 9), with a mean score of 54.4 (
SD = 8.5). FKGL ranged from 7 to 12.8, with a mean score of 9.6 (
SD = 1.6). GFI ranged from 8.9 to 16.6, with a mean score of 12.4 (
SD = 1.8). CLI ranged from 9 to 14, with a mean score of 11.1 (
SD = 1.3). SMOG ranged from 6.6 to 12.4, with a mean score of 9.4 (
SD = 1.3). Separating the webpages by ‘.gov’ sources and ‘other URL domain names’ or by county of publication showed no influence on readability (
Table 4,
Table 5).
Table 4Mean readability by County of origin.
USA = United States of America, UK = United Kingdom.
Table 5Mean readability by URL domain name.
Discussion
Health literacy is an important subject across all specialities, including both infectious disease and sexual health. Whilst not a sexually transmitted infection, PEMs created about monkeypox rely on patients having an ‘ability to understand sexual health information and application of that information’ to be able to make informed choices with regard to sex and safe sex practices, also defined by the WHO as sexual health literacy [
[24]- Rakhshaee Z.
- Maasoumi R.
- Nedjat S.
- Khakbazan Z.
Sexual health literacy, a strategy for the challenges of sexual life of infertile women: a qualitative study.
]. Electronic health literacy is also crucial in our increasingly technologized world, and refers to the processing and understanding of health information from electronic sources. In contrast to those with high electronic health literates, research suggests people with low electronic health literacy levels are significantly more likely to believe in misinformation and also to experience information overload from online sources [
[25]- Kim H.K.
- Tandoc Jr., E.C.
Consequences of online misinformation on COVID-19: two potential pathways and disparity by eHealth literacy.
]. Although the internet remains the top source of health information for the vast majority of people as it is both accessible and offers anonymity, it is apparent from other readability studies in both infectious disease and sexual health that the majority of PEMs available online are pitched at audiences above the recommended reading level [
[4]- Meleo-Erwin Z.
- Basch C.H.
- Fera J.
- Garcia P.
Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
,
[18]- Kruse J.
- Toledo P.
- Belton T.B.
- Testani E.J.
- Evans C.T.
- Grobman W.A.
- et al.
Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States.
,
[22]The readability and suitability of sexual health promotion leaflets.
]. Our results are consistent with these findings, demonstrating that none of the fifty total sampled PEMs regarding the monkeypox outbreak identified via Google search were of an appropriate readability for the general public, based on the results of all five validated readability tools, and thus are likely to be suboptimal as educative resources.
The COVID-19 pandemic previously underscored the scale of the ‘infodemic’ we are currently facing as a result of both increased information consumption, especially via a digital means, and global health illiteracy [
[26]COVID-19 as an "infodemic" in public health: critical role of the social media.
]. An infodemic is defined by the WHO as an ‘overabundance of information - some accurate and some not - that occurs during an epidemic’, which can hugely hinder the evidence-based approach to managing a public health crisis, and make it difficult for the general public to make informed decisions [
]. The COVID-19 pandemic is even more topical when we consider the heightened need for individuals to be able to respond to health information at speed. Health advice relating to monkeypox has been evolving almost daily, which requires the general public to be able to rapidly acquire and apply health information. Low health literacy rates can impair quicker processing times and, combined with online information perhaps unintentionally better suited to a higher readability than recommended, may be further complicating people's ability to comprehend and utilise this information to make informed health decisions.
Limitations
There are several limitations of this study. Firstly, as noted in other readability studies, information changes rapidly online and search results can change day on day [
[4]- Meleo-Erwin Z.
- Basch C.H.
- Fera J.
- Garcia P.
Readability of online dengue materials: the need for accessible information as part of infectious disease prevention and control efforts.
]. Due to the cross-sectional design of a study such as this, online information is only captured at a snapshot in time. It is also important to consider that readability does not alone give a complete assessment of how understandable a webpage may be. Other factors such as visuals, both pictographic and audio-visual, and design elements, including font size and the amount of white space, have also been shown to influence comprehension of patient education information [
[28]- Doak L.G.
- Doak C.C.
- Meade C.D.
Strategies to improve cancer education materials.
]. None of the readability formulas used were designed to consider the effect of these design elements. Future studies of PEMs created for monkeypox should examine ‘understandability’, likely using the Patient Education Materials Assessment Tool (PEMAT); ‘quality’, likely using with the DISCERN instrument; and, ‘accountability’, likely using the Journal of the American Medical Association (JAMA) benchmarks of accountability, which are ‘authorship’, ‘attribution’, ‘disclosure’, and ‘currency’. This study was also limited to material written in English and did not include pamphlets or other downloadable content. Future research should assess readability of downloadable content, including pamphlets, and readability of non-English material, especially given the international nature of this public health issue. This study also did not include an analysis of which polysyllabic words were used most frequently. An analysis such as this could reveal the vocabulary contributing to the more difficult readability scores.
Another limitation of this study was that the sample was limited to the first 50 webpages returned by the search term. This sample size is, however, in keeping with the design of other readability studies and in the case of this study appears to have given representative findings, given the homogeneity of our results. Furthermore, individuals most commonly access the top webpages returned from any given search, therefore our sample is representative of the webpages that the typical lay individual would access.
Despite the limitations of readability tools, they remain an efficient way of testing ease of readability and their use should be encouraged so as to aid those designing PEMs to carefully select their words and content in a way that is mindful of the literacy levels in the general population. Future studies of online patient information regarding monkeypox might also assess quality of the content. Nevertheless, the findings of this study have started to fill a gap in the available literature.
Article info
Publication history
Published online: December 21, 2022
Accepted:
November 3,
2022
Received in revised form:
November 3,
2022
Received:
August 16,
2022
Publication stage
In Press Corrected ProofCopyright
© 2022 Australasian College for Infection Prevention and Control. Published by Elsevier B.V. All rights reserved.