Investigation 1 draws a blank
How easy is it for people to judge the quality and reliability of YouTube videos on Covid and Covid vaccines? And what can be done to make such judgements easier for people?
As part of my studies towards an MSc in public health, I searched the biomedical literature to find out. Disappointingly, I found no studies with the main aim of answering these particular questions.
As a PIF member, I think it’s worth pointing out straight away that the PIF TICK, a UK-wide quality mark for trusted health information, is one thing that seems well placed to help here. No matter what the overall quality of health-related videos on YouTube, the PIF TICK would help people to identify videos they can trust.
Investigation 2 offers learnings for PIF members from 25 studies
As the questions asked in my first literature search are quite specific, I broadened the line of enquiry and found 25 studies that evaluated nearly 3,000 YouTube videos on Covid and/or vaccines with over 1.7 billion views.
The studies recommend ways to help ensure information on YouTube about Covid and vaccination is reliable and high quality. They also detail how they go about judging quality.
I think they offer valuable learnings for PIF members who are creating, or signposting to, videos on any health-related matter, so I’ve shared findings here.
But first, why is this important?
YouTube is the second-most-visited website globally, after Google.
Over 1 billion hours’ worth of video is streamed on YouTube each day.2 Over 2 billion unique users visit the site monthly.
Given YouTube’s vast reach, it seems important that people can judge the reliability of videos they find, to understand what might help people to make such judgements and to know what videos on YouTube are like and how to make them better.
What recommendations did the studies make?
Between them, the studies made 28 different recommendations. The top five, in terms of number of mentions, are listed below:
- Increase the presence and visibility of videos from reliable and authoritative organisations
- E.g. videos from the World Health Organisation, national health agencies, and medical or academic institutions
- It would be interesting to know what PIF members think of sharing their content on YouTube
- Encourage more collaboration in video production between different groups/sectors
- E.g. between public health agencies or hospitals and the entertainment or news industries or social media influencers
- Again, I wonder how many PIF members join, or would like to join, such collaborative efforts
- Monitor the content and quality of information
- Use YouTube to disseminate public health information, e.g. WHO recommendations
- Block misleading videos and/or increase editorial control.
What is YouTube doing?
I can’t claim any expertise here, and YouTube is best placed to answer this, but many of the studies mentioned YouTube’s Covid-19 Medical Misinformation Policy, announced in May 2020. This policy states that YouTube does not allow medical misinformation about Covid-19 that poses a serious risk of egregious harm.
YouTube is also planning to showcase videos produced by credible outlets in a ‘health content shelf,’ near the top of search results for a few hundred health-related queries.
Such efforts seem important. And PIF members may want to be on that list of credible outlets.
It seems equally important to do what we can to help people to judge videos for themselves. It is worth plugging the PIF TICK again, because this quality mark is well placed to help.
How did the studies go about evaluating the quality of YouTube videos on Covid?
Combinations of approaches were used:
- Words: When using words to describe what high-quality videos are like – what ‘good’ looks like – the studies used a range of terms, including quality, reliability, understandability, actionability, completeness, relevance, depth and accuracy.
- Tools: The researchers used questionnaires and other tools designed for evaluating consumer health information, such as the DISCERN and modified DISCERN questionnaires, the Global Quality Score (GQS), the Journal of the American Medical Association (JAMA) benchmark criteria and HonCode (the Health on the Net Foundation Code of Conduct).
- Topic/content checks: Content coverage of the videos was checked against lists of topics deemed most relevant to Covid-19 and/or vaccines, with several researchers using a checklist called the Medical Information and Content Index (MICI).
- Fact checks: Claims made in videos were compared with information from authoritative sources (such as the World Health Organisation).
- Other characteristics, e.g.:
- Tone (e.g. whether pro or anti-vaccines)
- Categorisations (e.g. useful/misleading)
- YouTube metrics (such as number of views, likes and comments)
- Type of person or organisation that uploaded the video (e.g. public health organisations, news agencies, healthcare professionals and consumers)
- Other things such as the duration and language of videos
- It’s important to note that these characteristics aren’t necessarily all indicators of quality. The studies investigated relationships between quality and characteristics such as number of likes, with varying results.
A big thank you to my tutors at Brighton and Sussex Medical School
This blog summarises some of the findings of a literature review that I conducted as part of my studies towards an MSc in public health at Brighton and Sussex Medical School. I hope it helps.
I’d like to thank my tutors for their support, particularly Drs Louise Sigfrid, Priya Paudyal and Marija Pantelic, and for running the course in a way that enables students to pursue lines of investigation that fit with their own work and interests.
- Databases searched: PubMed, Embase, PsycInfo, Cochrane and MedRxiv.
- Chan C, Daniels E, Normahani P, Markar S, Sounderajah V, Acharya A, et al. The Reliability and Quality of YouTube Videos as a Source of Public Health Information Regarding COVID-19 Vaccination: Cross-sectional Study. JMIR public health and surveillance. 2021;7(7):e29942. Available from: https://publichealth.jmir.org/2021/7/e29942/
- YouTube. COVID-19 medical misinformation policy 2021. Available from: https://support.google.com/youtube/answer/9891785?hl=en-GB.
- Grant Currin. YouTube’s plan to showcase credible health information is flawed, experts warn. Scientific American, August 27, 2021. Available from: https://www.scientificamerican.com/article/youtubes-plan-to-showcase-credible-health-information-is-flawed-experts-warn/