Evaluating the impact of health information - HTML version

Executive Summary

This guide provides practical support for evaluating the impact of health information. It offers tips on best practice, evaluation models, and the tools you can use to collect and analyse your data. It is taken from a wide-ranging report(1) published by PIF in 2022.

This guide will provide an overview to help you plan your approach. It includes links to other sources of useful information.

It covers support for newly produced health information as well as advice on evaluating older legacy materials and content.

Information sits alongside diagnosis, treatment and care as an intervention that can influence people’s healthcare outcomes.

Key points

Evaluating the impact of information::

• Tells us if we are improving people’s health outcomes.
• Allows us to see if we are making a positive difference for patients and the public.
• Can reassure our funders about the impact of their investment.
• Tells us if the information we are providing is inclusive, accessible and fit for purpose.
• Is a vital element of acquiring and maintaining the PIF TICK(2).
• Allows you to understand what works well and what might not be so effective.

Evaluation models

Common evaluation models used in health information map to PIF TICK. These include:

• Learning cycle
• Plan, Do, Study, Act
• Logic Models
• Theory of change

Evaluation methods

A range of methods and tools can be used to evaluate new and legacy content. Combining quantitative (data) and qualitative (opinion) feedback methods often produces the best evaluation.

This guide supports the following PIF TICK criteria:

10. Impact: The impact of information is measured.

1. https://pifonline.org.uk/resources/evaluating-the-impact-of-health-information/

2. https://pifonline.org.uk/pif-tick/about-pif-tick/

Best practice

Evaluating the impact of your health information should be an integral element of your work. To do this well, start with a clear outline of:

• What would you like your health information to achieve.
• How will you know if you have been successful.

Remember to ask patients, healthcare professionals and users what success looks like to them. This may be different to your assumptions.

This outline will help you clarify the data you need to collect to measure your progress. Ensure you engage with a diverse range of stakeholders to help you plan as effectively as possible.

How to start evaluating impact

Evaluating impact does not need to start on day one of a project, but you need to be planning for it from the outset.

Establish your starting point and think about how you describe or assess what this is. How will you know that things are improving for your audience if you do not know how things are to start with?

Be structured about evaluating impact from the start, building in proposals for evaluation into any project documentation.

Think about what models for evaluating impact you are going to use and what tools and approaches to measuring or assessing impact you will use. The following sections will discuss this in more detail.

Be prepared to change

To ensure your project is progressing as intended, you need to collect data and insight throughout the development process. This allows you to demonstrate progress or any problems. Reviewing your plans with your key stakeholders can provide additional insights into ways to improve your impact. This approach of ongoing learning will ensure that considering and improving quality is an integral part of your work.

Try to share information in a way that will resonate with the people you are working with. Some audiences like to see numbers and graphs. Others find individual stories and insights far more engaging. Be flexible when presenting feedback.

1. Why evaluate impact?

There are many reasons to measure impact. Fundamentally, we are trying to tell if our work is making a difference. Your impact measurement can support multiple objectives. These include:

Improving the quality of your information

To improve, you need to determine your starting point, assess and measure what you have achieved, and identify where you want to go in the future. The Perfect Patient Information Journey – recommends a cyclical approach to information review based on the principles of ongoing quality improvement.

Showing impact in the real world

How effective is the information to users? If it is not working, how can you find and use that data to improve your offer? Remember, negative results can be valuable too.

Demonstrating effective use of resources

Explore effective use of time, resources and funding. Plan the time and resources for each stage of your work so you can gain some insight and learn from the process.

Improving equality, diversity and inclusion (EDI)

Consider EDI issues and think about whether you are reaching your intended audience - who will benefit from your offer?  Have you defined this?

Meeting your community’s needs

Collaborate with people and communities to get honest feedback about your information provision.

Support for fundraising

Support your organisation’s fundraising efforts by demonstrating value to your stakeholders or by identifying gaps in provision. What’s missing from your audience’s information picture?

Meeting assessment criteria

You may already be a member of the PIF TICK or you may be working towards it.  This guide will help you meet Criteria 10 of PIF TICK and answer the question: “Does your information make a difference?” In other words, is it meeting an identified information need? See next section for more detail about PIF TICK and evaluation.

2. PIF TICK³

To successfully achieve the PIF TICK, one of the questions you will need to address is: Does your information make a difference? This is Criteria 10. See box for the criteria in full.

You can map the PIF TICK criteria to a learning cycle to think about how you might evaluate impact.  This is shown below. For more information on learning cycles see page 10.

PIF TICK criteria 10

Minimum: Evaluate whether objectives set for the resource have been met. This should be based on quantitative and qualitative feedback. For example, number of views or resources ordered and general feedback.

Desirable: A full impact evaluation for each piece of information has been undertaken. This should include:

• Reach – for example how many people have downloaded a resource
• Outcomes – was it useful to them?

PIF TICK evaluation cycle

Criteria 3 – How do you identify genuine information need?

What are the issues people face?

What gap or need are you trying to address?

Criteria 5 – How are users involved in the development of health information?

Criteria 6 – How do you meet the communication needs of users?

What are you planning to do?

Criteria 8 – How can users give feedback on your information?

How will you track your progress?

What have you learned?

Criteria 10 – Does your information make a difference?

How will you use your findings?

3. Key principles

The principles of evaluating impact apply to all types of health information and all channels, but the tools and approaches needed may differ.

Approaches to evaluating new content or projects will differ to the approach used to evaluate and review legacy content. Legacy content could include existing webpages or patient information leaflets. See section 6 and 7 for more information on tools and approaches.

Getting started

Start by considering the following questions.

What is the information gap you are trying to address?

1. How do you know this is what matters most to patients?
2. Have you engaged with your stakeholders?

What are you intending to do?

1. Is your information resource what the patient population needs?

What resources will you use?

1. Are you being too ambitious or not ambitious enough in what you are hoping to produce?

What would you like to achieve?

1. Why do you want to achieve this impact on health outcomes?
2. Who is your audience?

How will you know whether it has worked?

1. What is your starting point in terms of the information gap?
2. What approaches will you use to find out if your information fills this gap?

What measurements or other approaches will help you assess your information’s success or otherwise?

1. What tools will you use?
2. Will these tools use numbers, descriptions, or a combination of both?

Choose your tools according to your budget and resource

It is important to think about how you will analyse and use the information you collect. Many evaluation tools are freely available, but some do involve costs, so you will need to factor that into your decision-making process. You need to consider the capacity and expertise available to do the analysis.  If you cannot analyse the data, it will not add the value you are looking for.

Many of the tools available can be used through various channels, including online, on paper, via telephone, or in person. Your choices will be influenced by how you deliver your information or service, and how your audience might access these channels. Ask if your evaluation approach is likely to exclude any of your intended audience.

Defining the ‘reach’ of your information

Before looking at more in-depth impact tools, it is important to evaluate your outputs and their reach. Reach is about how you quantify what you are producing, and who is accessing it. Some of this information may already be collected by your organisation or is available publicly.

What you can evaluate

It’s helpful to think about evaluation in terms of outputs, outcomes and impact.

Outputs are a good starting point for evaluating the work. It’s critical that you evaluate whether you’re reaching the people you planned to, otherwise your information serves no purpose. Evaluation can consider the numbers you have produced and shared (outputs).  It can look at whether your information has filled the gap you identified (outcomes). It can also look at whether filling that information gap has led to any changes in action or behaviour (impact).

Outputs – How many leaflets you sent out, how many sessions you ran, or how many videos you published.

Outcomes – What did people do as a result? Did the leaflet provide the information they needed or direct them to the right service?

Impact – Did people feel that they had the information needed? Did they feel able to make informed decisions?

4. Evaluation models

A challenge when evaluating the impact of health information is that projects are almost always part of a complex system, with treatment, support and other information all contributing to a patient’s outcome and experience. In most cases, you need evidence to demonstrate that your health information works. Specifically, is it contributing positively towards someone’s overall experience? See the example below.

Measuring impacts for the system and patients

• Outputs
  An information leaflet which provides information about how and when an individual can manage the symptoms of their condition. It could also describe the signs and symptoms that would usually suggest they might need to seek additional clinical support.
• Outcome
  People feel confident to manage their condition effectively and know when they need to seek support from their clinical team.
• Impact
  Reduced emergency calls to the clinical team.
• Evaluation
  “If you didn’t have this information, what would you have done instead?” Use an open question, or with a range of choices e.g. called my GP, attended A&E, called my named nurse, tried to find information online.

Learning cycles

All health information should be produced as part of a cycle of activities. Start from a baseline of identifying the information need, outlining your intentions, and how you will track your progress. Build improvements into your information. Document what you have learned during the production of the information and how this learning will feed into the next version.

The logic model as a learning cycle

What are the issues people face?

What gap or need are you trying to address?

What are you planning to do?

How will you track your progress?

What have you learned?

How will you use your findings?

Learning cycles and PIF TICK

The learning cycle maps closely to the PIF TICK, combined they provide an excellent model to think about producing, evaluating and improving health information. See page 6.

Plan, Do, Study, Act (PDSA) cycles

Another way of planning how you will evaluate impact is to use the PDSA cycle. It was developed by the NHS as a model to describe the process of ongoing improvement.
You need to determine:

• What you want to do
• How you will do it
• How you will know the change is an improvement
• How you will roll out any improvement.

Act, Plan, Do, Study

Aqua has produced a useful guidebook on the use of the PDSA cycle, which is used within the NHS(4).

Logic models(5)

A logic model aims to summarise your project visually. It shows the relationships between the inputs, activities, outputs, outcomes and impact. It tends to be presented as a linear diagram.

The biggest challenge is finding the right tools to measure the outcomes and the impact of your work. In most instances, you will need a mix of methods. More information is provided in sections 6 and 7.

Once you have the findings from your logic model, think about the evaluation of impact as a cycle of improvement.

Use the data to inform the next stage in your project as you begin to understand whether you have delivered the outcomes and impact you planned. A logic model is effectively a simple theory of change.

INPUTS

e.g.

• People
• Funding
• Facilities
• Equipment
• Knowledge
• Expertise

ACTIVITIES

e.g.

• Workshops
• Training
• Creating resources
• Marketing + promotion

OUTPUTS

e.g.

• Number of leaflets produced
• Number of sessions run
• Number of participants

OUTCOMES

e.g.

• People have access to the information they need
• People understand their treatment options

IMPACT

e.g.

People feel better able to manage their condition
People know how and when to seek extra support

Theory of change

For large or complex projects a theory of change model can be a useful approach. It operates on the same principles as a logic model.

Developing a robust theory of change can take longer, and ideally you will develop it with your key stakeholders to ensure buy-in from your intended audience. Stakeholders might include patients or service users, carers, representatives from key partner organisations and members of your organisation.

A theory of change model enables you to show the complexities of the factors contributing to change, and the relationships between them. New Philanthropy Capital (NPC) provides detailed step-by-step guidance and offers training and expert support if required. The National Council for Voluntary Organisations(6) (NCVO) offers advice on approaches that can be used to help you create your theory of change.

Summary of Theory of Change in 10 steps (NPC)(7)

1. Situation analysis – identification of the problem or issue, and what you can offer.
2. Intended audience – who are the people or institutions you want to work with?
3. Impact – what is the sustained or long term change you would like to see?
4. Outcomes – what are the shorter-term changes for your intended group that will contribute to impact?
5. Activities – what are you going to do?
6. Change mechanisms – how will your activities cause the outcomes you would like to see?
7. Sequencing – map out the order in which you think outcomes and impacts will be achieved.
8. Theory of Change diagram – New Philanthropy Capital recommends trying to map all your information into a summary diagram.
9. Stakeholders and ‘enabling factors’ – what could stakeholders or organisations do that might affect your Theory of Change; what other factors might help or hinder you? For example political or environmental factors; personal circumstances affecting individuals.
10. Assumptions – this is intended to help you test your theory for any weaknesses or uncertainties.

6. Tools for evaluation and impact of new projects and content

There are lots of ways to gather evidence to evaluate the impact of your health information – it is important to choose the right tools for your project. This section outlines some approaches you might want to consider using, taking into account whether you want your evidence in numbers or descriptions, or both.

Demographics

If you are designing a survey or web form you can ask demographic questions to see who you are interacting with. This might include, for example, age, ethnicity, gender and education.

The more information you have, the more you can stratify into different user groups. Breaking down your reach into user groups provides more rich and interesting data and highlights potential gaps in your reach.

Once you have established the reach of your outputs you can start to assess the impact they may have overall and for specific groups.

Accessibility

Make sure your evaluation is accessible and health literacy friendly.

Consider simple options such as a pair of jam jars labelled “yes” and “no” and a bowl of marbles – or the online equivalent – with a simple question above. This could be something like: “Were we able to offer the support or information you needed today? Please place a marble in the most appropriate jar.” This can help you start to gather relevant data about your service.

When you are thinking about collecting data do not be afraid to be creative. Working with children? There’s no reason why you cannot gather data about how they are feeling by asking them to draw a picture rather than trying to conduct an in-depth interview. You can find lots of information and inspiring case studies in PIF’s Guide to Producing Health Information for Children and Young People.

What measurement and assessment tools can you use?

There are two main types of evaluation data, quantitative and qualitative. Both types of data are valuable and will provide different types of insight.

Starting your process with a clear outline of what success will look like helps you to choose the most appropriate methods to assess your progress. Frequently a mixture of quantitative and qualitative approaches will be needed.

Measuring impact

The following pages cover impact measurement tools:

• Surveys
• Validated questionnaires
• Interviews
• Complaints and compliments
• User stories and case studies
• Outcomes Star

The following pages list what type of data can be collected using these tools. This includes the pros and cons of each measurement tool, some broad comments about each approach, and some top tips. More than one tool may be needed to gather the data you need, see example on the next page.

Evaluating the impact of new webpages

• A survey could help you test the effectiveness of the work you have done to improve the visibility of your information through Search Engine Optimisation (SEO). This could provide evidence of your ability to reach the people who need it.
• An online focus group could help you gather feedback about your online community and identify if it is meeting people’s needs.
• You can add short feedback tools to individual webpages asking questions such as: Was this content useful? Did you find what you were looking for today? Some tools also provide the option for free text responses.
• Other methods of getting feedback include QR codes which link to surveys, simple emoji ratings scales, or tick boxes about the usefulness of the information.

Conclusion and feedback

Whatever tools you use and whatever data you gather,  make sure you use it to improve your health information. Creating an ongoing learning cycle will ensure quality improvement is an integral part of the way you work.

Feedback

We welcome your comments and suggestions about this guide. By giving us constructive feedback you can help us to produce higher quality and more relevant information in the future. If you would like to give your feedback, email info@pifonline.org.uk with your comments.

References

1. Patient Information Forum, Evaluating the impact of health information. pifonline.org.uk/resources/evaluating-the-impact-of-health-information/
2. Patient Information Forum, About PIF TICK. pifonline.org.uk/pif-tick/about-pif-tick/
3. Patient Information Forum, About PIF TICK. pifonline.org.uk/pif-tick/about-pif-tick/
4. AQA, Plan, Do, Study, Act (PDSA) cycles and the model for improvement. aqua.nhs.uk/wp-content/uploads/2023/07/qsir-pdsa-cycles-model-for-improvement.pdf
5. NHS Midlands and Lancashire CSU, Logic Model guide. midlandsandlancashirecsu.nhs.uk/images/Logic_Model_Guide_AGA_2262_ARTWORK_FINAL_07.09.16_1.pdf
6. NCVO, Theory of Change. ncvo.org.uk/help-and-guidance/strategy-and-impact/strategy-and-business-planning/theory-of-change/
7. NPC, Theory of Change in ten steps. thinknpc.org/resource-hub/ten-steps/
8. Insignia Health, Patient Activation Measure (PAM). insigniahealth.com/pam/
9. NHS, Patient Reported Outcome Measures (PROMs). digital.nhs.uk/data-and-information/data-tools-and-services/data-services/patient-reported-outcome-measures-proms

Further guidance

• Better Evaluation – betterevaluation.org/frameworks-guides/rainbow-framework
• The Health Foundation – q.health.org.uk/
• Health Improvement Scotland – healthcareimprovementscotland.scot/
• National Council for Voluntary Organisations – ncvo.org.uk/help-and-guidance/strategy-and-impact/impact-evaluation/
• National Voices – nationalvoices.org.uk
• Nesta – nesta.org.uk/blog/the-iron-triangle-of-impact-measurement/
• New Philanthropy Capital (NPC) – thinknpc.org/
• NHS England  – digital.nhs.uk/data-and-information/data-tools-and-services/data-services/patient-reported-outcome-measures-proms
• NHS Impact – england.nhs.uk/nhsimpact/
• Outcomes Star – outcomesstar.org
• Wales Measuring What Matters – info.copronet.wales/measuring-what-matters/
• UK Evaluation Society – evaluation.org.uk/
• UK Government WhatWorks – whatworks.blog.gov.uk/about-the-what-works-network/

Acknowledgements

Revised and updated by Victoria Thomas, Independent Patient and Public Involvement Specialist. June 2025.

Reviewed by:

Sheena Campbell – Communications Manager, Patient Information Forum 

Dr. Ruth Carlyle – Head of Library and Knowledge Services, NHS England

Selina Corkery – Service Manager (Editorial), Macmillan Cancer Suppor

Helen Leech – Editorial Manager, Information Team, Mind   

Sophie Randall – Director, Patient Information Forum

Dr. Knut Schroeder – GP, Founder, Expert Self Care

Julie Smith – Content Director, EIDO Healthcare

Based on the Perfect Patient Information Journey Part 3  - Evaluating the Impact of Health Information 2022. Written by Kim Diprose and Sophie Randall.