This page provides details of frequently asked questions related to evaluating public engagement projects or activities:
- What is evaluation?
- What is the difference between monitoring and evaluation?
- Why evaluate?
- What kind of evaluation is appropriate for my project?
- What is the difference between outputs and outcomes?
- How do I know if my project was a success?
- Is evaluation something I do at the end of the project?
- What methods or tools can be used to evaluate?
- What makes a successful evaluation?
- What should I do once the evaluation is complete?
- I’ve read all the FAQs here. Where can I get further information on evaluation?
What is evaluation?
Evaluation is a systematic way of reflecting on and assessing the value of what is being done (i.e. a project, a programme, an event). Evaluation is commonly interpreted as an end product or an activity taking place at the end of a project. However evaluation should be considered as a process, taking place across all phases of a project, used to determine what has happened and whether the initial aims of the project have been carried out and achieved.
Evaluation is more than assessing and measuring; it helps set the stage for a culture of learning, change and improvement.
What is the difference between monitoring and evaluation?
Evaluation involves monitoring, but they aren’t the same thing. Monitoring is the collection of data to measure what is being done. The data and information collected during monitoring can be used to help the evaluation. Evaluation is about using monitoring and other information to make judgments on your project, programme or activity.
Evaluation is essential to learn from projects, programmes or activities. Evaluating can help you to:
- Provide evidence of what works and doesn’t work to interested parties including funders and colleagues
- Learn from and improve your practice and performance
- Show others that your work is effective
- Understand and share the impacts of your work
Evaluation can be useful to provide feedback to your funders and partners, which can ultimately help to justify future funding for projects, activities or events. A funder who can see that an applicant is committed to evaluation knows that their money won’t be wasted. Even if a project doesn’t succeed in meeting the initial aims, the learning about why this happens will be valuable to other people involved in similar projects or activities.
What kind of evaluation is appropriate for my project?
There is no “off the shelf” or “one size fits all” approach to evaluation. There are various approaches to evaluation, ranging in focus (i.e. assessing processes, outputs, outcomes and/or impacts), timing (i.e. throughout the life or at the end of project), audience (e.g. funders, stakeholders, beneficiaries) and methods used (e.g. quantitative and/or qualitative). Ultimately, decisions over what to investigate, for whom and for what purpose determine how the evaluation is approached.
So, a good starting point when planning a project evaluation is to think about the aims of the project – the changes or benefits that you want to achieve. If you’re clear about the aims it will help you to plan the project. This focus, in turn, shapes the criteria on which to judge the progress and success of the project, as well as what information you need to collect
What is the difference between outputs and outcomes?
Outputs can be defined as direct, measurable results of activities undertaken (e.g. number of people involved, amount of funding received). Outcomes relate to the changes resulting from such activities, (e.g. what has changed, why it has changed, how changes relate to activities and how these changes fit within the aims of the programme).
Although it is easiest to measure outputs of activities undertaken, it can be more useful to understand the outcomes of an activity. For example, if a learning event (such as a lecture) is organised, it is easier to measure the numbers attending (the output) than the learning that occurred as a result of the event (the outcome). However, it is more useful to know what impact the event has had (i.e. if the audience learnt anything), and the quality of this impact.
Under some circumstances it can be appropriate to measure this impact in a binary way (e.g. an increase or decrease in learning), but learning from information limited in this way can be difficult. As an alternative, it can be useful to find out more about people’s experiences, and use these to build a more textured picture of the impact.
There is a need to measure what is important, rather than making things that are easy to measure sound important. Assessing the multi-layered outcomes of projects is often complex, but articulating and capturing the outcomes of projects will provide a richer understanding of the achievements of your activities.
How do I know if my project was a success?
You can’t tell whether your project was a success unless you have decided what it was supposed to accomplish. A good project should have a positive impact on all stakeholders involved. In the UCL Public Engagement Unit, we often think of the potential impacts of projects on a grid such as this:
|Knowledge and awareness||Attitudes||Skills||Empowerment|
Participants (i.e .staff)
|Audience (i.e. publics)|
However, it is recognised that the answer to questions about success depends upon the evaluation undertaken.
Is evaluation something I do at the end of the project?
Ideally, evaluation should be considered as a process, taking place across all phases or stages of a project. Evaluation activities (i.e. looking at the purpose, the focus, who is involved, the methods used) should be planned at the start of the project, rather than being an add-on at the end.
What methods or tools can be used to evaluate?
Once you know what you want to evaluate there are lots of different methods and models that can be drawn upon. The more you think about the purpose and scope of your evaluation, the more you will be drawn towards particular methods of evaluation.
The list below outlines some of the most common evaluation techniques:
- Interviews (e.g. structured, semi-structure, face-to-face, telephone)
- Surveys (e.g. online, postal)
- Poster exercises or activities (e.g. graffiti walls, posing questions on posters at event, stickers on charts)
- Self reflection tools (reports, case studies)
- Focus groups
- Meetings or workshops
- Diaries (e.g. video, log books)
You shouldn’t restrict yourself to this list – if thinking about your project aims and the focus of the evaluation has given you ideas for evaluation methods don’t be afraid to experiment with them.
What makes a successful evaluation?
A successful evaluation generates outcomes that are valid and reliable, whilst indicating directions for improvements. From our experience, your evaluation is more likely to succeed if:
- The evaluation is planned at the start of the project, rather than being an add-on at the end;
- As far as possible, monitoring and evaluation systems are integrated into project practice;
- Enough time is set aside for monitoring and evaluation activities, including organising and processing the data;
- All project stakeholders understand the purpose and value of the evaluation;
- The evaluation plans are both realistic and transparent (i.e. clear about what is measured, why and how);
- It describes what works well and what doesn’t;
- It gives directions or actions for improvements;
- Uses several methods, combines a range of data and perspectives;
- The findings from the evaluation are shared.
What should I do once the evaluation is complete?
Good evaluation will be useful to you, because it will help you to improve your work. It will also be useful to other people, and therefore you should write it up in a format that other people can use and understand. A short evaluation report, could be written up detailing what your project was, what your aims were, how you evaluated and what you found out.
You could share the final report with other people in your department, as well as supplying the UCL Public Engagement Unit with a copy that we can share across the University via the website. If your project has been funded, your funder will also want a copy of the report.
Don’t be afraid of sharing the evaluation if you find that you haven’t met your aims. We’ve all taken part in lots of activities that didn’t meet their aims – the crucial thing is to learn for next time, rather than continuing to repeat the same mistakes.
I’ve read all the FAQs here. Where can I get further information on evaluation?
The UCL Public Engagement Unit are happy to help with evaluation of your projects. If you have any questions on evaluation or would like any advice please contact the team at firstname.lastname@example.org
UCL works closely with the National Co-ordinating Centre for Public Engagement, who are trying to improve interaction between all UK universities and members of the public. Their website includes a page collating the very best guides to evaluation.
Page last modified on 02 may 13 11:52