EVALUATION:

HOW IT CAN WORK FOR YOU

An explanatory paper prepared by:
Department of the Secretary of State
Pacific Regional Office,
1525 West Eight Avenue,
Vancouver, B.C.
Telephone:
732-4111

January 1981

GLOSSARY OF TERMS

EFFECTIVENESS:

the degree to which the objective and goals

of an activity are realized, measured by comparing real
results with expected results.

EFFICIENCY:

the rate and cost of achieving the actual results

of an activity.

A test of efficiency determines whether

the cost of producing the results was reasonable.

EVALUATION: process by which one attempts to measure the

degree to which a project is accomplishing its purpose.

EVALUATION RESEARCH:

the systematic collection and analysis

of information about a project or program, leading to

judgements about the project or program.

GOAL: a measurable statement of what one wishes to accomplish
within a specified period of time.

OBJECTIVE:

the essential purpose that motivates undertaking

a project or program; what one wishes to accomplish.

1)

WHY EVALUATION

The Secretary of State has received a number of requests from
citizen groups for help with evaluating their projects.
These
groups have wanted to find ways of taking a look at their
objectives and activities that would help them improve their
efforts.
This paper was developed in response to these requests.
Groups consider evaluation an important means of determining
whether they are achieving what they set out to do, or whether
they need to rethink their methods.
The Secretary of State
department asks groups to evaluate their programs in order to
determine whether public funds are being spent in the most
effective way possible.
Good evaluation reports can also assist
other groups who may plan similar projects in the future.

2)

INTRODUCTION

Projects initiated by citizen groups are usually limited in
duration, and started without any commitment from the funding
agencies to their long-term continuation.
The funding agencies
often see the project as an opportunity for innovation, for
trying out new ideas, new strategies, or new structures.
Citizen
groups, on the other hand, tend to perceive their projects as
providing a needed service, as solving a pressing problem, or as
resolving an urgent issue.
These are two very different percept­
ions of the same project.

The explanation of evaluation research in this paper is based on
two considerations:
- Projects initiated by citizen groups should be viewed both as
providing a needed service (solving a pressing problem,...) and
as an opportunity for producing useful information about the
needs for such a service, the nature of the problems being
tackled, the difficulties with implementing certain strategies,...
In other words, the projects should be seen as action research.
- Evaluation research is a tool which can both improve the project
as it develops, and produce a final report telling what has
been learnt through the project.
It can also be used to make
a case for further funding.
A good evaluation report allows
a funding agency to rely on a more balanced analysis than can
be presented through statistics, or a financial report.

3)

WHAT IS EVALUATION RESEARCH?
Evaluation is the process whereby you find out if you have
accomplished or are accomplishing what you set out to do.
Evaluation
research is the systematic collection and analysis of information
concerning a project or program, leading to judgements about the
project or program.

.

.

./2

- 2 -

A key step in setting up an evaluation process is to define clearly
and accurately what you wish to accomplish ( the objective).
In
order to evaluate your progress you will probably need to translate
objectives into a number of more short-term, specific goaZt.
Defining your objectives and goals clearly can be a time consuming
process but it is essential to carrying out a good evaluation.
The
process can also be very useful in the day to day management of
your project or program.
The process of evaluation, based on clearly defined objectives and
goals, will enable you to judge the
ecZZv
and e^ZcZenct/
of your project or program.
ecZZvene.44 is the degree to which
the objective and the goals are being realized.
This is measured by
comparing the real results of the project or program with its
expected results.
E^ZcZenci/ is the rate and the cost of achieving
the results.
A test of efficiency determines whether the cost of
producing the results was reasonable.
There are two main types of evaluation research:

Outcome evaluation compares the actual result of the project or
program with its stated objective.
Pa.occ44 evaZuaZZon measures how well you are progressing toward the
objective, as a means of helping the day to day decision making
of the project.
This type of evaluation concentrates on describing
the various activities of the project, and their effects, and on
making judgments about the effectiveness of these activities.

4)

WHAT ARE THE USES OF EVALUATION RESEARCH?

Evaluation research has been used for many purposes, to answer
different questions.
For example:
- to demonstrate to others that a project is worthwhile.

- to justify past or projected expenditures.
- to support project continuation, expansion, or redirection.
- to compare different types of programs and program methods.
-

to determine whether the needs for which a project was designed
are being met.

-

to determine whether the initial objectives were realistic.

There are at least four possible uses of evaluation from the
perspective of citizen groups:

a)

To Help Seek Further Funding

.

.

./3

- 3 -

Citizen groups spend a great deal of time and energy looking for
further funding.
Some of this time and energy could be saved,
and the chances of success increased, if the project is
structured to collect and analyze the information which funding
agencies require, as the project proceeds.

Note that funding agencies may view the project differently
from those who initiated the project.
The funding agencies
have to view the project within the context of their own
programs, objectives, and priorities.
On way of finding out
what information funding agencies require is to look at their
application forms.
It might be useful to do this at the
beginning of a project, long before further funding is necessary.

b)

To Manage the Project Effectively

It is extremely easy to become caught up in the day-to-day
activities of a project, squeezed between overwhelming demands
on limited resources and initially idealistic objectives.
There
is little time for reflection about what is being achieved, or
where the project is headed.
Evaluation research can be used
as a tool to collect information that would help project staff
(and board members) regularly take a look at what they have
been doing and what they have achieved.
Perhaps the most important
outcome of this approach would be a continuous refinement and
modification of the objectives of the project, making them more
feasible and specific, and easier to accomplish within the
time frame.

c)

To Obtain Feedback From Those Affected by the Project
The success of a project or program quite often depends on its
acceptance by the people it is designed to assist.
Evaluation
research, by regularly publishing an overall picture of the
project (through a newsletter, for example), could make
communication between staff and consumers easier.

d)

To Produce a Final Report that Other Groups Could Use
This report should describe the history of the project, the
objectives and the ways in which they were modified, the
activities and the ways in which they were altered, the effects
(good and bad, expected and unexpected), the obstacles
encountered, the environment of the project.
Thus, other
groups who are interested in pursuing objectives similar to
yours will be able to use your experience.

.

.

./4

- 4 5)

HOW DO YOU EVALUATE ?
Some important questions should be answered before starting
an evaluation:

Why do you want to evaluate ?
Like any project, an evaluation
needs a clear statement of objectives and goals in order to
succeed.
For whom are you producing the evaluation report ?
Information
that you may find useful for the day-to-day management of the
project may not necessarily meet the needs of a funding agency.

What are you going to evaluate ?

When should a report be available ?
What resources will be needed to carry out an evaluation ?
How much will the evaluation cost ?

6) WHAT NEEDS TO BE CONSIDERED IN SETTING UP AN EVALUATION
An evaluation should determine whether a project or program
is achieving the results it was designed to achieve.
In
order to carry out an evaluation a number of factors must be
considered: the objectives of the project, its environment,
the project activities, and the ultimate effects.
These
lead to an analysis and a final report.
The diagram shows
how information flows from one area to another.

a) Objectives
- What are the project’s objectives ?
- Can they be clearly stated at this stage ?

. . ./5

- 5 - Are the objectives measureable in any way ?

- Can the overall objectives be translated into goals for
specific periods of time ?

- Do the objectives appear realistic given the resources
(staff, time, energy, money, facilities) of the project ?
- Are some objectives more important than others ?
If so,
how is this being taken into account in the operation of
the project ?

- What specific effects or changes are considered desirable ?
- Who is supposed to be affected by the project ?
b)

Background knowledge
- What is known about the nature of the problems being tackled ?

- What is known about the community in which the project is
operating ?
- What assumptions are being made about the way people,
groups, or organizations change ?
- If the project has a previous history, what can be learned
from it ?

- What can be learned from similar projects carried out
elsewhere ?

- What is the general environment: geographic,
political and economic of the project ?

social,

c) Project activities
- Is the project aimed at individuals, groups, communities,
organizations or the entire society ?

- What are the various project activities ?

- How are these activities supposed to further the objectives ?
- Who is carrying out these activities ?

- What skills do project staff need ?
- What attempts are being made to help project staff develop
these skills ?
. . ./6

- 6 d) Effects
- Who is/was affected by the project ?
- How are/were they affected ?

- Are there long-term effects ?
- What unexpected effects

(good or bad) are occurring ?

e) Analysis

- Is the project proceeding as planned ?
changes have been made and why ?

If not, what

- Are the resources of the project being allocated according
to the priorities agreed upon ?
- Are the expected effects being achieved ?
- Are the objectives being achieved ?
If not, is this because
the assumptions underlying the project (connecting activities
to objectives) are not valid ?

- Is the project reaching the people it was supposed to reach,
and does the project have their support ?
If not, why not ?
- Where the initial understandings and estimates of the problems
that the project is supposed to tackle accurate ?

- Is the project trying to do too much ?
If so, which
activities and objectives need to be given top priority,
and which can be left for the future ?
- Are undesirable effects occurring ?
- Are there factors, not under the control of the project, that
are affecting the success of the project ?

- What obstacles are being encountered, and what steps are
being taken to overcome them ?

7)

SUGGESTIONS FOR COLLECTING INFORMATION

a)

Project historian
It is probably useful for one person in the project, as

. . ./7

- 7 -

part of their job, to take on the role of project historian:
to take responsibility for the collection, analysis, and
reporting of information.

b)

Logbooks

One means of ensuring that an accurate and full description
of the project activities is kept is for each staff person
to fill out a daily logbook, describing their day’s activities,
the accomplishments,
the obstacles encountered, the people
contacted, etc.
c) Data collection sheets

In those projects which have a great deal of regular contact
with people outside the project, it is useful to develop a
standardized data sheet, which can be filled out for each
contact.
The sheet should be as simple as possible, detailing
the purpose of the contact, the date, the outcome, any need
for further action.