Definition and characteristics of an indicator
An indicator can be defined as the measurement of an objective to be met, a resource mobilised, an effect obtained, a gauge of quality or a context variable. An indicator produces quantified information with a view to helping actors concerned with public interventions to communicate, negotiate or make decisions. Within the framework of evaluation, the most important indicators are linked to the success criteria of public interventions.
In order to be useful it is preferable if an indicator has the following characteristics:
- The indicator definition is closely linked to a policy goal, objective and/or target. (Indeed, indicators are most helpful when objectives have been specified in terms of targets or milestones that apply the definition of the indicator).
- The indicator is measured regularly. It is helpful to have time series information where the precise indicator definitions have been applied consistently. Ideally data should be available from prior to the adoption or implementation of the intervention. However, interventions often themselves call for new data to be collected.
- Steps are taken to ensure data gathered is reliable. For example, for output and some result indicators where data are provided directly by the projects, sample checks should verify the data. For impact indicators, it is probably better that data are measured on an independent basis.
In practice indicators rarely exhibit all of these characteristics and it is likely to be necessary to gather evidence from a variety of disparate sources including:
- The inputs to and timing of the programming process;
- Secondary sources;
- Primary sources, including Stakeholder surveys;
- Administrative information.
Much of this information may have been gathered for purposes other than evaluation.
An indicator quantifies an element considered to be relevant to the monitoring or evaluation of a programme. For example: "1200 long-term unemployed received training financed by the programme" or "75% of the participants in training courses claim to be satisfied or highly satisfied".
A good indicator should provide simple information that both the supplier and the user can easily communicate and understand. This is, however, a necessary but not sufficient quality. The following are examples of indicators that are readily understood: rate of budget absorption; percentage of regional firms assisted; number of net jobs created; and number of jobless in the eligible area.
An indicator may have several values over time. The unemployment rate, for example, may have a different value at the outset from a value taken mid-way through the implementation of a programme, and so on. Variations over time constitute trends.
Type of indicators
There are several typologies of indicators:
- In relation to variables: Complete, partial and complex indicators
- In relation to the processing of information: Elementary, derived and compound indicators
- In relation to the comparability of information: Specific, generic and core indicators
- In relation to the scope of information: Context and programme indicators
- In relation to the phases of completion of the programme: Resource, output, result and impact indicators
- In relation to evaluation criteria: Relevance, efficiency, effectiveness and performance indicators
- In relation to the mode of quantification and use of the information: Monitoring and evaluation indicators
The most useful of these typologies for socio-economic programmes is the distinction between: resources; outputs; results and impact indicators. Contextual indicators, which can be the same as impact indicators in some cases, is a further useful category.
Resource indicators provide information on the financial, human, material, organisational or regulatory means used by to implement programmes. Resources are the joint responsibility of the financing authorities, which allocate them, and the operators who implement them. Most resource indicators are regularly quantified by monitoring systems. Examples of resource indicators include: the total budget (quantity of resources); annual budget absorption (resource absorption rate); percentage of expected over/under spending; percentage of European financing in the total public financing; number of people working on the implementation of the programme; number of organisations involved in the implementation.
Output indicators represent the product of the programmes activity. More precisely, an output is considered to be everything that is obtained in exchange for public expenditure. Outputs are normally under the entire responsibility of operators who report on them through the monitoring system. Examples of output indicators include: kilometres of roads built; progress rate of the building of a road; hectares of urban wasteland rehabilitated; capacity of purification plants built; number of trainees whose training was paid by the programme; and percentage of this training of which the quality is certified.
Result indicators represent the immediate advantages of the programme (or, exceptionally, the immediate disadvantages) for the direct beneficiaries. An advantage is immediate if it appears while the beneficiary is directly in contact with the programme. The full results may be observed when the operator has concluded the action and closed off the payments. Result indicators are generally easily known to the operators, so they are generally quantified during monitoring.
Result indicators provide information on changes which occur for direct beneficiaries, for example, time saved by users of a road; reduced rates for telephone calls; qualifications earned by trainees; new tourist activity generated by a farmer; use of new productive capacity created by a firm; and the satisfaction of businesses which have received consultancy services.
It is at the time that beneficiaries receive support or programme services that results can be quantified. Either direct measurements are made (e.g., by counting the number of trainees recruited during their training) or the direct beneficiaries are asked to state the advantages they have obtained (e.g., by means of a questionnaire at the end of a consultancy mission).
Impact indicators represent the consequences of the programme beyond its direct and immediate interaction with the beneficiaries. An initial category of impacts group together the consequences for direct beneficiaries of the programme, which appear or which last into the medium term (specific impacts), e.g., traffic on a road one year after it is opened; the placement rate of trainees after twelve months; sustainable jobs created in an industrial plant built with programme support; and the survival rate of businesses created with programme support. Some impacts are unanticipated (spin-offs) but indicators are rarely created for unanticipated impacts.
A second category of impacts consists of all the consequences that affect, in the short or medium term, people or organisations that are not direct beneficiaries. These impacts may be similar (e.g., improvement of the quality of life for people living near a rehabilitated industrial wasteland; improvement in the quality of beaches near a new purification plant). They may, in contrast, spill over to affect people or organisations far from the programme, as in the case of macro-economic impacts.
The mechanisms of impact propagation can be separated into two categories: market effects (e.g., impact on suppliers or sub-contractors of the assisted firms) and non-market effects (e.g., positive impact of the improved image of the region or negative impact of a deterioration in the environment). Because non-market effects or externalities are not reflected in the price system on which individual socio-economic actors largely base their private decisions and because these decisions have economic consequences for other actors, it is particularly useful to take these effects into account in the context of a public programme.
Because of the time lag or their indirect nature, impacts cannot easily be known to operators during their daily management of the programme. Impact indicators are therefore quantified from time to time only, usually during evaluations. One way of establishing impacts is to carry out a survey of direct beneficiaries, for example a year after they have left the programme. The questions asked might concern facts (e.g., how many new jobs have been created since the support was obtained?) or opinions (e.g., how many jobs would have been lost without the support?). When analysing values for impact indicators, evaluators may pay particular attention to causality. To what extent do those values relate to the programme or, intervention being evaluate and what other factors may have had an influence?