Indicators serve a number of useful roles in evaluation. The use of indicators normally forms part of an evaluation particularly where objectives are expressed in clear operational terms. The information they provide needs to be carefully interpreted in the light of other evidence in order that evaluative conclusions can be drawn. Indicators have the potential to contribute to the evaluation of socio economic programmes in several ways:
- The analysis of the indicator scores can be used to provide support for a rationale for intervention and resource allocation.
- Indicators can be used to compare inputs and outputs in order to measure efficiency.
- Indicators can be used to compare actual outcomes with expectations in order to assess effectiveness.
- Indicators can be used to compare inputs relative to impacts and hence allow the assessment of the value (value added) of policy, legislation or initiatives.
The system of indicators and the programme cycle
Indicators are used at the beginning of the programme cycle to help to define territories eligible for assistance, to analyse the regional context, to diagnose economic and social problems to be addressed, and to assess the needs that the programme has to meet. At this stage, indicators such as the unemployment rate or disparities between infrastructures often play a decisive role.
The choice and validation of the intervention strategy constitute the second stage in the programming cycle. At this stage the programme designers should define the objectives precisely and quantify them. Indicators depend on quantification and are also very useful for clarifying objectives. Box Defining indicators can clarify objectives provides an example.
Once defined and adopted, the programme is implemented. It is monitored and evaluated on an on going basis. At this stage indicators are indispensable for circulating, in a simple and condensed form, information required by programme managers. Typically, indicators serve to monitor the pace at which budgets are spent, the extent to which the schedule is adhered to, the proportion of the eligible population reached, the rate of satisfaction of beneficiaries, the number of jobs created.
The programming cycle ends with an ex post evaluation, of which one of the main functions is to report on the programme results and on the extent to which aims have been achieved. The use of indicators is strongly recommended at this stage in so far as it allows the communication of simple information that is immediately understood by a wide public, e.g., cost per job created or rate of placement of jobless people assisted.
Indicators for integrated programmes
Most socio-economic programmes adopt integrated strategies, in other words, they try to solve all the problems affecting a given territory and they use all available instruments for intervening in that territory. This characteristic necessarily entails a multiplication of needs for indicators, which would lead to confusion if the programmes were not highly structured. Programmes financed by European Cohesion Policy are usually structured on three levels:
- the overall programme level to which the global objective is related, for example, economic development or employment. This level consists of a small number of priority axes (usually less than six) which break down the global objective into its main strategic dimensions;
- the measure level (from one to several dozen) existed up to 2006 and corresponded to the basic unit of programme management. Each measure had its own specific management apparatus. While measures are no longer part of the Cohesion Policy programme per se, managing authorities continue to operate "measures" for the purposes of managing their programmes;
- the project level (often a few to many hundred), which is the implementation unit of the programme, since each project is a point of articulation between the programme and its beneficiaries.
Organisational aspects: Involving users and suppliers of information
A system of indicators has more chance of functioning when the suppliers and users of the information have been involved in its creation. In contrast, a closed group of specialists will be tempted to construct an expensive, technically ideal system that may never be operate satisfactorily.
As far as the users are concerned, explicit support from the highest level of the authority managing the programme has to be assured. It is then advisable to create a group of future users of the system, and to give it the job of defining the indicators.
A team should then be appointed to support the group and provide a secretariat. Typically the team members belong to the authority managing the programme. They should have the required human and financial resources. The team must, in particular, ensure that the system of indicators clearly reflects the programme objectives and favours comparability. It is preferable for the same team that is responsible for creating indicators to subsequently be responsible for the implementation of the system.
The public may also be involved in designing the system of indicators. An example of involving beneficiaries in the choice of indicators from American experience (Benton Harbour region, see Box Involving beneficiaries in the choice of indicators ), started with a series of focus groups involving representatives of regional enterprises. The work of these groups made it possible to select indicators most likely to attract the public's attention and to be understood by it.
The main suppliers of information are the operators who implement the programme in the field. Their participation is likely to ensure that the system is pragmatic because they are familiar with the practical possibilities and limits of data collection.
It is also advisable to involve the operators in a preliminary test of the system of indicators. The recommended procedure starts with the selection of a few volunteer operators who will participate in the design of the system. These volunteers should represent all the components of the programme. They help to choose the indicators, to define them and to plan the information collection process. They express their needs in terms of information feedback (frequency and form of information fed back to them). The test comprises an initial quantification of all the indicators by voluntary operators. The normal duration of such a test is a year. Depending on the conclusions of the test, and after introducing the necessary modifications, the system is validated. The definitions and the data collection and restitution procedures are clearly established, and a manual is written.
Information relating to the context is drawn from statistics. It is therefore advisable to involve an expert with recent and complete knowledge of exploitable statistical data, in designing the system. Depending on the case, this expert will belong to a statistics institute or a university or, if possible, to the institution running the programme.