The use of indicators will be far greater if their quality is constantly improved. Evaluation has an important role to play in assessing the quality of systems of indicators and recommending ways to enhance them. Although there is no standard method for this quality control, an approach is proposed based on the following criteria, which are divided into two groups: quality criteria applicable to each indicator and quality criteria applicable to the entire system.
Quality criteria applicable to each indicator
The first quality criterion for an indicator is the capacity for it to be quantified at regular intervals. Sometimes one or more indicators featured in the programming documents have never been quantified and therefore cannot inform the evaluation or the monitoring of the programme. The availability of data to allow quantification is the primary factor to be considered. Monitoring indicators (inputs, outputs and some results) should be quantified at each monitoring meeting, that is to say, every six to twelve months. Evaluation indicators (some results and impacts as well as context indicators) are quantified less frequently, typically annually or every three to six years.
Once an indicator has been quantified, it may take several months or even years before the information can really be used for monitoring and evaluation. This is particularly true for certain context indicators drawn from statistical publications. The freshness of information is an important quality criterion. Sometimes statistics are published two years or more after the collection of the data.
When evaluating programme effects, the indicators chosen must be such that the programme is capable of bringing about a change in the indicator value. The capacity for interventions to impact on an indicator is known as sensitivity. Take the example of an intervention supporting exports; the turnover of businesses is not a sufficiently sensitive indicator. A better indicator would be the turnover relating only to new customers contacted with the support of the programme.
The results produced by applying the indicators need to be reliable and credible. Reliability tends to apply to facts and figures and can be defined as the fact that the same measurement, taken by two different people under identical conditions, will produce the same value for the indicator. In cases where indicators are quantified on the basis of questions put by one person to another, reliability can no longer be defined so mechanically, although the tests need to be credible. Credibility tends to depend on the soundness of the method, although the independence and reputation of the evaluation team may also be important.
The usefulness of an indicator depends largely on whether it allows for internal comparisons between different measures of the programme or inter-regional external comparisons. The comparability of the indicator is therefore a quality criterion.
A further quality criterion of an indicator is normativity. Indicators should relate to outcomes that can be judged to be satisfactory or not. Indicators should avoid ambiguity. Any indicator value must therefore be compared to a norm, for example: objective to be met; norm to be surpassed; or European average to be attained.
A good indicator must be understood by everyone who has to use it. In the minds of both decision-makers and the public, the meaning of the indicator must be the same as for the programme managers and the project promoters providing the source data. It must accurately reflect the concept to be measured. This is sometimes referred to as validity.
Quality criteria applicable to the entire indicator system
The following criteria are proposed to assess indicator systems:
- The indicators selected should cover a sufficiently large proportion of the programme measures. This coverage should be equal to or greater than three-quarters of the planned expenditure.
- The system should consist of a good balance between indicators in the different categories.
- The system of indicators should be simple. The selectivity criterion requires that the programme managers' capacity to absorb information be respected. The information must therefore be limited to a maximum of a few dozen indicators.
- The relevance of the system implies that the indicators are developed primarily for those measures or themes that have significant implications in terms of decision-making. For example, measures with a very high budget; innovative measures; themes considered to be strategic.
Very often the setting up of indicators will not start from scratch and wherever possible systems and indicators should be consistent with those already operating.