What does this mean?
A judgement criterion specifies an aspect of the evaluated intervention that will allow its merits or success to be assessed. Whilst "judgement criterion" is the appropriate word, an acceptable alternative is "reasoned assessment criterion". The criterion is used to answer an evaluation question. One or more judgement criteria are derived from each question.
What is the purpose?
- To avoid subjectivity and to formulate judgements on accepted terms.
- To improve the transparency of the evaluation by making the judgment explicit.
- To structure the answers to the questions asked, since the judgement criteria will determine the indicators and, more generally, the nature of the data collected and the type of analysis.
How can a judgement criterion be clarified on the basis of a question?
All the evaluation questions relate to one or more judgement criteria, unless they are designed only to further knowledge or understanding about the intervention or its effects.
The following is an example of a question:
To what extent has EC aid improved the capacity of the primary educational system to enrol pupils from underprivileged groups without discrimination?
Like most evaluative questions, it has two parts:
- What is being judged: "EC aid".
- The way of judging: Has it "… improved the capacity of the primary educational system to enrol pupils from underprivileged groups without discrimination".
- " This question belongs to the effectiveness family
The judgement criteria develop and specify the second part of the question, for example:
Capacity of the primary school system to enrol pupils from ethnic minority X satisfactorily.
Capacity of the primary school system to enrol pupils from disadvantaged urban areas satisfactorily.
The judgement criteria derive from the question, for instance in the case of the first criterion:
- It concerns the way of judging and not what is judged. This is why the beginning of the question concerning "EC aid" has been removed.
- It specifies the type of success to be evaluated, that is, an improvement in the "capacity of the primary school system to enrol pupils from underprivileged groups without discrimination", and specifically "pupils from ethnic minority X".
- It emphasises the judgement and not the causality analysis. That is why the terms "To what extent … has it improved" have been removed.
To be used in practice, each judgement criterion has to be accompanied by a target level and one or more indicator(s).
- Always define the judgement criterion before selecting an existing indicator or designing a new indicator. This is essential in order to clarify the concepts. By focusing too soon on indicators, one is likely to get trapped into existing information, even if it is inadequate for answering the question asked.
- Have the definition of the judgement criteria discussed by the reference group so that the diversity of points of view relevant to the intervention can be taken into account.
- There may be disagreement on the judgement criteria, for instance the same effect may have a dimension that is judged positively by certain members of the reference group and another dimension judged negatively by others. In this case there are two options: (1) choose only one judgement criterion but be careful to avoid biasing the evaluation; or (2) choose several criteria, although this will increase and complicate the data collection and analysis work.
- To optimise the collection and analysis of data, it is best to define a limited number of judgement criteria for each question. This recommendation also takes into account users' capacity to absorb information.
- Where relevant, explain any gaps between the criteria used to formulate the judgement at the end of the evaluation process and those identified in the first phase (desk) of the evaluation.
Be careful not to confuse concepts
On this site, the word criterion is used for three different concepts:
- Judgement criteria presented on this page
- The evaluation criteria: relevance, effectiveness, efficiency, sustainability, impact, Community value added, and coherence/complementarity.
- The quality assessment criteria of evaluation reports.
According to the EC, the value added of an evaluation is the formulation of value judgements on the basis of evidence and explicit judgement criteria. When dealing with organisations which are not familiar with evaluation, it may be wise not to use the word "judgement", which may induce resistance. An acceptable alternative is "assessment", or preferably "reasoned assessment".