Example: Use of expert panels to compare impacts, using a scoring system

An evaluation team had evidence of the equal opportunities impact of a range of measures. These are presented in the form of the synoptic table , in which each column concerns an impact cluster and contains the conclusions drawn on all related measures (see extract below).

Extract from synoptic presentation of conclusions relative to an impact

Measure

Description of the equal opportunities impact

1

Training intended for the long-term unemployed

The training had a positive impact in terms of orientating women towards secure jobs traditionally occupied by men

2

Aid for new business creation

 

45% of the new businesses were created by women, as compared to the national average which is 30%. The difference can be imputed to the programme.

.

Other measures

Etc.

At this stage of its work, the evaluation team noted that the impacts are not directly comparable and that it could not go much further, on its own, towards a synthetic conclusion. A way of formulating comparative judgements was to get the steering group or an expert panel to express an opinion.

The approach was to:

After the seminar, convert the classifications into scores ranging from 10 (maximum negative impact) to +10 (maximum positive impact), through 0 (neutral impact). Convert the synoptic table into a table of ratings (impact scoring matrix). The construction of scoring scales allows for comparisons to be made within the same column (e.g. one particular measure has a better rating than another). On the other hand, scores in different columns cannot be compared (e.g. a score of 5 for employment is not comparable to the same score for an environmental impact).