The process of selecting evaluators needs to be transparent. Careful drafting of the Terms of Reference (ToR) is the best possible way to attain this goal, together with the use of a formal selection committee. This should include representatives of the people in charge of the evaluation and, when possible, representatives of the potential and actual users of the evaluation. Sometimes, it is useful to include also an independent expert.
Members of the selection committee should reach their own judgements on tenders against the criteria given in the ToR. The judgments should then be combined. The criteria normally include: the quality of the proposed method of approach; the quality and experience of the evaluation team; and the price.
Judging the quality of the proposed method
The suitability of the proposed approach and methods to answer the questions asked in the ToR should be central to the selection of the preferred tender. The selection committee can ensure this by checking each of the tenders against the points in Box Assessing the quality of the method of approach in a proposal .
Above all it must be remembered that judgements on the quality of the method proposed are qualitative and not quantitative. These are judgements, which need to be made by those with experience. Many of the items for which judgement has to be made are also qualitative. For example, the size of the sample for a survey or a number of case studies may be less important than the quality of the process through which the sample is extracted or the case studies identified.
Judging the qualifications and previous experience of the team
The qualifications and previous experience of the team are always important and especially so if the methods proposed are experimental or do not completely fulfil the specifications in the ToR. It could be argued that if the evaluation is standard or the ToR are very precise about what must be done, the quality of the personnel and the price are the only things that actually matter. However, although this may be so, the reverse is not true. When the tender process asks candidates to propose the methodology that they consider suitable for the task the danger is that too much attention is paid to the originality of the approach and not enough to the ability of the candidates to actually deliver what they have promised.
The capabilities of the team must be matched with the methodology proposed in order to avoid problems occurring whilst the evaluation is being implemented. However there is a danger that this will discriminate against new entrants and therefore make the creation and maintenance of competitive markets more difficult.
A useful way to judge a proposed team is to ask to see previous examples of their work. This can be further supported by asking for references from previous evaluation customers, i.e., named persons who can be consulted.
Finally, it is always good to pay attention not only to the presence in the team of highly qualified personnel, but also to the time that they are prepared to devote to the task. As evaluations are time consuming, the most qualified people will not undertake all the fieldwork themselves. Time allocated by those with experience needs to be sufficient to provide supervision for those working in the field. Evidence of the proposed team having worked together successfully is also relevant.
Assessing the price
Assessing the proposed price for the services is an important aspect of the selection process, but should not be overestimated. As a rule of thumb for any evaluations that are not entirely routine, the financial criterion should not exceed 20-25% of the overall assessment.
A second point worth noting is that not only should the total price should be taken into consideration but also the unit cost of the workday for the different categories of personnel employed. For instance, if 80% of the total price is absorbed by junior personnel at, say, a low day rate, then the merits of this can be compared with a situation where 50% of the work is carried out by better qualified/experienced researchers working at twice this daily rate.
In some countries, in order to avoid a race to the bottom the price is judged not in absolute terms but in relation to the average proposed by the teams bidding for the work. In this case, if an offer is exceptionally low, the tenderer could be asked to justify the reasons why such an offer is possible.