The Terms of Reference (ToR) is the document that serves as the basis of a contractual relationship between the commissioner of an evaluation and the team responsible for carrying out the work. Devising the Terms of Reference is a vital step when an evaluation has to be performed by outside consultants. This work is equally important when part of the evaluation is performed in-house. The ToR may concern either the evaluation operation as a whole (when it is entrusted to a single team) or a part of the research work programmed in the evaluation project (in-depth analysis of an evaluative question).
The ToR should be brief (typically 5-10 pages) supplemented if necessary by administrative annexes. A model content for a ToR is listed in Box Standard layout of the terms of reference and is then elaborated.
1. Regulatory framework
The legal, contractual and institutional framework for a programme needs to be stated. This would, for example, include regulations of national authorities or the European Union. The ToR should specify who initiated the evaluation project and, where relevant, who was involved in formulating the evaluation brief. Underlying motives and intentions should also be stated. For example: Is the intention a change of policy direction? If so why? Is the intention to modify the implementation procedures? Is the intention to reallocate funds?
2. Scope of the evaluation
We have already discussed the importance of defining the scope of the evaluation. The ToR should clarify the project/programme/policy/theme to be evaluated, the period under consideration, the point of the policy/programme cycle at which the evaluation is set and the geographical area of reference for the study.
3. Main users and stakeholders of the study
We have already noted the importance of evaluation use and users being identified at the earliest stages of planning. It is therefore important to include statements about how the evaluation results will be used in the ToR. If there is to be user-involvement, for example in a Steering Committee, this should also be stated.
4. Evaluative and research questions
We have already noted that different evaluation and research questions can be addressed (descriptive, causal, critical, prescriptive or normative) and different criteria can be employed in formulating evaluation judgements.
It is important to state the evaluation questions but it is also important to limit the number of questions that the evaluation asks. To focus the evaluation on a narrow list of questions that are relevant for the commissioner ensures better quality control.
5. Available knowledge
The ToR should contain a review of the current state of knowledge on the programme and its effects. This will include extracts or references from programming documents, lists of previous analyses and evaluations with relevant extracts, a description of the monitoring system in place, quantified indicators and the various reports and databases available from the services managing the intervention. This inventory is relevant for the evaluation teams to adjust their proposed methods.
6. Main methods or techniques to be used
Each evaluation will have its own particular methods relevant to its scope and content. It is not generally good practice to fully specify methods and approaches but to leave scope for those who propose an evaluation to indicate how they would wish to proceed. The priority is for those who commission the evaluation to specify what they consider to be their requirements in terms of outputs, e.g., answers, to key questions. They may or may not specify particular methods consistent with their intentions, for example, the need for a survey of beneficiaries.
The choice is generally made to maintain sufficient flexibility to allow those answering the ToR to differentiate themselves in terms of the relevance and clarity of their methodological proposals. This is especially important in the selection phase because assessing the methodological qualities of the proposals is a crucial step in selecting the right evaluator.
When possible from an administrative point of view, the best way is to determine a budget (see below) and to describe only the main lines of the method in the ToR and then to select the team that proposes the most promising method. Those selecting the team will then need to have the ability to judge the methodological quality of a tender.
7. Schedule
The evaluation schedule should be established by taking into account various constraints, especially those concerning the decision-making schedule and possible use. It is also necessary to integrate the main deadlines, generated by the procedures of calls for tenders and by the phases of primary data collection. It is advisable to define in the ToR the overall length of the exercise and to leave the first period usually between 10-20% of the duration of the overall evaluation to the detailed planning of the work. This phase should be concluded by an Inception Report in which the design of the activities as well as the detailed schedule will be spelt out. Equally advisable is to imagine the different outputs of the exercise, and among them, specific reference should be made to the submission of the draft final report allowing enough time for the suggestion of changes and amendments before the end of the study.
8. Indicative budget
It is good practice to suggest an indicative budget and then to leave those competing for an evaluation by open tender to suggest what they would be able to provide for the budget available. This allows value-for-money assessments to be made. It also provides the commissioners of the evaluation with greater control over expenditure. An alternative to this top-down approach is to leave it to proposers to come up with their own estimates based on the tasks they see as necessary. In general, those tendering for an evaluation should be encouraged to breakdown their costs into basic categories, including for example, data collection, report preparation, fieldwork, etc..
9. Required qualifications of the team
The ToR should specify a number of requirements of the evaluation team. This should include: methodological skills required; prior experience of similar evaluation work; knowledge of the regional and institutional context; professional background and disciplinary expertise; and the ability to manage and deliver an evaluation in a timely fashion.
Independence of the evaluation team
We have already noted the importance of independence in terms of credibility. This can be heightened by entrusting the evaluation to an external team. It is also useful to:
- Put in place management arrangements that will support the independence of those evaluators chosen; and
- Request confirmation that there are no conflicts of interest within the potential team.
These requirements should be stated in the ToR.
At the same time, how evaluators will be able to have access to key personnel within the programme and its management and to information that they will require for their work, should also be described. (Issues of evaluator independence are discussed in greater detail below).
Profile of the evaluation team
In the field of European Cohesion Policy, a large number of organisations are present in the evaluation market, including local, national or international consultancy firms. The commercial sector accounts for most of the market, although university research centres also make a significant contribution.
Opting for a consultancy firm or a university department can have implications in terms of the approach and therefore the results of the evaluation. Academics have the advantage of being perceived as independent and highly credible owing to their own institutional and professional requirements. On the other hand, private firms are often more readily available as far as time is concerned and are more concerned with meeting the commissioner's expectations.
The overall choice should depend less on the institutional origins of the evaluation team and more on the required competencies, i.e., their expertise, skill and prior knowledge. Those proposing an evaluation should also be asked to indicate how the different expertise, skills and experience within the team will be integrated and encouraged to work together.
10. Structure of the Proposal
In order to facilitate the adjudication and to provide guidance to the potential applicants, the ToR should specify how the proposal should be structured, possibly indicating the maximum number of pages for each section of the document.
11. Submission rules and adjudication criteria
The tender should specify: the deadline, the modes of transmission (post, fax, e-mail), how long their offer will remain valid, etc.. It should also indicate the criteria according to which the proposals will be judged. The ToRs should state for example in percentage points the relative importance that will be given to:
- the quality of the methodological approach;
- the qualifications and previous experience of the team;
- the price.
It is of course important that these criteria are applied systematically once proposals are received.