| |
| |
The evaluation should first identify the outcomes of the assistance and compare them with the objectives to be achieved. The structuring stage requires the analysis of the objectives, from which the intended effects are deduced.
After determining the evaluation scope, the evaluators should construct a faithfully reconstructed objectives diagram from the strategy and planning documents. A logically reconstructed objectives diagram and an effect diagram (intended effects) will be derived from the faithfully reconstructed objectives diagram.
STAGE 1: HOW IS THE EVALUATION SCOPE DELIMITED
Usually, the evaluation scope is strictly defined in the terms of reference which includes information about the timeframe and the financial tools to be assessed.
In any case (concerning the evaluation timeframe or the financial tools), the managing authority must provide the evaluation team with a precise definition of the scope, although the evaluation team must feel free to warn the latter about the consequences of a too restrictive scope.
Period of the evaluation
The period to be assessed fully or partially corresponds to the period covered by the strategy and/or planning papers.
However, during this timeframe, projects and programmes which are being implemented may have already been described and authorised in previous documentation.
What importance should be given to the objectives or the effects of such projects and programmes?
Two situations should be considered, depending on the nature of the projects and programmes implemented during the period under study (i.e. their objectives are in continuity with or differing from the objectives planned in previous projects and programmes).
Projects and programmes whose objectives differ
The evaluator should undertake a general analysis of the differing topics in the overall strategy and type of activities, and avoid the study of specific objectives (or effects) included in these projects and programmes.
Projects and programmes whose objectives are in continuity
In this case, the study of their specific objectives (or effects) should be included in the analysis of the objectives.
 To find out more:
Financial tools
Explicit terms of reference
Terms of reference can focus the evaluation on the activities carried out by a specific institution (a General Directorate of the European Commission) with a specific financial tool. The scope is strictly limited. The evaluation team should however examine other institutions, include them in the category "other donors", and seek the elements of coherence in their activities which could be missing from the objectives system or the effect system of the institution under evaluation.
The evaluation team should also examine the coordination between activities under assessment and others, and formulate its conclusion on the possible effects of insufficient coordination.
Inexplicit terms of reference
When terms of reference do not determine the evaluation scope, the analysis of the objectives or the effects should cover the range of activities, whatever the implementing institution and the financial tool implemented.
To find out more:
|
|
| |
STAGE 2: HOW TO IDENTIFY THE OBJECTIVES AND EFFECTS
Collect the documentation
As a first step, the documentation needed for the construction of the diagram is collected in the relevant Unit of DG RELEX or DG DEV. It comprises:
- Policies, strategies and programmes baseline documentation
- Complementary official documentation
The documentation varies with the type of evaluation (geographical, sector-based and thematic).
Record the references to the objectives in the baseline documentation
The task should begin with a careful analysis of the baseline documentation, to assemble a record of all references (quotations, sources) to the objective (understood as the target of the European Commission's assistance) and any explicit mention of the links between objectives.
The documentation seldom uses the term "objectives" and only introduces the term in expressions such as "with a view to", "in order to", "so as to", etc. Objectives can be called "priorities", "fields of intervention", or sometimes "focal sectors".
A precise table of such terminology must be established.
Distinguish strategic from background objectives
Les citations relevées dans les documents de référence sont traduites en objectifs. Chacun des objectifs est assorti d'un numéro d'identification renvoyant aux citations.
The quotations taken from the reference documentation are converted into objectives and each is given a unique identification number.
The documentation identifies the objectives which should be targeted by the assistance. The remaining objectives are treated as background items.
In an annex to the reports (how are the findings presented?), the evaluation team should present the list of all the objectives, marshalled into two categories: strategic and background objectives.
|
|
| |
STAGE 3: HOW TO CONSTRUCT A FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM
Classify the objectives by rows
Several situations can be encountered:
- The row of the objective is explicit. This is often the case for the objective of the first row, also called the overall or long-term objective.
- The cause-and-effect link between two objectives is explicit, which facilitates the understanding of the interconnections between them.
- No indication is given about the objective's row (per se or in relation to others). The evaluator should refer to additional documentation in order to find an indication of a cause-and-effect link; if this task fails, it should be deduced and/or the evaluator should ask the assistance of experts (human resources and working arrangement).
Each of the classifications by row should be explained by means of an interpretation expressed as an assumption.
A provisional classification could be carried out on the basis of the distinction between three levels of objectives:
- Overall objectives
- Short-term and medium-term intermediate objectives
- Operational objectives
The definition of the logical relationship is paramount in the classification. This task is a question of experience, because the decision of the evaluation team that two events are logically connected depends on the judgement of experts in the field under study, and managers responsible for the implementation of the strategy and policies.
Establish a temporary diagram
The first faithfully reconstructed objectives diagram can be drawn from this first classification. It usually reveals incoherencies that will have to be corrected in accordance with the evaluator's interpretation (these corrections must be indicated in the diagram).
Should the construction of the diagram start with the overall objective
or the operational objectives (objectives describing the attended results)?
Usage has shown that these two types of objectives are more easily identified than intermediate objectives, which are also more difficult to rank. Thus, it is recommended that the roots of the diagram and the extremities of the branches are developed concurrently.
In the diagram, each objective is presented as a box with a heading. The evaluator should also indicate the identification number which refers to the quotation list. Providing the boxes with colours corresponding to the level to which the objective belongs may also be helpful.
Each link should refer to one or more quotations (in accordance with the identification number).
The diagram should also mention the specific means (financial and non financial) used in the implementation of each operational objective.
Identify the authors of the objectives' wording
Before testing the temporary diagram, the evaluator should identify the main and secondary authors of the objectives' wording (writing of a chapter, sector-based contributions, participation in working meetings, etc.).
It may be useful to check the consequence of the wording on the definition of the operational objectives of the programme planning.
Test the temporary diagram
The authors of baseline documentation and additional documents should test the diagram in order to validate the classification of the objectives by rows and links. The aim is to check that the evaluator's interpretations accurately reflect the authors' intentions. If the authors are not available or, with a view to complement their contribution, the evaluator can seek the participation of other actors responsible for the drafting process (both from the writing and the discussion process).
Contacting the authors is usually possible when the documentation is recent and the authors are still in position or contactable in one of the services. This task is challenging when the documentation is old and its authors are not easily identifiable, nor reachable.
At the end of the interviews, the evaluator may also encounter difficulties evaluating what was actually done during the drafting process and the justifications of respondents for the decisions taken.
Establish the final version of the faithfully reconstructed objectives diagram
This version takes into account the opinions collected during the test of the temporary diagram.
It should be considered an accurate report of the initial intentions of the European Commission, taken from the official documentation.
|
|
| |
STAGE 4: HOW TO CONVERT THE FAITHFULLY RECONSTRUCTED OBJECTIVES DIAGRAM INTO A LOGICALLY RECONSTRUCTED OBJECTIVES DIAGRAM
Advantages and limitations of a faithfully reconstructed objectives diagram
Advantages of the faithfully reconstructed objectives diagram
Some faithfully reconstructed objectives diagrams display a strong internal coherence: apart from the overall objective or the overall expect impact, each objective 'x' leads to an higher (immediate or not) objective and is supported by one or more subordinate objectives. This is particularly the case when faithfully reconstructed objectives diagrams are extracted from a carefully established logical framework. Faithfully reconstructed objectives diagrams can be deemed as completely logical and stand as logically reconstructed objectives diagrams.
Logical links of a given objective 'x' (in objectives diagrams)
The establishment of a logically reconstructed objectives diagram is not required when the objectives system of the strategy and planning documents is thoroughly and consistently presented.
To find out more:
Limitations of the faithfully reconstructed objectives diagram
Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:
- No definition of the overall objective
- Weak relevance of the causal links between objectives
- The objectives of a given row do not result in any objectives or are not linked to subordinate objectives which would have provided information about the implementation process
- Several unclassified objectives
In these cases, the definition of the evaluation thematic scope and the evaluation questions is difficult. The team should therefore draft a logically reconstructed objectives diagram.
Preparation of the logically reconstructed objectives diagram
In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts. This task should be completely transparent (how are the findings presented?): whenever possible, the diagram should display the reclassification of objectives, the changes affecting links between objectives, and the introduction of missing intermediate objectives. Each of these rationalisation operations should be explained in a technical note.
|
|
| |
Stage 5: how to construct an effect diagram
Objectives diagrams and intended effect diagrams share the same rules of construction.
The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect.
|
|
|
|