Why are these tools used in evaluation?
These diagrams are usually used as organisation tools. They provide a framework for the information collection and the undertaking of in-depth interviews during the assessment of the programme or strategy. They relate direct outcomes and field effects with the expected objectives and effects of the programme. The construction of the diagrams should therefore constitute one of the first tasks of the evaluation.
The objectives diagram illustrates the objectives classification, from the global objective to detailed operational objectives.
The effect diagram displays the classification of the results, outcomes and effects of what is intended from the implementation of the objectives system.
|Figure 1: Rationale of the objectives diagram and the effect diagram
What are the possible uses of these diagrams?
In the evaluation context, diagrams are used to reconstruct the intervention logic of the European Commission's support to the country. This reconstructed logic will be shaped into one or more logical diagrams of effect. Prior to the preparation of effect diagram(s), the team will have to prioritise the stated cooperation objectives and translate these into intended effects. These intended effects will form the "boxes" of the diagram(s). Possible "holes" in the intervention logic will be indicated and filled on the basis of assumptions to be validated by the reference group. The effect diagram(s) will help to identify the main evaluation questions.
|Figure 2: example of an objectives diagram
The faithfully reconstructed objectives diagram displays the objectives system and provides the evaluator with a first approach to the strategy and policies inner quality. Indeed, an unclear, incomplete or incoherent diagram means a lack of relevance in the resulting planning or a lack of faithfulness to the initial objectives system.
During the implementation of the strategy or programme, external events, such as the evolution of the world commodities market price, elections, political changes, or the conclusion of international agreements can influence the achievement of the objectives and expected outcomes. Comments dealing with the discrepancies between the expected and the observed outcomes should take these events into account.
A would-be coherence
Diagrams establish logical links between objectives or effects. Each objective or effect is presented as logically dependent on a higher row objective or effect. The outputs of the activities implemented or scheduled by the programme appear as a contributions to the overall objective/effect and support the coherence of the objectives and effect system.
A strategy or a programme seldom address the full scope of the overall objective, which is limited to choices for each row of objectives or effects. Unless the evaluation can find an explicit explanation of the choices made in strategic documentation, it must provide an answer to series of questions:
- Do these choices represent the only possibilities?
- Are the objectives the most relevant ones for the row 'x'?
- What external factors have led to this selection?
- What reasons have led to the elimination of certain objectives?
How are they constructed?
After determining the evaluation scope, the evaluators should construct a diagram illustrating the objectives presented in the strategy and planning documents. The objectives diagram is drawn from this diagram.
When the evaluation scope covers one or more strategy papers (geographical) or strategic policies (sector-based, thematic), it is recommended that one diagram per document is created (unless there is a logical continuity in the strategy or the policy).
When a logical framework has supported the drafting of a programme, it clearly states the various levels of objectives targeted by the programme. The logical framework is consequently a reference point for the evaluation, as a consequence of the presentation of the objectives diagram's basic constituents. Although the establishment of result-based logical frameworks has not yet been generalised, the effect diagram can usually be deduced from the logical framework of the intervention's objectives.
The objectives diagram and effect diagram can also be used in evaluations for projects and programmes whose rationale is not explicit in the logical framework. In practice, the objectives and intended effects of complex policies and strategies often lack explicit presentation and logical structure, whereas the justification for an evaluation is to be able to answer the following question:
- Have the objectives or intended effects of the policy or the strategy under assessment been achieved?
What are the preparation stages for the construction of the diagrams?
Stage 1: delimit the evaluation scope
The terms of reference include information about the timeframe and the financial tools to be assessed. Note that projects and programmes originating from previous documentation can also be implemented during this timeframe.
Stage 2: identify the objectives and effects
Collect the documentation required for the establishment of the diagrams. It should comprise:
- Policies, strategies and programmes baseline documentation
- Complementary official documentation
An overall analysis of the break points in the global strategy should be undertaken, without including the objectives of the projects and programmes in progress and scheduled for a different timeframe from the one assessed.
Establish a list of the objectives recorded in this documentation.
Stage 3: construct a faithfully reconstructed objectives diagram
A provisional classification could be carried out on the basis of the distinction between three levels of objectives:
- Higher-order objectives (goals)
- Short-term and medium-term intermediate objectives
- Operational objectives
La définition des relations logiques occupe une place centrale dans le classement. Elle est avant tout affaire d'expérience.
Stage 4: convert the faithfully reconstructed objectives diagram into a logically reconstructed objectives diagram
Some faithfully reconstructed objectives diagrams may reveal logical defects in strategy or political papers, such as:
- Strategy or political papers involving a range of objectives without sufficient details to classify them
- No definition of the higher-order objective (goals)
- Weak relevance of the causal links between objectives
- The objectives of a given row do not result in any objectives
In order to prepare a comprehensive and coherent objectives diagram, the evaluation team will need all available documentation, its own expertise and, if required, that of experts.
Each of these rationalisation operations should be explained in a technical note.
Stage 5: construct the effect diagram
Objectives diagrams and intended effect diagrams share the same rules of construction. The effect diagram is constructed from the conversion of each of the objectives into the corresponding intended effect:
- The higher-order objective corresponds to the higher-order effect
- Intermediate objectives correspond to intermediate effects or outcomes
- Operational objectives correspond to results
Most of the graphics can be addressed with software such as MS PowerPoint.
How are the findings presented?
Objectives diagrams are established during the organisation stage, where reports and notes should be provided. At this stage, the diagram's construction must be precisely described.
For the faithfully reconstructed objectives diagram, the sources of the objectives/effects (quotations, references to the original documentation) must be provided. References to documentation, interviews and expertise must support the objectives' location in the diagram, and the assumptions developed during the construction of the diagram must be explained.
|Figure 3 : example of the EC strategy in Tanzania
The process through which the logically reconstructed objectives diagram has been extracted from the faithfully reconstructed objectives diagram must also be clearly explained.
The evaluation team will need to present its work (methodology and findings) to different types of people (the evaluation reference group, participants in debriefing sessions). The objectives diagram and/or the effect diagram are very efficient tools for this purpose, providing that they are readable without being over-simplistic.
To do so, a main diagram, and several sub-diagrams developing fundamental sections of the main diagram, should be presented, each of them not exceeding 20 items (boxes).
What are the preconditions for its use?
|Figure 4: the preconditions for its use|
The time span
Collection of the documentation and identification of the objectives: 4 to 6 working days
Analysis and construction of the diagrams: 5 to 10 working days
Test of the diagrams: 1 to 4 working days
Knowledge of the European Commission's development procedures for strategies and programmes
Knowledge of strategies and programme documentation
Knowledge of computer tools
Logical process of thinking
Experience in the fields covered by the strategies and programmes
Specific knowledge of the country
A budget around €5,000 should be fixed for the problem diagram, objective diagram and decision diagram.
What are the advantages and limitations of the tool?
|Figure 5 : The advantages and limitations of the tool|
Logical classification of the objectives and the effects: the diagram explicitly illustrates the objectives/effects and their various rows, from the global objective (more or less long term), down to the range of activities that have already been implemented or are to be undertaken.
It reveals the logical links between objectives/effects, or the lack of a linkage.
Definition of the evaluation questions. Through the diagram, the evaluator examines a range of questions whose aim is to help answer the crucial question: To what extent have the stated objectives and the intended effects been achieved?
Presentation of the strategy: when the objectives diagram is well structured and clearly presented, it is a valuable educational tool which facilitates the understanding of the strategy.
Replacing the objectives tree for the objectives diagram in the illustration of the objectives system avoids most of the construction constraints and their inherent limitations (with one exception: the representation of retroactive links).
A simplified representation of reality: as a chart, the diagram is a simplified representation of reality, and its educational value depends on the selection of a sensible degree of simplification.
In order to avoid an over-simplification of the facts, the evaluator can develop sub-diagrams focusing on specific parts of the main diagram.
Check-list for evaluators
- Has the preliminary analysis of the strategies under evaluation been undertaken?
- Has the preliminary analysis of the institutions participating in the preparation and implementation of the strategy and/or the programmes been undertaken?
- Has the list of the relevant documents been established?
- Has the list been submitted to the group in charge of the monitoring of the evaluation?
- Has the dating of the documents been confirmed by their authors or contributors?
- Has a cross-reading of the documentation been conducted?
- Have the missing elements been sought (?) during the test?
- Are hypotheses and uncertainties about the objectives' links clearly stated?
- Did their authors and/or contributors confirm this classification during the test?
- Was there a triangulation of the perspectives?
- Have specialists been consulted by means of written exchanges, if necessary?
Check-list for managers
- Are documentary references and quotations provided in the report?
- Is the list of the documents consulted to establish the diagram complete?
- Do the diagrams appear coherent?
- Are the relations between objectives and effects explicit?
- Are hypotheses and uncertainties about the objectives' links clearly stated?
- Have the external effects been identified?
- Are the constraints and the risks exposed?