What is it about?
The quality of the evaluation report is rated along with nine criteria by means of a grid which involves five levels: excellent, very good, good, poor and unacceptable.
This document puts forward indicative descriptions for each of the five ratings for all nine quality criteria.
The purpose is to facilitate and harmonise the quality assessment.
Rating the various criteria
 Criterion 1: Meeting needs
The demands stated in the terms of reference have received an adequate answer. The evaluation questions have been answered satisfactorily.
The evaluation report includes a clear overview of how the stated objectives have been achieved and has clarified the intervention logic. The evaluation report goes beyond the demands stated in the terms of reference and addresses other topics of interest.
Certain questions about the terms of reference have been inadequately or only partially answered.
Too many questions about the terms of reference have been only partially answered or have not been addressed at all.
The questions addressed cover not only the requirements in the terms of reference but they also place the evaluation within a much more general framework which is linked to the bases of development, cooperation or foreign policies and to any other EC policy or partner country's policy.
Criterion 2: Relevant scope
The report deals with the whole intervention in its temporal, geographic and regulatory dimensions. The main intended and unintended effects have been identified.
In addition to the previous points, the evaluation took an interest in interferences with the other EC policies, other donors' interventions and the partner government(s)' policies. Unintended effects have been addressed.
One of the three dimensions of the intervention and/or one major effect has been inadequately or insufficiently addressed.
Several dimensions of the intervention and/or several major effects have been inadequately or insufficiently addressed.
In addition to the remarks for a "very good" rating, the report has systematically examined the unintended effects in detail.
Criterion 3: Defensible design
The evaluation method is clearly explained and has been actually applied throughout the process. The methodological choices have been appropriate enough so as to meet the requirements of the terms of reference.
The limitations inherent in the evaluation method have been clearly specified and the choices have been discussed while standing up against other options.
Upon reading the evaluation report, the methodological choices seem to have been made without being either explained or defended.
Either there is no evaluation method, or the methodological choices are not in line with the results sought after.
In addition to meeting the expectations for a "very good" rating, the evaluation team submits a critique of the method and methodological choices. The report points out to the risks that might have been incurred if other methodological options had been adopted.
Criterion 4: Reliable data
This criterion does not apply to the intrinsic validity of existing data but on the way in which the evaluation team has collected and used the data.
Both quantitative and qualitative sources have been identified. The evaluation team has tested and discussed the reliability of data. The data collection tools have been clearly explained and adjusted to the data sought after.
Data have been systematically cross-checked by relying upon sources or data collection tools that are independent of one another. Limitations pertaining to the reliability of data or to data collection tools are made explicit.
Both quantitative and qualitative data provided are not very reliable regarding the question asked. The data collection tools are questionable (for instance, insufficient samples or off-the-target case studies).
Certain data are manifestly distorted. The data collection tools have not been applied correctly or else they provide biased or useless information.
All biases deriving from the information provided are analysed and rectified by means of recognised techniques.
Criterion 5: Sound analysis
The quantitative and/or qualitative analysis is done rigorously, following the recognised and relevant steps depending on the type of analysed data. Cause-and-effect links between the intervention and its consequences are explained. Comparisons (for example: before / after, beneficiaries / non beneficiaries, with / without) are made explicit as well.
The analysis approaches are made explicit and their validity limitations are specified. Underlying cause-and-effect assumptions are explained. Validity limitations of comparisons made are pointed out.
Either one out of three elements (analysis approach, cause-and-effect relations, comparisons) is not well addressed or two out of such elements are dealt with inadequately.
2 out of 3 elements are addressed inadequately.
Every analysis bias (across the 3 elements) has been systematically reviewed and presented, including its consequence in terms of limiting the validity.
Criterion 6: Credible findings
The findings derived from the analysis seem both reliable and balanced, especially in view of the context in which the intervention is being assessed. Interpretations and extrapolations made are acceptable.
The findings acceptably reflect the reality described by the data and evidence recorded on the one hand, and the reality of the intervention as perceived by the actors and the beneficiaries on the other hand.
The limitations applying to interpretations and extrapolations are explained and discussed.
The effects of the intervention under evaluation are isolated from the external factors and contextual constraints.
Both internal validity (absence of analysis bias) and external validity (generalisability of findings) are satisfactory.
Analysis results seem imbalanced.
The context is not made explicit.
Neither extrapolations made nor generalisations of analysis are relevant.
Credibility of analyses seems very poor.
Some assertions in the text cannot be sustained.
Neither extrapolations made nor generalisations of analysis are relevant.
Imbalances between the internal and external validity of findings are systematically analysed and the consequences this has on the evaluation is made explicit.
Contextual factors have been identified and their influence has been demonstrated.
The biases involved with the choice of interpretative assumptions and in the extrapolations are analysed and their consequences are made explicit.
Criterion 7: Valid conclusions
This criterion does not assess the conclusion's intrinsic substance but the way in which the conclusions have been reached.
Conclusions derive from analysis.
Conclusions are grounded on both facts and analysis that are easily identifiable throughout the report.
The limitations to conclusions' validity are pointed out as well as the context in which the analysis was done.
Conclusions are debated upon in connection with the context in which the analysis was done.
The limitations to conclusions' validity are made explicit and well grounded.
Conclusions stem from a hasty generalisation of some of the analyses.
The limitations to conclusions' validity are not pointed out.
Conclusions are not backed up by relevant and thorough analysis.
They are based on unproven data.
Conclusions are partial because they reflect the evaluator's preconceived ideas rather than the analysis of the facts.
Conclusions are organised along hierarchical lines and reached in relation with the global nature of the intervention under evaluation. They take into account the intervention's connection with the context in which it takes place, considering other programmes or connected public policies in particular.
Criterion 8: Useful recommendations
This criterion does not judge the recommendations' intrinsic substance but the way in which they are articulated and whether they really derive from the conclusions.
The recommendations follow logically from the conclusions.
They are impartial.
In addition to the previous points, the recommendations are prioritised and presented in the form of options for possible actions.
The recommendations are not very clear or they are mere evidence without any added value, their operability is arguable. The connection with the conclusions is not clear.
The recommendations are disconnected from the conclusions.
They are biased because they mostly reflect certain players' or beneficiaries' viewpoints or the evaluation team's preconceived ideas.
In addition to meeting the requirements for a "very good" rating, the recommendations are tested and the validity limitations are pointed out.
Criterion 9: Clear report
The report is easy to read and its structure is logical.
The summary is brief and reflects the report.
Specific concepts and technical explanations are presented in an annex with clear references throughout the body of the text.
The body of the report is short, concise and easy to read.
Its structure is easy to memorise.
The summary is clear and presents the main conclusions and recommendations in a balanced and unbiased way.
The report is hard to read and/or its structure is complex.
Crossed references are hard to understand or make reading difficult.
The summary is too long or does not reflect the body of the report.
Absence of summary.
Illegible report and/or disorganised structure.
Lack of conclusion (and recommendations) chapter.
The report can be read "like a novel" and its structure has an unquestionable logic.
The summary is operational in itself.
Overall assessment
The general quality of the report results from the ratings assigned to each one of the 9 criteria. If there are over 4 "unacceptable" ratings, the report must be considered as unacceptable altogether.
|