What is this?
Establishing quality check-points at the main phases of the process, defining the responsibilities of quality assurance, and establishing the rules of the game in case of quality defects.
Quality assurance is organised on a step-by-step basis: successive aspects of quality are secured at each step, thus creating a sound base for the subsequent step.
What is the purpose?
- To enhance the conclusions of the evaluation in users' eyes by showing that this is impartial and rigorous work that meets professional standards.
- To gradually construct quality and thus avoid discovering a quality defect in the final report stage.
- To clarify relations between the evaluation manager, the reference group and the external evaluation team.
Rules of the game
The rules of quality assurance are specified in the terms of reference and pointed out when the external evaluation team is engaged.
These rules concern:
- Approval of documents
- Quality criteria
- Dissemination of the quality assessment of the final report
The quality assurance process benefits from the contribution of all actors whilst limiting the potential conflicts that might arise between them, e.g.
- The actors who hold responsibilities in the evaluated intervention (the evaluees) are in a good position for assessing the relevance of data collected and the fairness of interpretations. Their assessment may, however, be distorted by a confirmation bias. They should therefore be given only an advisory role, e.g. through their participation in the reference group.
- The evaluation team leader has a major role in assuring quality, especially in designing an adequate method, and in securing the accuracy and fairness of the report. He/she may, however, overweight the views of powerful stakeholders, especially those who are likely to commission other evaluations in the future. The evaluation manager's quality assessment helps to prevent such a risk.
In the inception report stage, quality assessment is used to ensure that the evaluation team has clearly understood the scope and aims of the evaluation, as well as the logic of the evaluated intervention and the questions to answer.
First phase report (desk)
In the first phase report stage (desk), quality assessment is used to check the quality of the documentary analysis as well as the appropriateness of the proposed method for the field phase, including the indicators, data sources and analytical approach.
In the final report stage, quality assessment checks whether the evaluation provides valid and impartial answers to the questions asked, and whether the form of the report is suited to the targeted users' needs.
At this stage the quality assessment concerns first the draft version of the report and then the final version.
It is expected that all key players involved in the quality assurance process have a sufficient evaluation capacity, except reference group members. If a key player is not qualified enough, he/she ensures that appropriate support is provided.
Evaluation team leader
The evaluation team leader is primarily concerned with preventing major risks that threaten quality, e.g. overlooking a major question, not consulting an important stakeholder, elaborating upon unreliable data, judging in a partial way.
He/she also ensures that each report is submitted to a detailed quality check before it is released to the evaluation manager.
The company or consortium in charge of the evaluation contract appoints one or more quality assessors who carefully check each report for quality before it is released to the evaluation manager. The quality assessor should be experienced in evaluation and should not belong to the evaluation team.
The evaluation manager has the ultimate responsibility for assessing whether or not the methodological quality of a report is sufficient for allowing the next step to be taken.
In this process he/she takes advice from the reference group members, and relies upon technical support as far as necessary.
Reference group members receive all draft reports for comments. They may comment on the factual bases substantiating the evaluation, which is a contribution to quality assurance. When their comments apply to the content of conclusions, they are passed on to the evaluation team which freely decides on whether or not to take them into account.