WHAT CRITERIA SHOULD BE USED
TO APPOINT THE PANEL?
Core criteria of the expert panel's composition
The pre-requisite for the expert's selection is his/her professional experience. He/She should have specialised in the field under evaluation, and be recognised and respected by his/her peers. The credibility of his/her conclusions is highly dependent on these elements.
Experts must be independent of the programme under evaluation, because they should not be judge and judged. Thus, experts having a direct conflict of interest (such as experts participating in the programme or belonging to a body which benefits from it) should not be appointed.
However, securing sufficient independence is difficult in fields where there are few acknowledged experts. In this case, the panel, and not the experts, must be independent, and differing points of view must be dealt with in an even-handed way.
In addition, the experts' appointment to the panel is personal, which means that they do not represent their home institution and cannot be substituted on the panel.
Ability to work in a group
The ability to work in a group, listen to other experts and be open-minded is an essential criterion. Otherwise, working conditions may quickly turn out to be unmanageable, which would impede the panel process.
In addition to these core criteria and specific requirements relating to the expert's profile, some commissioning agencies may have their own views on the panel's composition (such as the expert's nationality, a balanced representation of differing points of view, the participation of specific categories of panellists such as beneficiaries, consumers).
Ideally, experts should also speak the language used in the area subjected to evaluation.
HOW ARE EXPERTS SELECTED?
Selection process for the panel members
Procedures for the recruitment of experts
Varying with the importance of the task and the complexity of the themes, the recruitment of panel members can be relatively straightforward and speedy, or require a time-consuming selection process.
In a straightforward selection, the evaluation managers have access to a list of acknowledged experts in specific fields, and limit their selection process to ensuring the expert's independence regarding the programme under evaluation.
Gradual selections have become a common procedure. Preferred profiles of experts are developed with respect to the topics under scrutiny in the evaluation. These profiles are critical to satisfactory recruitment to meet the evaluation's needs.
If the panel's intervention only focuses on technical issues, the expert's lack of independence within the fields under consideration will usually have minor impact on the evaluation, because the evaluator is responsible for the main tasks.
However, if the panel has to carry out a significant part of the evaluation, it should include an expert experienced in evaluation processes and a socio-economist, in addition to experts specialised in the topics under review. Depending on the nature of the mission assigned to the experts, additional fields of competence may need to be proposed.
Development of the panel profile
The Royal Society of Canada details elements to be taken into account in developing the panel profile, as follows:
- Project scope: will the study be limited to technical problems, or will it address broad issues of public policy?
- Degree of controversy: do the problems to be addressed have alternative resolutions which are controversial? Do they affect parties who have strong emotional, political or financial stakes in the outcomes?
- Technical support: will the panel's conclusions and recommendations be based on data analysis or on the panel's expert judgement?
- Uncertainties: will the panel's conclusions discuss the uncertainties?
- Number of required disciplines: do the issues to be evaluated involve a single discipline or are they interdisciplinary?
The selection process
Once these profiles have been developed, the institution managing the evaluation should establish a "long list" of experts and remove experts with possible conflicts of interest. Thereafter, the institution contacts the selected experts.
Depending on the number of experts required (5 to 10 in most cases), the institution should pre-select a larger number of potential experts (2 to 3 times more than the final list), in order to ensure the availability of experts. Within this "long list", some funding institutions may determine nominees and alternates. This procedure is also recommended for the nomination of the chairman and key experts.
Once the selection process is over and the experts are recruited, the commissioning agency (or the experts) proceeds with the appointment of the chairman. As the role of the chairman is essential for the efficient working environment of the panel, the importance of this appointment is paramount. He/She will be responsible for setting the tone and rhythm of the panel's work.
Secretarial work should be the responsibility of a technical writer, whose ongoing availability should compensate the pressurised timetables of the experts. The tasks of the technical writer include the production of working reports, the incorporation of suggestions, and, if required, the monitoring of external studies.
To find out more:
WHAT ARE THE PRE-CONDITIONS FOR THE EXPERT PANEL'S WORK?
A precise definition of the panel's field of work
Evaluators must take time to explain to the experts the context of their work and provide them with information about the programme, the procedures, and evaluation methodologies. In doing so, evaluators help the experts formulate their conclusions with a full knowledge of the environment of area assessed.
Usually, the content of the panel's working sessions is confidential.
To find out more about the practical aspects
of the panel's tasks in the research field:
- Bibliography: 'Strategic management of the research and technology fields' [FR]
Availability of documentation for the experts
Terms of reference
Terms of reference are often provided to ease the panel's working arrangements. They should specify:
- The nature of the evaluation
- The function of the panel within the evaluation and how this work is to be organised
- The questions submitted to the experts or the nature of their investigations (such as points of view about the relevance, the coherence, and the efficiency of the programme)
- The available data (for example, relating to the implementation of the programme) and the means at the disposal of the panel (including the possibility to go on-site)
- The content and timetable for the reports
Reference interview guides
In evaluations where calls for expert panels are systematic, developing procedures for the management of the panel process, as well as interview guides may ease the panel's work (for example, during interviews with officials, field study, interviews with beneficiaries). These procedures can also benefit repetitive evaluations, through the standardisation of reports, which allow for comparisons.
Interview guides may be provided to evaluation teams unfamiliar with evaluation process, with a view to assisting them in their tasks.
WHAT ARE THE PROCEDURES FOR THE MANAGEMENT OF THE PANEL?
Fundamentals for the expert panel's work
There is no unique working process, and the expert panel should be encouraged to plan and implement its own workplan. Experts can focus their work on documentation and sessions, or broaden it to include meetings with project managers, field visits, implementation of surveys, etc. The choice of working process is highly dependent on the area studied, the experts mission, and the information and resources at the disposal of the experts.
Experts are expected to investigate and analyse the assigned topics and present their conclusions in a written report. The quality of the drafting and writing of the report is crucial and must be given careful attention.
Expert panels are usually expected:
- To provide studies incorporating the scientific and technical standards of the relevant fields
- To show complete independence from the programmes under review, which avoids any conflict of interest
- To reach a consensus in their conclusions and recommendations
Some commissioning agencies believe that the search for consensus usually results in the experts producing an anodyne and unrealistic report. They require a report highlighting the experts' different points of view and the reasons for these differences.
Confidentiality of the panel's debates and intermediate findings is another rule impacting on the panel's working arrangements.
Guidelines for the panel sessions
The first panel session
The content of the first panel session derives from the terms of reference. This session must result in the experts having a full understanding of their role in the evaluation.
During this session, the applicable methodology for the management of the panel's work must be discussed and validated. The discussion usually focuses on:
- The panel's organisation and the role of each member
- The type of investigation, the data collection methodology, and details of each panellist's task (such as field visits)
- The intervention work programme, the organisation of future sessions and their contents
During this session, panellists should be reminded of the general rules (such as independence and consensus) because although experts are generally familiar with the topics under study, they are often less well informed about evaluation principles.
The possibilities of conflict of interest with the programme to be studied should be closely examined, discussed and resolved during this session.
The next sessions
The following sessions (ranging from 3 to 5) will be directly linked to the panel's work. They will systematically deal with:
- The work carried out since the previous session
- Findings from investigations which are completed or in process
- Problems encountered (such as difficulty of collecting data, problems concerning the intervention timetable, the budget)
- Progress in editing the various documents, the review process, and quality control over these documents
- The tasks to be achieved before the next session and its envisaged content
With a view to ensuring the confidentiality of the panel's work, certain commissioning agencies recommend that records, summaries and intermediary reports of the sessions are destroyed. Only the final report is kept as the formal output required from the panel.
The organisation of the mission
The organisation of the mission depends on the panel's tasks. Most of the experts will be unfamiliar with evaluation techniques and may live far from the session's location. Thus, the production of the expert panel work programme should be scheduled well in advance, preferably as soon as the mission starts. This work programme should be adhered to whenever possible.
Experts responsible for tasks between two sessions, such as field visits, will be expected to work at least in pairs, in order to avoid bias of interpretation or empathy (the limitations). The formation of small groups should reflect the various points of view represented on the panel.
WHAT IS THE ROLE OF THE PANEL CHAIRMAN?
The panel chairman plays a crucial role. He/She guides the study panel, proposes the working arrangements, records findings, encourages contributions, facilitates debates and is the chief spokesperson for the panel. The quality of the working arrangements often depends on the chairman's leadership.
The various roles of the panel chairman
The chairman as Panel Facilitator
- At the first session, the chairman guides the panel to an agreement on a workplan and report architecture (as a working outline)
- Thereafter, the chairman schedules the work of the panel, the production of the documentation and its revision
- Throughout the study, the chairman ensures that each expert takes part in the working process and understands the evaluation's content.
- Given the essentially diverse composition of a panel, and the often considerable differences in initial views, the chairman must steer the panel's progress toward consensus on the range of issues involved. Fairness and flexibility should be employed.
The chairman as Report Architect and Integrator
The chairman guides the study, defines methodologies, reviews outputs, ensures that timetables are respected, and records the findings of the panellists, which includes:
- The provision of a critical overview to the panellists' outputs, designed to improve the debate rather than to control it
- The drafting of the successive versions of the report with the assistance of the technical writer and the commissioning agency
The chairman as Project Manager
The chairman ensures that the available resources are sufficient and properly employed throughout the study. He/She is in permanent contact with the commissioning agency on financial and technical issues. If sub-contractors work for the panel, the chairman is responsible for the management of their studies, the supervision of their progress and their successful completion of their work.
He/She ensures that the panel's sessions have been properly prepared by the technical writer, and that all documentation and means required for their participation in the sessions are provided to the experts in a timely fashion.
If the mission requires a readjustment of its budget, its time allocation or its objective, the chairman presents this to the commissioning agency in order to reach an agreement.
The chairman as Spokesperson
The panel will need to be represented in various bodies (such as monitoring committees) and possibly in meetings with the commissioning agency and the press. As it is impracticable to gather all the experts for these meetings, the chairman serves as the spokesperson for the panel. He/She may delegate certain tasks to other panel members, but he/she should conduct the most important meetings.
HOW DOES THE EXPERT PANEL
REPORT ON ITS WORK?
Synthesis of the panel's study
At the end of their mission, the experts report on their work. The report, which supports the experts' contribution to the evaluation, is the only output from the panel which is made available to the commissioning agency. Consequently, the report should be carefully prepared.
Guidelines for the final report
The report's structure depends on the nature of the mission. In technical or scientific missions, the report should at least include:
- An executive summary, which is not too technical if possible
- The mission's terms of reference
- The composition of the panel describing relevant information about the background of the experts (in an appendix if required)
- The evidence gathered and reviewed. The assumptions made and their impact on any conclusions should also be stated.
- The analyses carried out
- The conclusion of the experts in the context of the report's consensus findings. If consensus has not been reached, the report should state the consensus points, the elements of disagreement, and then explain them.
Reaching a consensus is the most challenging task for managers of expert panels, because a consensus strengthens the value of the panel's conclusions. In this context, the role of the chairman is crucial in seeking consensus, or formulating the final position of the panel, even if it include some dissenting views.
To find out more:
- Bibliography: reports of expert panels [FR]
- Example 4: expert panel report (World Bank) [EN] (178 kb)