IMPORTANT LEGAL NOTICE - The information on this site is subject todisclaimercopyright notice
Evaluation
  EUROPA > European Commission > EuropeAid > Evaluation > Methodology > Evaluation tools > Expert Panel
Last updated: 27/09/2005
EuropeAid

Expert panel

EuropeAid
 

Why and when is an expert panel established?

 


Methodology
• Evaluation Guidelines
• Methodological bases
• Evaluation tools
• Examples
• Glossary
• Sitemap

Expert Panel
• why and when
• how
• quality control
• examples
• bibliography

 
 

WHAT IS THE ROLE OF AN EXPERT PANEL IN AN EVALUATION?

What is an expert panel?

Definition
An expert panel usually comprises independent specialists, recognised in at least one of the fields addressed by the programme under evaluation. The panel specialists arrive at conclusions and recommendations through consensus. Depending on the project proposal, these recommendations deal with the implementation or the impact of a programme, or part of it. When consensus is not reached for particular questions, the panel must report on the various perspectives of the experts.

The expert panel is specifically appointed for the evaluation, and in conformity with standard procedures. The panel holds meetings and provides conclusions and recommendations in accordance with a precise and replicable working plan, which accounts for its reliability.

This tool is therefore designed to take advantage of the experts' knowledge in assessing policies, programmes and projects implemented in the field of their expertise.

Various types of expert panels in evaluation
International funding institutions such as the World Bank, the European Commission, Scandinavian countries, Canada and the United States regularly appoint expert panels to evaluate programmes.

In environmental evaluations, funding institutions such as the World Bank use expert panels to assess the quality of the service providers' work and suggest improvements. These panels examine the evolution of the project studies, from their preliminary stages to completion, including the first years of implementation.

The Delphi Method is another type of expert panel used for evaluation, based on an anonymous and repeated postal survey with experts (see Means Documents).

Where does the expert panel come from and how has it evolved?

The panel's origin
The concept of the expert panel originates in the research field. It derives from the peer review of scientific work processes or programmes, where expert panels have developed as an evaluation tool.

Its evolution
Two main developments over the past few years are noteworthy.

  • The panel's composition continues to diversify. Whereas the expert panel's missions were originally focused on limited areas of inquiry, their content and the panel's composition have been progressively diversified. Thus, in addition to programme specialists, economists, evaluation specialists and representatives of programme users are now appointed to panels, with a view to assessing all aspects of the evaluation.
  • The design of the panel's tasks has become more professional. The emergence of rigorous methodologies (and additional studies conducted by independent consultants verifying the panel's conclusions) has increased expert panel's credibility and changed expert panels into a reliable evaluation tool. These improvements have progressively given the expert panel the characteristics of a professional evaluative tool.

To find out more:

Top

 

WHEN IS AN EXPERT PANEL APPROPRIATE FOR AN EVALUATION?

What are the pre-conditions for its use?

Expert panels are appropriate for many evaluation situations, and particularly for:

  • Subjects which are well defined and require the advice of highly specialised experts. This is common in the research field, where the European Commission often uses this tool during evaluations.
  • Highly focused subjects (such as assistance to a small country evaluation) which do not require expensive allocations of resources.
  • Highly complex subjects (such as budget support), which are difficult to assess at a reasonable cost using other tools. In these cases, the point of view of experts specialised in the subject and/or in the country can constitute credible information and evidence for the evaluation.

Various tasks of an expert panel

With which tools can the expert panel be combined?

The expert panel can be combined with almost all the usual evaluation tools.

As well as combined with other tools, the expert panel's work is sometimes complemented by external studies. This usage has become widespread in fields such as research evaluations because it eases the task of the experts and provides them with information about the programme under evaluation. External studies include preliminary studies, surveys, database analyses.


Top

 

WHAT ARE THE ADVANTAGES AND LIMITATIONS OF AN EXPERT PANEL?

The advantages

The experts' knowledge of the subjects under evaluation is the principal advantage of this tool. It fosters:

  • Significant reductions in time allocations, when compared with other evaluation tools
  • Cost effectiveness compared with other tools, due to reduced time allocation
  • Credibility of the conclusions, due to the experts' reputations
  • Adaptability to a variety of situations encountered in evaluation

The limitations

The tool's limitations which should be minimised essentially derive from a series of risks:

  • Because the panel must come up with consensus-based conclusions, its organisation tends to eliminate minority points of view and tone down conclusions.
  • The point of view of a 'dominant' expert can be over-influential within the panel, to the detriment of other perspectives and the report.
  • When the panel includes members who are the only specialists in the fields to be studied, an inappropriate empathic bias may occur. This could be exacerbated when the experts are acquainted with the actors of the field under evaluation. To mitigate this situation, appointments member of the panel must be reminded of the need for independence.
  • Experts have a tendency to go beyond their field of competence and the credibility of the conclusions can be adversely affected. Therefore, the panel's work should be strictly focused.

Even when these risks are controlled, social science detractors remain sceptical about the reliability of expert panel's conclusions.


Top

 

WHAT ARE THE PRE-CONDITIONS FOR A USEFUL PANEL CONTRIBUTION TO AN EVALUATION?

The time span

One of the advantages of an expert panel is in its speedy assembly process. For an evaluation, only 3 to 6 months work needs to be scheduled, and even less time for panel advice on a technical field within an evaluation.

Human resources

The core aspect of an expert panel is the issue of human resources. Experts must have recognised expertise in the field under evaluation, be independent of the programme being assessed, be able to work in a group and be available for a continuous work throughout the evaluation.

To find out more:

Financial resources

The expert panel is known for its cost-effectiveness, but in the case of travel into remote countries and field visits, the expenditure allocated for experts' salaries and expenses will need to be increased.

Budget line items normally taken into account while preparing estimates are as follows:

  • Salaries for the experts and the technical writer. If necessary, estimates may include salaries for subcontractors responsible for the panel's external studies.
  • Communication and travel costs, publication, and dissemination costs related to the reports
  • Translation costs, if required
Top

 

WHY AND HOW IS AN EXPERT PANEL ESTABLISHED IN COUNTRY/REGION EVALUATIONS?

Reasons for establishing an expert panel

The use of an expert panel in country/region evaluations can be helpful in several situations, such as:

  • Studying very specific fields requiring a high level of competence (such as assistance for the research field and high technologies)
  • Studying subjects for which other tools are difficult to implement (for example, budget support)
  • Carrying out limited-scope evaluations (such as assistance to small countries)
  • Assisting the evaluators in their conclusions on a subject in complex evaluations
  • Providing assistance in the drafting of final conclusions relating to the possible impacts of a programme in ex ante evaluations

Conditions for its use in country/region evaluations

No specific conditions are required for the use of expert panels in country/region evaluations. However, experts must be familiar with the context of the country assistance under study.

Examples of its contribution to country/region evaluations

No example of its contribution to country assistance has been found yet, but this tool is frequently used in programme evaluations in Western countries. A series of examples highlight the use of expert panels in the development of terms of reference documentation, recruitment profile processes, samples of reports.

To find out more:

  • Examples: specific references on the contribution of expert panels to country/region evaluations
Top