EU Science Hub

FAQ

How can I assess the robustness of a composite indicator?

The selection of an appropriate methodology is central to any exercise attem

pting to capture and summarize the interactions among the individual indicators included in a composite indicator or ranking system (Saisana and Tarantola, 2002).
Several practitioners have noted that the encoding process of building a composite indicator or a ranking system is fraught with uncertainties of different order. As a result, an uncertainty analysis should naturally include a careful mapping of all these uncertainties (/assumptions) onto the space of the output (/inferences). When this is done, two things can happen:

  • The space of the inference is still narrow enough as to be meaningful (the possible rank range is relatively narrow)
  • The space of the inference is too wide (no meaningful rank can be estimated for the universities).

The latter outcome calls in turn for a revision of the ranking system, or a further collection of indicators.

To this end, one needs to carry out a thorough uncertainty and sensitivity analysis of the Index or ranking system under a plurality of scenarios in which different sources of uncertainty are activated simultaneously. The sources of uncertainty should cover a wide and versatile spectrum of methodological assumptions (all with their advantages and implications). Such a multi-modeling approach allows one to deal with the criticism, often made to league tables and rankings systems, that ranks are presented as if they were calculated under conditions of certainty while this is rarely the case. This multi-modeling approach has already proven to be useful in the development and validation of several composite indicators such as the case studies available as related publications.

More info: Composite Indicators Research Group (COIN)

To use (or not) composite indicators?

The proliferation of composite indicators in various policy domains raises q

uestions regarding their accuracy and reliability. Given the seemingly ad hoc nature of their computation, the sensitivity of the results to different methodological choices during their development, and continuing problems of missing data, composite indicators can result in distorted outcomes and incorrect policy prescriptions. In practice, it is hard to imagine that the debate on the use of composite indicators will ever be settled.

Just to make an example, official statisticians may tend to resent composite indicators, whereby a lot of work in data collection and editing is “wasted” or “hidden” behind a single number of dubious significance.

On the other hand, the temptation of stakeholders and practitioners to summarise complex and sometime elusive processes (e.g. sustainability, single market policy, etc.) into a single figure to benchmark country performance for policy consumption seems likewise irresistible.

All things considered, composite indicators should be identified for what they are -- simplistic presentations and comparisons of performance in given areas to be used as starting points for further analysis.

More info: Composite Indicators Research Group (COIN)

What are the pros and cons of composite indicators?

Synthetically the pros and cons of composite indicators could be summarised

as follows:

Pros

  • Composite indicators can be used to summarise complex or multi-dimensional issues in view of supporting decision-makers.
  • Composite indicators provide the big picture. They can be easier to interpret than trying to find a trend in many separate indicators. They facilitate the task of ranking countries on complex issues.
  • Composite indicators can help attracting public interest by providing a summary figure with which to compare the performance across Countries and their progress over time.
  • Composite indicators could help to reduce the size of a list of indicators or to include more information within the existing size limit

Cons

  • Composite indicators may send misleading, non-robust policy messages if they are poorly constructed or misinterpreted. Sensitivity analysis can be used to test composite indicators for robustness.
  • The simple “big picture” results which composite indicators show may invite politicians to draw simplistic policy conclusions. Composite indicators should be used in combination with the sub-indicators to draw sophisticated policy conclusions.
  • The construction of composite indicators involves stages where judgement has to be made: the selection of sub-indicators, choice of model, weighting indicators and treatment of missing values etc. These judgements should be transparent and based on sound statistical principles.
  • There could be more scope for Member States about composite indicators than on individual indicators. The selection of sub-indicators and weights could be the target of political challenge

More info: Composite Indicators Research Group (COIN)

What is a composite indicator?

An indicator can be seen as something that provides a clue to a matter of la

rger significance or makes perceptible a trend or phenomenon that is not immediately detectable. An indicator's main defining characteristics are that it quantifies and simplifies information in a manner that promotes the understanding of complex phenomena, e.g. environmental problems, to both decision-makers and the public. Above all, an indicator must be practical and realistic, given the many constraints faced by those implementing and monitoring projects. They are often a compromise between scientific accuracy and the information available at a reasonable cost.

A mathematical combination (or aggregation as it is termed) of a set of indicators is most often called an "index" or a "composite indicator": Composite indicators are based on sub-indicators that have no common meaningful unit of measurement and there is no obvious way of weighting these sub-indicators.

More info: Composite Indicators Research Group (COIN)

JRC wide hidden block