University ranking: the best approach is yet to be found© Yaroslav B (stock.xchng)
Higher education rankings: are they statistically reliable?
The Academic Ranking of World Universities carried out annually by Shanghai's Jiao Tong University (SJTU) has become a reference for scholars and policy makers in the field of higher education. However, a recent study by the JRC has cast doubt on the substance of statistics upon which the ranking is based. The study, carried out by researchers at the Centre for Research on Lifelong Learning based on indicators and benchmarks (CRELL) and published in September, asks whether the Jiao Tong ranking serves its intended purpose and compares it with the most widely cited European alternative, Britain's The Times Higher Education Supplement (THES).
Uncertainty analysis of the 2008 SJTU ranking
The results of the JRC report, written by Michaela Saisana and Beatrice D’Hombres and entitled Higher Education Rankings:Robustness Issues and Critical Assessment, show that neither ranking actually succeeds when it comes to assessing Europe’s universities. The Jiao Tong system is statistically robust only in the identification of "top performers" either side of the Atlantic and is found to be quite unreliable in its classification of other institutions. Furthermore, Jiao Tong focuses exclusively on universities' research performance, thus equating research and education. While the THES ranking takes a step in the right direction by including measures of educational quality, it is otherwise fragile in its ranking approach which is undeniably biased towards British institutions. The British survey is also found to be inconsistent in its treatment of objective data and subjective variables extracted from surveys.
The JRC report points to the need for a ranking system suited to European universities in order to make institutions more accurately comparable, encourage student mobility and facilitate the implementation of a European Research Area. In the meantime, it suggests that an assessment of university performance based on a hybrid set of the twelve indicators used in the THES and SJTU rankings may provide a more reliable average ranking of the institutions.
CRELL is a part of the JRC Institute for the Protection and Security of the Citizen (IPSC), based in Ispra, Italy.
24/11/08