Study finds evaluation culture on the rise
A European study offers fresh insight into the systematic cataloguing of the emerging patterns of policy evaluation in Europe. The findings are an outcome of the ADAM ('Adaptation and mitigation strategies: support European climate policy') project, which was funded under the 'Sustainable development, global change and ecosystems' Thematic area of the EU's Sixth Framework Programme (FP6) to the tune of EUR 12.9 million. The study is presented in the journal Policy Sciences.
The politics surrounding new policy development has piqued the interest of Europeans since the early 2000s. In particular, Europeans targeted and implemented several policies during this period. Despite the increase, however, little information exists on what is being done to ensure the success of the resulting policies.
Led by the University of East Anglia (UEA) in the United Kingdom and the Vrije Universiteit (VU), University Amsterdam in the Netherlands, the researchers conducted a meta-analysis and found that a culture of evaluation is emerging. More and more evaluations have materialised in recent years. Information obtained for six EU Member States, and for the EU in general, show that there were eight times as many reports generated between 2000 and 2005. It should be noted, however, that a more profound increase was found in some Member States than in others.
Britain-based policy effects were more commonly evaluated than those in Poland and Portugal, according to the researchers. But other differences exist with the culture of evaluation as well. For instance, most of the 259 evaluations identified and probed also adopt a relatively narrow selection of evaluation tools and insufficient intensive stakeholder involvement.
The data shows that more than 80 % are not critical; they take existing policy goals as given. The researchers say the majority are also quite narrowly framed, targeting mostly on the environmental effectiveness and/or cost effectiveness of existing policies.
'Whether climate governance is undertaken through the United Nations or — as now seems more likely — via more informal 'pledge and review' type processes, evaluation practices are absolutely crucial for fine-tuning policy interventions and building and sustaining public trust,' says one of the lead authors of the paper, Professor Andrew Jordan of the Tyndall Centre for Climate Change Research at the University of East Anglia.
'The most striking finding of our analysis is just how undeveloped and unsystematic are most current evaluation practices,' he adds. 'Great efforts have been made to inform and understand policymaking procedures in Europe, but most policy evaluation remains piecemeal and non-consultative.'
Calls for evaluations to be performed in a more transparent manner will increase as the political pressure on policymakers to elucidate the efforts being made to deal with climate change intensifies. But Professor Jordan says the current policy systems in Europe are not ready to rise to the challenge.
For his part, co-lead author Dr Dave Huitema from University Amsterdam's Institute for Environmental Studies explains that there was a 'wide gap between evaluation theory and practice, which suggests that current evaluations underestimate the complexity of climate change issues.'
The findings reveal that university researchers are the most active policy evaluators in Europe, and 58 % of evaluations are not commissioned. Policymakers have the capacity to give the overall evaluation effort a boost by commissioning more evaluations from more organisations. But they note that this may not result in a more active and questioning culture of evaluation.
On the one hand, there is a greater chance of non-commissioned evaluations questioning policy goals as commissioned ones. On the other hand, parliamentary bodies have produced a relatively large number of critical evaluations. So, improving the quality and quantity of evaluations is a shared responsibility, the researchers say.