Navigation path

News rss

Evaluation of Labour Market Policies and Programmes: the use of data-driven analysis

19/11/2012

The Peer Review investigated how administrative data can be used to conduct labour market policy evaluations, alone or in combination with other methods, as well as issues related to access, availability and quality of data and basic infrastructure needs.

The Department of Work and Social Economy of the region of Flanders (Belgium) hosted a Peer Review in Brussels that brought together ministry officials, social partners and independent experts from thirteen countries (Austria, Bulgaria, Denmark, Estonia, Germany, Greece, Ireland, Italy, the Netherlands, Norway, Poland, Slovenia and Sweden), as well as representatives from the Flemish government and the Belgian federal administration and from DG Employment, Social Affairs and Inclusion at the European Commission.

The Host Country presented its approach to using-data driven analysis for labour market policy evaluation, including the processes used to determine the priorities and to contract out such research. As emphasis on evidence based policy making has increased, but public resources available have declined, increasing emphasis is being placed on the use of administrative data in evaluation, combining this, where required with survey and other types of research. The Host Country outlined the key administrative database used to carry out labour market policy research; the unemployment register of the public employment service VDAB and the Crossroads bank of social security. The way in which these datasets combine with others relating to personal characteristics, social security and pensions within the data warehouse was explained in order to illustrate the available data and possible methods of analysis, including the example of mandatory activation for older unemployed persons and the calculated effects of this policy.

The main conclusions of the discussions are summarised under the following headings:

Do Member States make significant use of administrative data to conduct labour market policy evaluations - what is the impact of this trend?

The frequency with which administrative data is being used by Member States is increasing. There is also an increasing understanding of the potential use of administrative data in evaluation of labour market policy and a greater appreciation of the other sources of government-held administrative data which can widen the scope for policy evaluation.

A number of key developments were discussed which have either facilitated the increased use of administrative data or have acted as drivers to the development of more detailed administrative data. Among the factors which have facilitated the use of administrative data is the development of IT infrastructure and capacity. This has enabled better storage of data and increased the options for complex data interrogation. Additionally, many countries have witnessed mergers of administrative organisations (in an effort to provide one-stop-shop services), which has led to the merging of administrative systems and datasets.  Another important factor is the fact that in many countries, active labour market policies are co-financed by the European Social Fund (ESF); indeed, ESF managing authorities are legally required to process personal data concerning beneficiaries of ESF measures, to provide the necessary resources and to ensure that procedures are in place in order to produce and collect the data necessary for evaluation purposes. The above mentioned enhanced emphasis on evidence based policy making by increased use of quantitative evaluation methods in the context of greater budgetary stringency has also been a driving force in the further exploitation of such data.

What basic infrastructure needs to be in place to effectively utilise administrative data?

A number of Member States have well established systems for data warehousing which link a number of relevant datasets. The role of having a single national ID number was recognised as being of particular importance in this regard. For constitutional or political reasons not all Member States can however make use of such a unique identifier number. Typically these systems have been developed over a number of years, allowing linking of systems and streamlining of the infrastructure required to store and extract data. For Member States where the use of administrative data is less well developed there remains significant work to do in order to develop existing systems or establish new infrastructure.

A number of key data were identified by participants in the Peer Review as core administrative data which were most useful in conducting evaluation activity. It was recognised that these data would form a sound basis for evaluation activity but that they would require additional specific data depending on the specific requirements of an evaluation. The following were identified:

  • Monthly/weekly benefit data
  • Monthly/weekly employment data
  • Measures used
  • Household composition; medical information
  • Dates of notification letters

What restrictions exist in terms of access, availability and quality of data?

A number of data restrictions recognised by participants have significant implications for transparency and comparability between Member States.

A key issue relates to the process of accessing administrative data. To varying degrees access restrictions were relevant to both established data warehouses and those under development or proposed development in Member States. In most cases access to data is through a central administration, requiring a signed confidentiality agreement. This can add to the time taken in order to gain access to data, with consequent impacts on the time taken in order to carry out research. Furthermore it was noted that, in many cases, access to administrative data was restricted to registered citizens of the Member State concerned and/or could only be accessed if the proposed research related to a specific research project that had been commissioned by the Member State government or government agency. In some cases the right of citizens not to have their personal data stored and used in research was noted, with this varying both across Member States and within Member States.

Additionally the extent to which administrative data could be accessed remotely is varied, as is the extent to which central administrations provide services to advice on data use, pre-treated data and use data warehousing to generate revenue. These means that the costs for researchers of using administrative data can vary significantly, acting as a restriction to the ways in which it can be utilised in evaluation.

Restrictions in terms of the availability and quality of administrative data are also important considerations. Across Member States it was evident that there are significant differences in the availability of key data, for example medical profile data is not available in all Member States in a format which allows linking with Public Employment Service data. Administrative data on education and training is also generally limited. It was recognised that the quality of administrative data can be low, due to the limited time case workers are allocated to fill in data. In addition there are significant differences in the way that Member States store raw data, with some data categorised in ways that make analysis of data problematic, for example, wide banding of income data.

Analysis of administrative data is also impacted by the expertise of those that input, store and provide data. In some Member States it was noted that where these organisations had experience of carrying out research the service that they provided to researchers was of higher quality than those that did not. It was also noticeable that the quality of data was impacted by the size of Member States and/or the size of the programme for which data was being stored.

What can appropriately be measured by using administrative data and when is a combination of methods needed?

It was recognised that the specific research requirements must determine the sources of data that are used and the combination of research methods that are applied. The following key considerations were noted in this respect:

  • Establishing the extent to which the evaluation is seeking to establish how a policy works or why it works is a key issue in determining the required data and methods.
  • Using administrative data in isolation can provide an incomplete picture; the primary use of administrative data should be for monitoring purposes.
  • A combination of administration and survey data is usually the most effective way of evaluating labour market policy

The choice of data and methods will always depend on data availability in different countries.

Ideally, the type of administrative data to be collected for monitoring and evaluation purposes should be discussed prior to the launch of new policy measure is launched.

Two additional factors were recognised as influences on the sources of data used and the selection of research method. Firstly, the fashion for certain research methods and the specialities and research interests of research teams can influence the selection of data and methods. Secondly, it was noted that where there is a requirement to publish research this can impact on the quality of the work undertaken, usually ensuring that it is of higher quality.

What role is there for transnational/European level analysis and what role can the EU play?

There was agreement that transnational comparisons using administrative data was desirable, however there are a number of issues associated with this which could be mitigated through EU coordination. The key issues to be developed in this respect are:

  1. There is a need to firstly agree a small number of indicators and the definitions for these indicators in order that they can be used as the basis for accurate comparison. Work on this issue has already been carried out in the EMCO indicators group and the PES benchmarking group and further co-ordination may be helpful in this area.
  2. In recognition of the different starting points of Member States, it was suggested that pilot activity in this respect should initially involve a small number of Member States that were at similar stages of data warehouse development.
  3. Improving the use of panel data for the Labour Force Survey and enhancing the commonality of definitions and reliability of the European Labour Market Policy Database would improve the ability to carry out transnational evaluation.

more news

    Share

  • Tweet it Share on Facebook Share on google+