Evaluators depend on data: the raw material that once collected is organised, described, grouped, counted and manipulated by various methods and techniques. Distinctions are often drawn between data that are primary - generated as a direct consequence of a programme or intervention - and secondary - generated for other purposes and pre-exist the programme or intervention. For example secondary data sources might include:
- Statistical sources such as national and regional statistics, EUROSTAT and other data bases kept by DG REGIO,
- Annual reports of development authorities or federations of enterprises, and
- Administrative records of public employment agencies, taxation returns, qualifications and training data.
None of this data happens without considerable effort and evaluators need to know how secondary data was put together before using them. What samples were used, how were outcomes defined, what is the timescale covered, what is the unit of analysis? It is only by asking these questions that a judgement can be made about their usability in a particular evaluation. Typically, for example, the geographical unit for which administrative or statistical data is gathered does not conform with the boundaries of the socio-economic development programme in question.
It is easier for an evaluator to understand the provenance of primary data. These can include:
- Monitoring data produced by a programme as part of its reporting obligations to funding authorities,
- Usage data generated by the use or uptake of services, funds or facilities provided by a programme, and
- Data collected from development sites and intended beneficiaries by evaluators through surveys of beneficiaries, counts of those using a consultancy fund, focus groups and stakeholder consultations.
However, here also data does not emerge fully formed. Their collection has also involved the application of protocols and techniques that specify what they can be used for. Does usage data differentiate between different types of users? Is monitoring information confined to financial data? How representative are samples of beneficiaries?
Because all data is processed and is the result of decisions made in the course of collection, evaluators need to understand what these decisions were, especially when these decisions were made by others, as with secondary data. This is not always easy, but it is necessary. With regard to primary data which is generated by or close to a programme and its evaluation - the evaluation team is better placed to know what decisions were made. Even here there will be a distinction between those forms of data that are directly produced by an evaluation and those that are generated by the programme, e.g., through monitoring, over which the evaluation team will have less control. However even when data is collected directly by the evaluation team its strengths, limits, scope and relevance need to be though through in terms of the kinds of future analyses that will be made and the kinds of arguments that the data will be expected to support.
This is a further argument for thinking through evaluation at the design stage with care. The collection of data needs to be considered in tandem with the choice of methods for analysis.