- Choosing methods and techniques follows directly from the kind of questions one wants to ask and these questions are part of an extensive design exercise that includes consulting stakeholders and assessing programmes characteristics. Choosing methods and techniques first and trying to make them fit with questions for which they have not been specifically chosen will always create problems. The techniques chosen need to reflect the purpose and focus of the evaluation.
- Most techniques have strengths and weaknesses; these need to be recognised and where possible different techniques need to be applied together to strengthen the analysis and make the evaluation results and conclusions more reliable.
- Because of the distinctive character of socio-economic development: bottom-up, using different combinations of interventions and tailored to territorial and sectoral needs, it is difficult to measure and compare outcomes across socio-economic development programme settings. This doesn't mean that measurement, quantification and statistics are not relevant. They can be powerful tools when comparisons are at the level of the particular development programme and do not attempt to compare non comparable settings.
- Qualitative methods and techniques are well suited to socio-economic development because of the subtlety and holistic nature of what is being attempted and because of the differences in contexts which need to be described in qualitative ways. The participatory nature of local development building on the potential and ambitions of local stakeholders and citizens is especially suitable for both qualitative methods and participatory methods.
- Thematic priorities which are very common in European programmes pose real difficulties for evaluators. Because policy makers want to understand how far their policies are successful as a whole, there is often pressure to aggregate results and find a common way of describing or even measuring what is happening. This often cannot be done. Sometimes only qualitative descriptions will work. Take care not to add up apple and pears.
- There is often a tension between choosing evaluators who know a lot about a particular policy area and those whose evaluation skills are more generic. Ideally in an evaluation team one tries to span both of these sets of knowledge and experience. Commissioners need to be aware of the dangers of contracting evaluators who have lived most of their professional lives working in one specialised area and using a limited set of methods and techniques. This is another argument for looking at the balance of skills in the evaluation team.
- It is important to distinguish between methods and techniques for gathering data, for analysing data and for informing evaluative judgments. This distinction is not always made partly because those who undertake evaluations may be more preoccupied with one stage of the process rather than another. As in all things there needs to be a balance. It is no good investing in sophisticated methods to gather data but to be relatively simplistic in the way data is analysed.
- Data is never pure or naturally occurring, it needs to be produced. Because of this evaluators need to know from where their data comes and what decisions have made in the course of their production. At the end of the day the strength of the arguments and conclusions that can be drawn depend on the strengths and characteristics of the data being used.
- One important aspect in evaluating data follows from the way they have been accessed and how access has been negotiated. In socio-economic development programmes in particular there are a host of problems to be resolved. Different partners have to be willing to share information, excluded groups often distrust evaluation as one further example of official behaviour and need to be brought on board, and all stakeholders need to be convinced that they are going to get something out of an evaluation before they give access with any enthusiasm to any information they hold. Investing in these kinds negotiation processes will make a difference to quality and evaluation as a whole.
- The quantitative/qualitative divide is overstated. Data are often more of a continuum, beginning life as qualitative and once analysed becoming quantitative. Weaker forms of quantitative data (e.g., categorisations or ranking) are close to qualitative data. What is needed when evaluating socio-economic development programmes is qualitative data able to capture subtleties, people's experience and judgements and quantitative data to provide overviews, for example to aggregate results of an intervention and provide a comparative perspective.
- Well conceived indicators and monitoring systems are a powerful adjunct to evaluation. Very often evaluators depend on monitoring systems which are indicator based. If these are not put in place early in the programme design cycle it may be too late to create such systems later on.
- Over elaborate indicator systems may be counter productive. Whilst there is a temptation in multi-sectoral and multi-objective programmes to measure everything, this should be resisted. This can be costly and the results difficult to use.
- Indicators are often used for management and accountability purposes. It can be difficult to reuse indicators that have been exclusively developed for such purposes as part of an evaluation. There can be too much pressure to shape information in positive ways or at the very least make sure that bad news does not come through. On the other hand, within a well-developed performance management culture these kinds of indicators can help improve programme content and management.