STAGE 1: DEFINE THE CONDITIONS FOR ITS USE
Conditions for the implementation of cost-effectiveness analysis
Check the relevance of the analysis to the objectives of the programme
This stage checks whether cost-effectiveness analysis constitutes a relevant analytical tool for the programme under study.
If the outcome of a programme cannot be defined as a priority outcome, or if homogeneous and quantifiable units cannot be determined, the use of cost-effectiveness analysis should be avoided.
Identify the availability and reliability of the data
Before undertaking cost-effectiveness analysis, the availability and reliability of the data relating to the costs and the quantitative indicator on the results and outcomes should be checked. The analysis requires reliable data, i.e.:
Determine the effectiveness criteria and develop the relevant indicator
- In ex post evaluations, a quantification of the outcomes
- In ex ante evaluations, a rigorous modelling of the most probable results
Selection of the effectiveness criteria
Once the feasibility of cost-effectiveness analysis is confirmed, the choice of the effectiveness criteria depends on the main objective of the intervention.
In an intervention where the objective is clearly determined, the identification of the effectiveness criteria is straightforward. Conversely, when the intervention's objective is broad (for example, an intervention improving the efficiency of basic education), the identification of the intervention's main objective should be discussed (for example, choosing between the reduction of illiteracy, the increase in the average level of the basic knowledge provided in primary school, or an increase in the number of primary school pupils).
The quality of the answers produced by cost-effectiveness analysis depends on the selection of the criteria. This selection could be done, when required, through the participation of stakeholders in a focus group. In a national programme, the selection should be the responsibility of the specialists of the area under study, gathered together in an expert panel.
Example of the selection of effectiveness criteria
From 'Cost-effectiveness analysis', 2000, p.110.
|Objectifs du projet
- Reduce the consumption of drugs per person
- Increase the number of graduates from secondary school
- Improve the level of Brazilian secondary school students
- Increase adult literacy
- Quantity cocaine ingested per person
- Number of students with the secondary certificates
- Results of the Portuguese language and mathematic examinations
- Results from the testing of adults benefiting from the literacy programme
Selection of a relevant criteria
The development of criteria depends on several factors. In many countries, the scarce and limited reliability of statistical sources limits the range of choices. Thus, finding means of observation required for the use of an indicator is crucial.
For example, when an intervention aims at improving the effectiveness of the basic education, the effectiveness criteria could be the increase in the average level of the primary school's basic knowledge. This increase can be measured through the evolution of the grades obtained in all the courses followed by the students, or in the two courses deemed as the most important (for example, reading/writing and mathematics), or through the organisation of a single examination for all primary school students.
In most of the interventions, the selection of the indicator can strongly influence the conclusions of the analysis. Thus, it is recommended that this selection should be conducted during a focus group or an expert panel session. The selection of the effectiveness criteria should be conducted likewise.
Although a single focus group is enough to determine the criteria and the indicator, complex cases require more time to determine the criteria and organise a second working session for the indicator selection.
STAGE 2: EVALUATE THE TOTAL COST OF THE PROGRAMME
Add direct costs
In the context of the analysis of the development assistance intervention, the simplest and more commonly used methodology consists of adding all the public resources used in the programme. In this type of calculation, only the direct costs invested in the intervention are considered. These costs are often financial: subventions, financial transfers, decreases in taxes, financing of projects and activities, etc.
Direct costs calculation
Examine indirect costs
|Direct costs in health projects
Direct costs are all the resources and expenses directly attributed to the programme, i.e. during its creation and implementation (for example, in a health programme, the investment in buildings and medical equipments, fees, salaries, medicines). Two types of costs can be distinguished: variable costs which depend on the volume of the services provided (for example, consumables and medicine), and fixed costs, which do not depend on the volume, at least within the short-term (for example, buildings and equipment).
Direct costs include the medical and non-medical expenses spent at the hospital or at home by the patient or a member of his/her family: payment of the additional medical costs, transport expenses, food, medical equipment at home, etc. are included in this category of costs.
Extract from 'Méthodologie de l'évaluation économique des programmes de santé'
Indirect costs, such as the costs of human resources required for the management of programmes, indicate the value of civil servants' work in charge of monitoring the programme or intervention. They should be taken into account whenever they constitute an important part of the intervention's costs.
However, estimating indirect costs can be complex and it is recommended that the relative importance of these costs in relation to the intervention's direct costs is assessed first, in order to decide whether the indirect costs are likely to be important for the analysis.
Indirect costs calculation
Examine other types of costs
|Indirect costs in health projects
In the context of a health programme, indirect costs are:
- Production losses due to working hours lost for the patient or his/her family because of his/her illness or participation in a medical programme (for example, working hours lost against the benefit of a vaccination or a compulsory medical examination)
- The range of resources of a medical programme which are not available, because they have been already spent (for example, opportunity cost of the time needed by the patients to go to the doctor)
- Social costs, i.e. costs caused by a change in the economic resources beyond the scope of the medical consequences of the programme under study. For example, the adjustments in the timetable and working conditions of a pregnant woman in the context of a premature birth alleviation programme require an increase in investment in equipment and staff.
Extract from 'Méthodologie de l'évaluation
économique des programmes de santé'
Cost calculations can include the loss of earnings and benefits due to the fact that public financing have been attributed to a specific objective (this is called the lost opportunity cost).
This methodology for cost estimation is justified when alternatives for the use of resources are clearly identified, which is seldom the case.
Calculation of other costs
|'Intangible' costs in a health programme
In the context of a cost-effectiveness analysis of a health programme, intangible costs are related to stress, anxiety, pain and, more generally, to all the losses felt by the patient to well-being and the quality of life.
They should be systematically considered in medical evaluations, where their non-monetary dimension is crucial. Yet, the main difficulty is the quantitative estimation and the monetary appraisal of costs in cost-effectiveness analyses, which are mainly qualitative and subjective. Thus, depending on the perspective and the type of analysis, intangible costs can either be considered as costs, or as the consequences of the programme under study.
Extract from: 'Méthodologie de l'évaluation
économique des programmes de santé'
STAGE 3: MEASURE THE IMPACT OF THE PROGRAMME
This is the most difficult stage of the process.
Preconditions: nature of the information
The study of the programme's impact depends on the nature of the information at the disposal of the evaluator.
Ex ante evaluations
In ex ante evaluations, the evaluator must forecast the quantitative results of the programme. Depending on the complexity of the intervention, the use of simulation techniques may be required.
Ex post evaluations
In ex post evaluations, the evaluator can use empirical techniques if the primary data at his/her disposal are sufficiently numerous and reliable. If not, the evaluator needs to estimate the actual quantitative results from secondary data.
The analysis may be undertaken, as follows:
- Allocate weightings in order to examine the various impacts or secondary variables.
- Define the criteria and methodologies used to quantify qualitative elements. Theoretically, cost-effectiveness analysis only uses quantitative elements; yet, quantifying variables which are a priori qualitative (for example, quantification of the results of environmental policies) may also be required and helpful.
- Transform quantitative data of various types and sources into a single and homogeneous quantitative unit.
To find out more:
|Cost-effectiveness analysis in the context of the ex post evaluation of a reafforestation programme
Cost-effectiveness analysis studies one of the objectives of the United Kingdom's Action Plan for biological diversity: expansion of the forests of indigenous pinewood so as to protect nature. The analysis examines the various ways to realise this objective in the context of the forest administration's subvention programme for afforestation. This programme enables private landowners to get a subvention. Costs and effectiveness are assessed for the 200 newly pinewoods created since 1988. The costs are the estimated public expenses defined in the subvention programme. The effectiveness is the extent to which the objective is reached.
The evaluation was carried out by a group of experts in afforestation. They have attributed weighting coefficients and grades to specific characteristics of the woodland in relation to the role they played in the creation of a natural pinewoods ecosystem. The effectiveness grades for each woodland are deduced from the weighting coefficients and the woodland grades. They are recorded in a geographic-based information system. Average cost-effectiveness ratios are established for various types of woodland, such as the ones originating from new plantations and regeneration measures.
The evaluation suggests how to change the subventions ratios provided by the programme to landowners in order to improve the correlation between financing and effectiveness.
Extract from: 'Analyse coût-efficacité de la création d'un écosystème
de terres boisées', Résumés des mémoires volontaires, 1997, p.210.
STAGE 4: ESTABLISH A COSTS-TO-EFFECTIVENESS RATIO
The analysis requires stable elements to support the comparison between:
- Interventions of different nature, but similar objectives, and occurring in the same country
- Similar interventions occurring in similar contexts
- The results and what would happen without the intervention, etc.
Comparison of programmes with identical outcomes
When the analysis compares different programmes with identical outcomes, the study is straightforward because the chosen parameter is the cost comparison criteria.
Comparison of programmes with identical costs
When, for the same objective, the analysis compares different types of interventions with identical costs, the study becomes challenging.
The comparison between indicators from different quantitative outcomes will be partly supported by qualitative elements. This is the case, for example, when the study compares the number of people cured, and the number of infections averted, with a view to assessing the relative cost-effectiveness of curative and preventive measures in the health field.
Decision-making is usually political. The evaluator uses cost-effectiveness analysis as a decision-making assistance tool designed to facilitate the decision.
Develop and lead the process of thinking
The tool can be used:
- To structure the thinking-process prior to decision-making
- To lead the groups representing various categories of beneficiaries and stakeholders involved in the sectors in which the intervention is planned toward the definition of preferences
Combine cost-effectiveness analysis with multicriteria analysis
Cost-effectiveness analysis is not a sufficient tool to conduct a debate between the representatives of various interests and help them define priorities together. It must be supported by a multicriteria analysis. In this context, cost-effectiveness analysis constitutes the first stage of the multicriteria analysis.
To find out more:
|Cost-effectiveness ratio for a HIV prevention programme
Controlled testing has been carried out in Mwanza Region (Tanzania), following collaboration between the African Medical and Research Foundation, London School of Hygiene and Tropical Medicine, and the Tanzanian National Institute for Medical Research.
The impact of improved STD services on HIV infection was assessed in the general population over two years. Cost data were collected and valued by means of the standard 'ingredients' approach. Because of the nature of the available information, it has been possible to calculate the cost per HIV infection averted and the cost per DALY (Disability Adjusted Life Year) of the intervention. The findings were dramatic in their positive effect and in the strategy's cost-effectiveness. As a result, the improvement of STD services in Africa has become a policy area of high priority.
What were the study's qualities?
From 'Cost-effectiveness analysis and HIV/AIDS', 1998, p.7.
- Knowledge gap identified - is improving STD services an efficient way to reduce HIV infections?
- Selection of a reproducible intervention
- Selection of a large sample
- Standard ingredients approach for the costing
- Averted cases measured with randomised controlled testing
- DALYs per averted infection and cost per DALYs calculated with the standard approach
- Sensitivity analysis performed with key variables