Deliverables

D4.1 - Evaluation Framework

  • Open Public Rev... profile
    Open Public Rev...
    22 February 2017 - updated 4 years ago
  • 4 comments
    Total votes: 4
Closed since 28/03/17
Document(s): 
PDF icon d4.1_693514_evaluation_framework_.pdf
Author(s): 
empirica
Year of publication: 
2017

 

This report presents a conceptual and methodological framework for evaluating the three WeGovNow trials which are to be implemented during the second half of the project. Precise evaluation objectives are specified and evaluation methods as well as tangible outputs to be generated are described. Also, a roadmap for preparing and conducting on-site evaluation activities at each of the three trial sites is presented.

Tags: 
WeGovNow trialsevaluation objectivesconceptual and methodological evaluation approachinformation gathering techniquesevaluation outputsevaluation roadmap

Comments

Log in to post comments
Antonio Opromolla's picture
Antonio Opromolla,
27/03/2017 07:00

This document seems not totally focused. In details, the specific objectives (and the related indicators) at the basis of the information gathering sessions are not well defined and in general the document should have given more detailed information related to the aims of the different used techniques. For example: the interview trace, the number of the involved people during the focus group, the focus group trace, etc.
Moreover, the overview of the page 10 shows many thematic focus for the evaluation objcetive, but some of them would require the use of other techniques. For example, the usability of a system is generally measured by organized specific and well structured tests that could give more detailed information than the focus group or the survey.
In general, the advice is to use quantitative techniques (e.g.: the survey) during the first stage of the process and qualitative ones (e.g. focus groups) during the second stage, in order to focus specific emerged elements.

Total votes: 0
Yuri Misnikov's picture
Yuri Misnikov,
20/03/2017 22:07

It s important that the public value of a new type of citizen-centred services is discussed in this deliverable (on page 7). Yet it is advised in the future to provide some metrics to indicate how this value could be measured and thus convince that the project – as it claims – is going to break new ground in public service delivery and organizational and technological innovation. Impacts will indeed depend on ‘wider uptakes’ but these need to be detailed to the maximum extent possible, at least for the pilot cities where such value should be feasible to foresee and measure (e.g. via citizens’ perceptions and expectations).
Step 4 – Value case assessment (on page 9, also on page 13) mentions briefly ‘quantified costs and monetised benefits for each stakeholder, as far as these can be identified from the pilot implementation’. However, some benefits could have been already described in the use case context in this deliverable to make it less generic and descriptive and better understand how to project is going to generate public values thanks to the creation of a new ecosystem to collaborate with citizens on public service innovation.

Total votes: 0
Andrzej Klimczuk's picture
Andrzej Klimczuk,
12/03/2017 18:36

"Information gathering techniques" have been described in detail. However, it is not clear how big will be research samples related to particular techniques.

Total votes: 0
Frank Wilson's picture
Frank Wilson,
01/03/2017 17:35

The so-called "objectives" at 2.2.1 are not really objectives. The first states that there is a web based platform to be used in three trials sites, but not what the objective of that is. The second states that participating authorities will adapt their processes so that not just ICT, but a socio technical infrastructure will be available. There are not clear objective to focus the methodology.
The perspectives offered for evaluation are stated as "viability" and "sustainability" (2.2.2), but these are only vaguely defined, and not sufficiently articulated in relation to evaluation, or the target system.
The remainder of the document states that referenced methods will be used to collect information in relation to stakeholders, expected impacts, and actual value of implemented cases. The overall methodology, although lacking detail, appears generally appropriate. Challenges may be faced in detailing how e.g. the stated "service assessment model" can be realised, since it is stated mainly as aims, and so lacks operational detail.
The lack of operational detail may pose risks, and the promised "viability assessment" may be difficult to achieve without a robust method.
Assesment of "sustainability" is almost completely ignored, despite being a claimed objective.
A robust framework seems partially achieved.

Total votes: 1