Deliverables

D4.5 - Pilots and Evaluation Plan - V3

  • OPR coordinator... profile
    OPR coordinator...
    16 January 2019 - updated 2 years ago
  • 6 comments
    Total votes: 4
Closed since 17/02/19
Document(s): 
PDF icon ogi_d4.5_pilots_and_evaluation_plan-v3_1.0.pdf
Year of publication: 
2018

This document reports the evaluation methods of OpenGovIntelligence (OGI) project. The OGI environment provides an ICT toolkit comprising easy-to-use and user-centric tools to facilitate realizing a Linked Open Statistical Data (LOSD) innovation ecosystem. Pilots will be executed to validate and prove the usability and effectiveness of OGI ICT toolkit to co-create and innovate ecosystems.

OpenGovIntelligence (OGI) goes beyond traditional top down approaches and proposes the co-initiation, design, implementation and evaluation of innovative, data-driven public services. Six pilot projects are developed which take different approaches and look at various aspects of Linked Open Statistical Data (LOSD). These services exclusively address specific society’s needs, improving in that way the effectiveness of public sector’s processes, but also promoting the citizen-centric character of these processes.

The first part of the evaluation is related to the co-creation steps performed. The second part concerns the evaluation of the OGI Toolkit by developers. The third part is the acceptance of the OGI ICT Toolkit by endusers. The fourth part is the evaluation of the outcomes of pilots’, which includes Transparency, Administrative Burden Reduction, and, Costs Reduction.

Comments

Log in to post comments
Tommaso Federici's picture
Tommaso Federici,
17/02/2019 23:54

The evaluation plan is well designed, encompassing several aspects and defining multiple criteria to evaluate, taking into account differences among pilots, and with a credited theory basis.

Having said this, I leave two comments: 1) I subscribe Aimo's remark about the need of differentiating the categories of users in Table 7, at least between citizens and business, and possibly even more; 2) the document should be revised and cleaned.

 

Total votes: 0
Aimo MAANAVILJA's picture
Aimo MAANAVILJA,
17/02/2019 09:06

Evalution plan is well planned based  on scientific literature. However in chapter 3.3  figure 7 and Table 7 - Taxonomy for Benefits for OGI innovation ecosystem stakeholders use of open data to create new applications and bussiness by bussiness and Startup community clould be more clearly identified as one possible stakeholder. As background one example on research in this area please find this doctoral thesis: https://lutpub.lut.fi/handle/10024/158560

Total votes: 0
Antonio DE NICOLA's picture
Antonio DE NICOLA,
13/02/2019 16:23

The deliverable presents the evaluation plan of OpenGovIntelligence pilots.

The plan is well detailed. It provides web references to the six selected pilots.

The plan considers different stakeholders: developers, decision-makers, policy-makers, and end-user. This multi-perspective evaluation of the pilots is highly appreciated. Furthermore the project identifies four areas to be evaluated: the co-creation framework, the OGI toolkit, the acceptance of OGI pilot applications, and the outcomes. 

The pilots’ timeline makes sense.

As minor comments, Section 1.1 refers to Deliverable 4.4 instead of Deliverable 4.5 and there are some parts of the text written in greek (probably references to other parts of the deliverable).

Total votes: 0
Gemma Molero's picture
Gemma Molero,
12/02/2019 17:36
  • An evaluation plan of pilots based on time and money savings and the increasement of the quality service should be issued.
  • The samling strategy is sufficiently clear regarding the questionnire design (number of questions per each point), but not regarding the popullation to fill the questionnaire per each use case.
Total votes: 1
Francisco Santarremigia's picture
Francisco Santa...,
11/02/2019 12:49
  • Good literature review and application of scientific methods, however at the end of the day OGI benefits are finally evaluated on opinions through questionnaires. The monitoring of some quantitative KPIs measured in each of the 6 pilots, based on OGI focal points according with the figure 7 would be useful.
  • References should be alphabetically ordered.
  • Some text is in Greek in the deliverable. Please review it (page 8, pg 14, pg. 32)
Total votes: 1
Antonio OPROMOLLA's picture
Antonio OPROMOLLA,
03/02/2019 20:22

One aspects to consider: it is not clear if the intention of the deliverable was to define a plan of evaluation for all the project or exclusevely of the interactive platforms. It seems that it was addressed to the different aspect of the project, but in the document we also can find element to the evaluation of the interactive tools.

The description of the tools and methods for evaluation and of the sources to collect data is not specific enough. In particular, the description of elements as user workshops, survey, continued participation, etc. is not accurate, since more specific objectives and characteritics need. The Pilots’ Data Quality Evaluation Methodology is clear but more specific elements need (number of involved respondents, is the most important lack). Concerning the Pilots’ Acceptance and Outcomes Evaluation Methodology, it is not clear how the "User Acceptance Evaluation for OGI" of Tablle 5 is integrated with the elements of Transparency and Administrative Burden that are listed below, and their link with the Pilots’ Outcomes Evaluation Questionnaire of Table 8. The elements which are part of the OGI ICT Toolkit evaluation methodology are described in a generic way.

The general elements which characterises the description of the four investigated areas are more clear, reading the specific survey in the Annexes. However, no indication about objectives and KPIs are provided, so it probably does not be clear if the evaluation will be positive or negative.   

Total votes: 0