Deliverables

D4.6 - Pilots Evaluation Results - Third Round

  • OPR coordinator... profile
    OPR coordinator...
    16 January 2019 - updated 2 years ago
  • 9 comments
    Total votes: 4
Closed since 17/02/19
Document(s): 
PDF icon ogi_d4.6_pilots_evaluation_results_third_round_v1.pdf
Year of publication: 
2019

This is a draft version of the deliverable “D4.6 Pilots Evaluation Results – Third round” of the OpenGovIntelligence (OGI) project. The evaluation of the project is divided in four main areas; the evaluation of the 1) Co-Creation Framework, 2) OGI ICT Toolkit, 3) Acceptance, and 4) Outcomes. The first two areas were evaluated in the first and second year reports (D4.2 and D4.4) and in this report are only summarized. The other two areas are evaluated in this deliverable as soon as the final version of the pilots has been delivered and the user-base is sufficient to arrive at sound outcomes.


Acceptance and Outcomes were evaluated using a mixed qualitative approach. Technical partners and pilots’ leaders were interviewed and surveyed to collect information about fundamental requirements and parameters for the ICT Toolkit development. This shows the potential and limits of the LOSD OGI Toolkit. Data quality was also evaluated. In particular, the 6 pilots were evaluated by employing a user – questionnaire involving 219 participants in total. Participants, in general, believe that among the pilots’ benefits are the creation of transparency, reduction the administrative burden by more efficient search of information and visualizing the results at a glance.

Comments

Log in to post comments
Tommaso Federici's picture
Tommaso Federici,
17/02/2019 23:59

I fully subscribe all Gemma's detailed remarks, as well as that of other commenters about the fact that, beside what already in the report D4.5, only results of Greek pilot are presented, not allowing the reader to draw a full picture of the project outcomes.

Total votes: 0
Francesco ARCIERI's picture
Francesco ARCIERI,
15/02/2019 11:23

The overall structure is good and compltete, but the deliverable should be completed with the OGI EVALUATION RESULTS for ALL the pilots an not only the Greece related one.

 

Total votes: 0
anastase ADONIS's picture
anastase ADONIS,
14/02/2019 18:38

The deliverable sounds complete.....and hyper-methodological....accompagned with theory on methodology.... How to do is good under the condition to have somethong to do.

Correct and somewhere overdimensionned set of evaluation criteria for so simple pilots applications and statistical datas...... There is no clear what was the in-field implementation/test/evaluation work..... what were the users, their work (application) context, the conditions to interact with.....

Absence of indicators which could allow the stability of the evaluation on the time. The project sounds like "we do something whithout interest on how to use-it once made..."

If this deliverable is structure in that manner, with references to "papers" it shows that the feedbacks from the field were poor or unexisting. There was a rare opportunity to collect feedbacks on what could be "intelligence on eGov" for the next decades.

A very disapponting deliverable.

 

Total votes: 0
Antonio DE NICOLA's picture
Antonio DE NICOLA,
13/02/2019 16:29

The objective of the deliverable is to present the pilots evaluation results. However most of the document is devoted to describe the evaluation methodology (already presented in the deliverable 4.5) and to the end-user questionnaires. Only the results of the Greek pilot are presented. It is not clear why the results of the other pilots, or a summary of them, are not presented.

Total votes: 0
Gemma Molero's picture
Gemma Molero,
12/02/2019 17:38
  • A large part of the deliverable, including Annexes, is exactly the same tan the deliverable 4.5. To use references to this deliverable in the same way that it has been done with other deliverables will avoid a lot of unfruitful pages in the document.
  • Why is only detailed the pilot outcomes of Greece (belonging to 45 surveyed people) and outcomes from other pilot cases as Lithuania, Estonia, England, Belgium and Ireland (belonging to the other 219 minnus 45 surveyed people) are not detailed?
  • Tables 4 and 5 are not introduced and explained in the text of the deliverable.
  • The table 8 is twice in the document. Please, review.
  • References should be in an alphabetic order.
Total votes: 1
Francisco Santarremigia's picture
Francisco Santa...,
11/02/2019 12:51

The contribution of this deliverable 4.6 additionally to contributions of the deliverable 4.5 is mainly the introduction of outcomes from the 219 surveyed people. These outcomes should be presented by each pilot and not only for the Greece one.

Total votes: 1
Ioannis Chrysakis's picture
Ioannis Chrysakis,
04/02/2019 14:57

Intro.

  • This deliverable is focused on pilot evaluation which is performed mainly by employing a user questionnaire.
  • It would be interesting as long as the number of evaluated users seems to be not as big enough (219), at least to be mentioned some details regarding the background of them (i.e. education level, ICT skills etc.). Since the project targets different kind of users (i.s decision-makers, policy-makers, I think that it makes sense to validate its maximal outreach on these different user types.

Section 2.

  • In this section six pilot projects links are mentioned but a small description of them does not exist. Furthermore, it would be better for each project to have a "how to use it" documentation section. These two suggestions would help the reader to the direction of understanding better how the co-creation is achieved in practice by using the OGI toolkit.
  • Also in Figure 1 there is no connection with the OGI toolkit. Is it WP3?

Section 3.

  • Although the evaluation methodology is based on four stakeholder perspectives (developers,end-users,decision-makers,policy makers), this scheme is not directly clearly mapped to the figure 2.
  • Furthermore, the four evaluation dimensions in figure 4 could not be easily extracted. Which ones are they?
  • On the other hand, the high level processes of each pilot plan is clearly addressed in figure 3. It contains a structured plan and well defined actions for all the pilots.
  • There are some typos on p.23.
  • Tables 3 and 4 are very informative and give to the reader the big picture of evaluation metrics and their respective results. The project has adapted the Matheus and Janseen Transparency Model that seems to fit well its needs.-From this deliverable we could easily extract that there is a big effort on reducing the administrative burden. To this direction a Cost Benefit Approach is being followed.
  • However, it could be better in Table 6 if we could have seen the different stakeholders in some of the appeared dimensions.
  • In my view, there is a need to redesign Fig.8 because it does not denote clearly the starting and ending point of the workflow.
  • Α informative list of the tools that have been used by the Pilots is listed in Table 9.
  • The software evaluation is based on ISO 25010 and ISO25012. In this deliverable ISO 25010 is well analysed in different parameters and respective measurement variables helping this way to the desired project evaluation.

Section 4

  • There is a reference problem on page 45.
  •  Ι did not understand what Figure 14 depicts.
  • The evaluation method seems well designed and the results are very promising. More focus is given on the problem on data integrity and data quality and I totally agree with this consideration.
  • I also found very interesting the approach of applying open data principles (page 54).
  • Also,the evaluation results are very positive as long as the negative reactions varying from 5-11 percent.
  • However I have an objection regarding the fact that only the results for Pilot 1 are presented. At least a justification for omitting the results of rest pilots could be given. I understand that a potential explanation for this maybe the lack of space, but personally I would prefer to have a synopsis of pilot evaluation and a comparison among the pilots instead of presenting the questionnaires per pilot in the Annexes. To go on step further, in my opinion this comparison could give more insights and could also lead to further knowledge and conclusions regarding the final project outcome.

Conclusion.
In general it seems that the evaluation of pilots gave satisfying results. It would be nice if in this deliverable could be reported how to reuse similar co-creation methods and the implemented framework and ICT toolkit for further improvement on other public services. This could clearly prove that the applied methodology and the implementation could be treated as a generic, extensible and scalable solution.

Total votes: 1
Antonio OPROMOLLA's picture
Antonio OPROMOLLA,
04/02/2019 08:21

For the considerations about the methodology for the pilot evaluation, see the comments of the deliverable 4.5.

The paragraph 4 of the deliverable offers useful information about the evaluation pilot. The results are opportunely shown, even information about the methodological approach miss (how precisely these tests are carried out), in some parts it appears confusing (for example the division in the different subparagraphs does not allow to better understand the contents) and it seems that some parts are missing. Moreover, only the results of the Greek pilot are shown, the other pilots are missing. 

Total votes: 0
Adriano Noli's picture
Adriano Noli,
29/01/2019 13:00

This report is very well written, giving a clear and easy-to-read flow of information on the whole process. An impressive set of bibliographic references gives accountability to the framework, while the good balance between qualitatitive and quantitative data gives credit to the outcomes. Perspectives of different stakeholders are taken into account, enriching the conclusions. 

Total votes: 0