Discussions

How to set KPIs that work

  • Tsvetanka Kalfin profile
    Tsvetanka Kalfin
    14 October 2015 - updated 4 years ago
  • 10 comments
    Total votes: 3

The European Commission adopted the Horizon 2020 Work Programme 2016 – 2017 European Research Infrastructures (including e-Infrastructures).

The programme specifies that "the proposals should define, collect and make available key performance indicators (KPIs) in support of operational, technical and socio-economic impact assessment".

For evaluation of proposals submitted under EINFRA-11-2016 and EINFRA-12-2017, KPIs are defined as part of the evaluation criterion Excellence. In addition to the standard sub-criteria, the following aspects will also be taken into account:

"The extent to which the Service Activities (Trans-national and/or Virtual Access Activities) will offer access to state-of-the-art infrastructures and high quality services, and will enable users to conduct excellent research as demonstrated, inter-alia, by both the quality of the associated catalogue of services and the baseline and expected future measurements of related KPIs ".

The Work programme also defines a new role of e-IRG - to collect and aggregate relevant policy information with impact on the development of EC-funded e-infrastructures, including KPIs and with the support of independent expert panels to provide strategic advice on the evolution of the European e-infrastructures landscape.

The service related KPIs collected by e-IRG will then be made publicly available through a web-portal. It shall support monitoring activities on take-up of open science and e-infrastructures policies.

In order to facilitate the process of proposal preparation we would like to open the debate on KPIs with some preliminary questions to be explored:

  • What are the KPIs defined in your project;

  • What was the reason for selecting them;

  • Do you prioritize your KPIs;

  • Which KPIs you find useful and what are the challenges to be addressed.

Tags: 
KPIeInfrastructures

Comments

Log in to post comments
Rossen Apostolov's picture
Rossen Apostolov,
10/02/2016 19:24

Below is a list of KPIs that BioExcel has identified:

• Performance and scaling improvements of the pilot codes
• Availability of pilot codes on European HPC/HTC systems - nationally, in PRACE, and internationally
• Usage of Workflows
• Number of runs submitted via web portals portals (% increase with respect to previous year)
• Number of academic and commercial institutions (both industry and SMEs) joining the user groups
• Number of projects developed with academic and commercial partners
• Number of organized training events
• Number of people trained (academic and industrial) 

• Number and severity of bugs found and fixed through targeted testing on workflows important to BioExcel user groups

Some important aspects of impact e.g. adoption of cutting-edge accelerated software, adoption of best practices for efficient usage, success of collaborative projects etc. might well take longer than the project's lifetime.

Total votes: 0
J-L Dorel's picture
J-L Dorel,
07/12/2015 17:09

Clear expectations and communication on requirements are essential when it comes to ease integration of innovative projects into integration projects.
For example, an operator may pay special attention to robustness & scalability when it comes to integrating new services while an innovative projects may put attention on new ideas and solutions that will require an open environment to be deployed. As a formal requirement of WP16-17, Key Performance Indicators can be one possible instrument for facilitating integration as a clear measurement of expectations from both side:
- What KPIs of interest for operators could be measured by innovative projects?
- What KPIs of interest for innovative projects could be measured by operators?
Are some of the questions that may arise within the context of the KPIs discussion.

Total votes: 1
judit gimenez's picture
judit gimenez,
24/11/2015 15:59

When writing the POP proposal we identified a list of KPIs covering different aspects, we tried to keep the list small so we focus on what we consider the most relevant metrics (some kind of prioritization).

- for the entire project: customer savings by the identified improvements, ROI, awareness of the application performance issues, customer satisfaction

- for the community development: size of the POP community

- for the technical work: number of codes served and some more detailed metrics of the percentage of improvement like N times faster codes

- for the dissemination: POP awareness levels among European community

We do not have yet experience on the usefulness of these KPI but there is a clear benefit being able to quantify how far we are from our target.

judit

Total votes: 1
Chris Morris's picture
Chris Morris,
23/11/2015 12:20

Here are some of our KPIs.

  • Some are binary, e.g. UI and Processing jobs use virtual folder mechanism.
  • Some are usage counts, e.g. Installations of repository, Visits to provenance reports, Number of jobs using new functionalities, Number of publications resulting from use of the infrastructure, Number of registered users.
  • Some measure resource consumption: CPU hours made available via the VRE.
  • Metrics that really represent the scientific goals of the VRE are more complicated, but still feasible: To follow discipline-hopping, we can monitor users who access portals for different structural biology techniques, accesses by researchers other than the original depositor, Structures solved using the infrastructure, by combining experimental data from different techniques, Jobs run by industrial user, Joint publications/position papers involving authors from infrastructures beyond the current partnership.
Total votes: 1
Donatella Castelli's picture
Donatella Castelli,
17/11/2015 18:06

Let me contribute to this discussion by reporting a slightly rephrased and abbreviated version of the current BlueBRIDGE project (www.bluebridge-vres.eu) KPIs. The change with respect to the original ones has been done to make them understandable also to those that are not familiar with the specific project objectives.

The following project description should (hopefully) suffice to understand the rest:
BlueBRIDGE is funded under the call H2020-EINFRA-2015-1 e-Infrastructures for virtual research environments (VRE). It develops a number of VREs supported by the D4Science infrastructure (www.d4science.org) for providing a more solid ground for informed advice to competent authorities and to enlarge the spectrum of educational and growth opportunities as addressed by the Blue Growth Societal Challenge.

1. What are the KPIs defined in your project

Infrastructure related operational KPIs:

• Number of operated VREs, i.e. web-based application environments each tailored to the specific working needs of a community of actors sharing a common goal;
• Number of interfaced e-Infrastructures /served e-Infrastructures;
• Number of exploited third-party data repositories and service providers;
• Number of datasets, algorithms and tools accessible through the infrastructure;
• Availability: per VRE and per each of the provided service.

Technology related KPIs
• Number of new/enhanced services and libraries;
• Number of software releases issued;
• Average incident resolution time;
• Average ticket request closing time.

Socio-economic related KPIs

Across-VREs KPIs

• Number of organizations (both internal and external to the project) sustaining the VRE operation and the development of new products through in kind-contribution & co-funding;
• Number of users exploiting the VREs on a regular basis [here we can distinguish between “direct users” of the VRE and “indirect users”, i.e. users of third-party services using the VRE];
• Number of organization and SMEs making available their resources through the infrastructure;
• Distribution of the users exploiting the VREs on a regular basis: number of countries, number of international organizations, SMEs, and academic institutions.

VRE domains specific KPIs

• Number of VRE-specific tasks executed;
• Number of access to the data products generated through the VRE;
• Number of discussion threads activated in a VRE (BlueBRIDGE VREs, as any D4Science enabled VRE, are provided with social tools);
• Number of activities facilitated through the VRE usage, e.g. number of VRE-enabled academic courses, number of students trained;
• Number of SMEs providing resources to and using the VRE;
• Number of planned outcomes delivered, e.g. aquaculture farms performance indicators, thematic regional maps, stocks and fisheries described with a unique id.

Communication & Dissemination related KPIs:

• Stakeholder community addressed communication: number of posts in social networks, press-releases, newsletters, organised events, presentations, qualified community contacts with whom the project engages;
• Scientific communication: number of publications [with (alt-)metrics - including data & sw publications – with a specification of how many of them are open access publications]
• Number of SME representatives trained
• Number of innovation focused webinars on BlueBRIDGE exploitable results

Project management related KPIs

• Timeliness in deliverables and milestones submission;
• Timeliness in internal reporting;
• Number of management meetings (with respect to the initially planned ones);
• Usage of the project social tools for internal communication;
• Number of new synergic contact established.

2. What was the reason for selecting them
The KPIs reported are those included in the BlueBRIDGE project proposal. They are based on the KPIs established for past projects and on the specific BlueBRIDGE objectives. The Project Steering Board will analyse them quarterly in order to identify risks and plan corrective actions. We plan to refine them in the course of the project if we realize that they insufficient to provide a complete picture of the project outcomes.

3. Do you prioritize your KPIs
For the time being we have not yet planned any priorization. The priorization is implicitly expressed by the planned values for the different KPIs.

4.Which KPIs you find useful and what are the challenges to be addressed
As written above, we have mainly selected KPIs that we found useful in past projects. One of the challenges is to monitor these KPIs. In the BlueBRIDGE project appropriate project procedures and tools have been set up to support this monitoring.

Total votes: 1
Tsvetanka Kalfin's picture
Tsvetanka Kalfin,
13/11/2015 17:06

First of all I would like to thank to all who have already contributed to the discussion on the role and definition of KPIs. As correctly mentioned in the comments the KPIs suggested by the projects will be used not only to measure the socio-economic, but also the operational and technical impact.
The KPIs are not a novelty per se, the currently running H2020 projects have defined their respective KPIs. What is new is the weight given to KPIs in the Work Programme 2016-2017. What will be taken into consideration during the evaluation process will not be the KPIs, but the impact of the proposal and how it will be assessed.
Considering some comments on the role of the EC, presently neither the Commission, nor the board will intervene beyond facilitating the discussion or synthesizing the information provided. Any recommendations in that respect are welcome.
Therefore I would like to encourage you to publish KPIs defined in your project as well as your opinion which KPIs you would consider useful and why.

Total votes: 1
Andrea Manieri's picture
Andrea Manieri,
17/11/2015 19:17

Dear Colleagues,
Last week at EGI Community Forum I had the possibility to discuss with some of you about the need for a guidance in defining KPIs for the projects and how to assess their Impact.

Luckily, I get an agreement from experts in past Impact Assessment projects (namely, eNventory, eFiscal, ERINAplus - thanks to all, indeed!) to help the community in shaping their own KPIs for current and future projects.

If you (and your project/proposal consortia) are interested to have a full day operational workshop devoted to:
- understand what's impact assessment is about;
- understand the various methodologies and perspectives for defining KPIs;
- understand how to set up the related parts of the next calls proposal templates;
- exercise yourself in selecting your KPIs with the help of experts; ...

.. then post your availability in this thread, indicating your personal or project/proposal interest, the number of potential participants from your consortia, the suitability of mid Jan and Bruxelles for the timing and location of the workshop.

If we reach 8 to 10 project/proposal (max 20 people for being effective) we will set-up an Open Workhop, on mid January, in Bruxelles, possibly in EC premises, to address all the aspects of this theme.

Hope this help in shaping a more impacting eInfrastructure community,

Best

Andrea Manieri

Total votes: 1
Andrea Manieri's picture
Andrea Manieri,
08/11/2015 16:09

Hi,

in the recent past, a number of experiences in the eInfra context (eFiscal, dFiscal, Erinaplus, Erina4africa, ERINA study) as well as in other domains (Sequoia in former Software & Services unit, Maxculture in cultural Heritage unit, EVARIO in Research Infras, etc) addressed the issue of identify the most suitable set of indicators for assessing the (socio-economic) impact of eInfrastructures and related projects.

Two lessons have been learnt from there:
- consortia need to be well aware of what impact assessment is and how to apply generic framework to their own initiative (the "one-not-fit-for-all" issue), identify right competences in their partnership and consider this activity in its plan since the very beginning;
- Impact assessment framework are of two nature (sometime strictly interrelated): qualitative and quantitative. Qualitative one are easy to apply but provide less evidence and cannot be measured, while quantitative are complex but verifieable. Erinaplus and eFiscal, demonstrated that quantititative approach can be suitable, provided that quantitative information are collected since the beginning.

New proposal should consider such past experience and learn how to implement their own KPIs framework.

EC should support this reflection, possibly with a focused actions and/or workshop.

Ready to support if needed,

A.

Total votes: 1
Stephanie MATT's picture
Stephanie MATT,
15/10/2015 11:12

Please check input of Davide Salomoni on https://ec.europa.eu/futurium/en/comment/4940#comment-4940 discussion.

Total votes: 3
Bob JONES's picture
Bob JONES,
12/11/2015 15:35

I would like to start by thanking Davide Salamoni for his clear input on KPIs and congratulate the INDIGO-DataCloud for publishing their work on KPIs.

I understand that the e-infrastructure projects to be funded in theme 1 (Integration & consolidation of e-infrastructure platforms) of the 2016-2017 work programme will be required to actively collaborate, publish their services in a common catalogue and measure a set of Key Performance Indicators (KPIs). In the presentation given by Augusto Burgueno-Arjona at the start of the 11th e-Concertation meeting for e-Infrastructures (Brussels, 9th November 2015) he said the KPIs are intended to provide an operational, technical and socio-economic impact assessment of each service and will be defined and measured by the service providers.
Given the potential importance of these KPIs, I think they need to be discussed with the user communities so that they are convinced they will be relevant for the end-users and their funding agencies. Let me give you an example: as explained by Davide, INDIGO-DataCloud is publishing KPIs related to the software development process. These are important for the project’s work programme but are perhaps less relevant for the user community. Users will be more concerned with KPIs that measure aspects such as service availability, reliability and percentage of service requests resolved within an agreed-upon period of time.
The socio-economic impact KPIs should include measures of service utilisation and cost-effectiveness. I would suggest that the ‘external board’ to be nominated by the EC should be given the role of endorsing an agreed set of KPIs and the manner in which they are measured for each class of services across all projects participating in theme 1. Practically, this set of agreed KPIs should be discussed between the projects and the user communities then presented to the ‘external board’ for endorsement when it convenes to assess the collaboration agreements. Such KPIs should be inspired by recognised international standards for IT service operation (such as ITIL).

Total votes: 1