Page tree
Skip to end of metadata
Go to start of metadata

Table of Contents


1. eHealth DSI - Monitoring and Reporting framework

The objective of this document is to present a proposal on how to specialise the common set of Key Performance Indicators (KPIs) for progress and performance monitoring defined in the CEF Monitoring Framework[1] for the eHealth DSI.

Additional information on the CEF Monitoring Framework will be provided as footnotes as well as on the AP1 appendix.

Issue at stake

(…)The progress and performance of the DSIs is monitored on a quarterly basis using a common set of 11 Key Performance Indicators (KPIs). These KPIs cover the results, the realisation of the CEF Work Programme and the project & financial management of the DSIs.

In order to collect comparable information, the 11 KPIs are common for all CEF DSIs. However, depending on the services offered and the level of maturity of the DSI, certain KPIs may be not applicable for certain DSIs. (…)[2]

Having the 11 common KPIs as baseline, this proposal suggests how to adapt the KPIs to the eHealth services nature and specificities.



[1] CEF Monitoring and Reporting Framework: https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/About+the+CEF+Monitoring

[2] CEF Reporting and Monitoring Guidebook, V1.2, September 2016



2. eHealth DSI - KPIs


eHDSI - Applicable

V1.4 Adopted during the eHMSEG



Previous Versions



3. eHDSI - KPIs Dashboard

KPI-1: Uptake

Ref: Monitoring Framework, start measuring 2019 in a quarterly basis

Data last update:  

KPI-1UptakeQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)Total (accumulated)
20191st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)Total (accumulated)
KPI-1.1Number of Countries with Operational NCPeH 

2

5

5
KPI-1.2Number of transactions between Countries11.48216.195 (draft)

27.677
KPI-1.3Number of ePrescriptions exchanged3.4523.335 (draft)

6.787
KPI-1.4Number of eDispensations exchanged 1.033

1.669 (draft)



2.702
KPI-1.5Number of Patient Summaries exchanged00

0
KPI-1.6Number of Operational Services25

5

KPI-2: Quality

Ref: Monitoring Framework, start measuring 2017 Q2, Semester bases before entering operation and quarterly basis after entering operation

KPI-2QualityQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2017
Before routine operationsQ1 & Q2 (Jan -Jun)Q3 & Q4 (Jul - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities 




2018
Before routine operationsQ1 & Q2 (Jan - Jun)Q3 & Q4 (Jul - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities


2019
Routine Operations (Q1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities




KPI-3: Reliability

Ref: Monitoring Framework, start measuring 2019 (after first NCPeHs enter routine Operations) in a quarterly basis

KPI-3.1 to 3.3Reliability (Core Services)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2019
1st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
KPI-3.1Configuration Services uptime (SMP services, in Production environment)






KPI-3.2Terminology Services uptime (CTS services and portal, in Production environment)




KPI-3.3Collaboration Services uptime (Confluence and Jira)




2020
Routine Operations (Q1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE







KPI-3.4NCPeHReliability (Generic Services )Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2019
NCPeH1st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
KPI-3.4FIFI - NCPeH uptime


99.985%


KPI-3.4EEEE - NCPeH uptime
100%


KPI-3.4HRHR - NCPeH uptime N/A100%


KPI-3.4CZCZ - NCPeH uptime N/A100%


KPI-3.4LULU - NCPeH uptime N/A100%


2020

Routine Operations (Q1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE








KPI-4: Milestones

Indicator:

  • KPI-4.1: List of proposed milestones available in AP3

Implications:

  • Approved milestones list; Quarterly updates; Start measuring 2016 Q3

Last update:   (further details available at: https://ec.europa.eu/cefdigital/wiki/x/_EQZAg )

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2017

01

Communication Services Go Live - Release 1

2016-01

2016-03

100%Complete

02

Collaboration Services Go Live - Release 1

2016-01

2016-11

100%Complete

03

1st eHDSI OpenNCP Boot Camp

2016-07

2017-01

100%Complete

04

Communication Services Go Live - Release 2

2016-07

2017-03

100%Complete

05

Interoperability Specifications - Release 1

2016-07

2017-03

100%Complete

06

Configuration Services Go Live - Release 1

2016-01

2017-03

100%Complete

07

Terminology Services Go Live - Release 1

2016-01

2017-03

100%Complete

08

NCPeH reference Implementation Operation ready release (V2.5.x)

2016-01

2017-06

100%Complete

09

1st eHealth Interoperability Test marathon at EC

2017-01

2017-09

2018-02

100%

Complete

10

Completion of the Audit for MS Going live in Wave 1

2016-07

2017-10

2019-04

100%

Complete

11

Approval decision for Wave 1 of NCPeH Go Live

2016-07

2017-11

until 2019-06

100%

Complete

12

Wave 1 of NCPeH Go Live

2016-10

2018-02

until 2019-06

72%

In progress (Delayed)

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2018

13

Communication Services - Release 3

2017-09

2018-06

--Skipped for next wave

14

Collaboration Services – Release 2

2017-09

2018-06

100%Complete

15

Interoperability Specifications – Release 2

2017-09

2018-06

100%Complete

16

Configuration Services – Release 2

2017-09

2018-06

100%Complete

17

Terminology Services  – Release 2

2017-09

2018-06

100%Complete

18

NCPeH reference Implementation Operation ready release (OpenNCP V2.5.3)

2017-09

2018-06

2018-08

100%Complete

19

2nd eHealth Interoperability Test marathon at EC

  • TESTA (June) PREP (October); Formal (January)
2018-01

2018-09

2019-02

100% 

Complete

20

Completion of the Audit for MS Going live in Wave 2

2018-03

2018-10

2018-10

2019-08

33%

In progress

21

Approval decision for Wave 2 of NCPeH Go Live

2018-10

2019-06

2018-11

2019-11

16%

In progress

22

Wave 2 of NCPeH Go Live

2018-10

2019-06

2019-02

2019-12

0%Not Started

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2019

23

Communication Services Go Live - Release 4

2019-09

2019-06

0%In Progress

24

Collaboration Services Go Live – Release 3

2019-09

2019-06

20%

In Progress

25

Interoperability Specifications – Release 3

2018-09

2019-06

20%

In Progress

26

Configuration Services Go Live – Release 3

2018-09

2019-06

0%Skipped for next wave

27

Terminology Services Go Live – Release 3

2018-09

2019-06

20%

Skipped for next wave

28

NCPeH reference Implementation Operation ready release (V3.1.x)

2018-09

2019-09

0%In Progress

29

3rd eHealth Interoperability Test marathon at EC

2019-06

2020-06

7%

In Progress

30

Completion of the Audit for MS Going live in Wave 3

2019-09

2020-08

0%Not Started

31

Approval decision for Wave 3 of NCPeH Go Live

2020-06

2020-12

0%Not Started

32

Wave 3 of NCPeH Go Live

2020-07

2020-12

0%Not Started

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

4th RELEASE Cycle

33

Communication Services Go Live - Release 5

2019-09

2020-06

0%Not Started

34

Collaboration Services Go Live - Release 4

2019-09

2020-06

0%Not Started

35

Interoperability Specifications - Release 4

2019-09

2020-06

0%Not Started

36

Configuration Services Go Live - Release 4

2019-09

2020-06

0%Not Started

37

Terminology Services Go Live - Release 4

2019-09

2020-06

0%Not Started

38

NCPeH reference Implementation Operation ready release (V3.2.x)

2019-09

2020-03

0%Not Started


KPI-5: Change Management

Description:

  • KPI-5.1: Number of Changes Requested

  • KPI-5.2: Average time for final decision

implications: Quarterly updates; Start measuring 2016 Q3

more info at: https://ec.europa.eu/cefdigital/wiki/x/GkUZAg


KPI-5.1: Number of Changes Requested

Last update:  


QUARTERCP IDTOTAL NUMBER OF NEW CP/sACCUMULATED NUMBER OF CP/sCP ID - IMPLEMENTATION IN PROGRESSCP ID - IMPLEMENTED
2016 Q3----------
2016 Q41, 2, 3, 5, 6, 7, 8, 9, 10, 121010----
2017 Q113,14, 15, 16414----
2017 Q2--------TOTAL13: 1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16
2017 Q317, 1411517TOTAL13: --
2017 Q41811617TOTAL14: 17
2018 Q1No new CPs--1614, 18TOTAL14: --
2018 Q219, 20218--TOTAL16: 14, 18
2018 Q3No new CPs--1819TOTAL16: --
2018 Q421, 22, 23, 24, 25, 2662419, 21, 23, 24, 25, 26TOTAL16: --
2019 Q127, 28, 29, 3042819, 21, 23, 24, 25, 26, 27, 28, 29TOTAL16: --
2019 Q2



TOTAL17: 19, 21, 23, 24, 25, 26, 27, 28, 29
2019 Q3




2019 Q4




KPI-5.2: Average time for final decision

Last update:   


CHANGE PROPOSALSTARTENDTIME FOR DECISION*STATUS
CP-eHealthDSI-001 : Add Evidence Emitter Feature

25COMPLETED
CP-eHealthDSI-002 : Add SMP/SML capabilities

17COMPLETED
CP-eHealthDSI-003 : Clarification on the usage of some RFNA optionality

----REJECTED
CP-eHealthDSI-005 : Implementation of a caching mechanism

----REJECTED
CP-eHealthDSI-006 : Adding message signature

----REJECTED
CP-eHealthDSI-007 : Add the eID functionalities

----REJECTED
CP-eHealthDSI-008 : Adoption of computable CDA Template specifications (HL7 DSTU template exchange format)

14COMPLETED
CP-eHealthDSI-009 : Formal approval of changes agreed in the EXPAND project for the description of medication

14COMPLETED
CP-eHealthDSI-010 : Formalization in ART DECOR of the CDA templates used for the transmission of the original Patient Summary and ePrescription

14COMPLETED
CP-eHealthDSI-012 : Remove all references to the epSOS LSP, work packages information, project management information, requirements and PAC, HCER and MRO services

8COMPLETED
CP-eHealthDSI-013 : MVC update for eHDSI Wave 1 Go Live

11COMPLETED
CP-eHealthDSI-014 : messaging profile and algorithm binding modification

 

 

20.5

re-Submitted,

COMPLETED

CP-eHealthDSI-015 : Update eHealth DSI PS and eP - NCPeH Deployment Guide : Semantic aspects

--
REJECTED
CP-eHealthDSI-016 : Revision of Semantic Services Specification

7COMPLETED
CP-eHealthDSI-017 : Change eHDSI connectivity channel from IPSec VPN to TESTA

 

 

7Implementation in progress

CP-eHealthDSI-018 : Remove support for multiple prescription items in a single prescription

 

 

14

COMPLETED

CP-eHealthDSI-019: Include Marketing Authorization Holder (MAH) in the description of the Medicinal Product

 

 

9Implementation in progress
CP-eHealthDSI-020 : Add Clarification of Dispensation Guidelines to ePrescription Functional Requirements

 

 

9REJECTED
CP-eHealthDSI-021: SAML - Make mandatory the Authentication Method used in Country B

 

 

11.5Implementation in progress
CP-eHealthDSI-022: SAML - Update the values for Purpose of Use to ''Treatment''

 

11.5REJECTED
CP-eHealthDSI-023: SAML - Update the values for the Structural Role of the HP

 

11.5Implementation in progress

CP-eHealthDSI-024: Formalize the ''Description'' element in the eP list

 

 

13Implementation in progress

CP-eHealthDSI-025: Allow the use of ''Original Text'' element to clarify the nationally used unit

 

 

9.5Implementation in progress

CP-eHealthDSI-026: Extend the value set of Units

 

 

9.5Implementation in progress

CP-eHealthDSI-027: Enhance the communication and presentation of ''Known absence'' and ''No information'' for PS required sections

 

 

5Implementation in progress

CP-eHealthDSI-028: Allow Displaying Concepts in English

 

 

9Implementation in progress

CP-eHealthDSI-029: Allow Multiple Code Systems for a single coded data element

 

 

7.5Implementation in progress

CP-eHealthDSI-030: Confirmation of dispensation by Country A

 

 

7.5REJECTED










*Time expressed in weeks (average foreseen 12 weeks), for approval or rejection decision

4. eHDSI CEF BB - Reuse Monitoring

Status of the CEF Building Blocks reuse in eHealth DSI.

Legend: 

Possible status of reuseConditions
No commitmentNo commitment to perform an analyses to evaluate the potential of reuse.
Commitment to analyseAnalysis in progress.
Commitment to reuseIf the analysis' outcome is positive for the reuse
Not going to reuseIf the analysis' outcome is not positive for the reuse
ReuseIf the integration has been completed and the system is operational
Reuse (partial)Integration of some components has been completed and the system is operational


Last update:  

YearReuse MonitoringQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)
2016
CEF Building Block
Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)
Preparation (year 1)


eDelivery



Commitment to reuseCommitment to reuse
eSignature

Commitment to analyseCommitment to analyse
eID

Commitment to analyseCommitment to analyse
eTranslation

No commitmentNo commitment
eTranslation

No commitmentNo commitment
2017
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Preparation (year 2)


eDelivery

Commitment to reuse

Commitment to reuseCommitment to reuseCommitment to reuse
eSignatureCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eIDCommitment to analyseCommitment to analyseCommitment to reuseCommitment to reuse
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2018
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Preparation (year 3)



eDelivery

Commitment to reuse

Commitment to reuseCommitment to reuseCommitment to reuse
eSignatureCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eIDCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2019
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Year 1 of Routine Operations




eDelivery

Reuse Partial [SMP, PKI]

Commitment to analyse (AS4)

Reuse Partial [SMP, PKI]

Commitment to analyse (AS4)



eSignatureCommitment to analyseCommitment to analyse

eIDCommitment to analyseCommitment to analyse

eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2020
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Year 2 of Routine Operations




eDelivery





eSignature



eID



eTranslation



eTranslation









  • No labels

2 Comments

  1. In version 1.4 of the document (current, adopted) KPI 1.5 still refers to eP/eD documents instead of PS documents... (rationale, aimed usage) on page 16.

    1. Hello Stephane SPAHNI

      Thank your for the feedback.

      I will check it and provide an update ASAP.

      i will keep you aware.

      Wish you a great week.