Page tree
Skip to end of metadata
Go to start of metadata

1. eHealth DSI - Monitoring and Reporting framework

The objective of this document is to present a proposal on how to specialise the common set of Key Performance Indicators (KPIs) for progress and performance monitoring defined in the CEF Monitoring Framework[1] for the eHealth DSI.

Additional information on the CEF Monitoring Framework will be provided as footnotes as well as on the AP1 appendix.

Issue at stake

(…)The progress and performance of the DSIs is monitored on a quarterly basis using a common set of 11 Key Performance Indicators (KPIs). These KPIs cover the results, the realisation of the CEF Work Programme and the project & financial management of the DSIs.

In order to collect comparable information, the 11 KPIs are common for all CEF DSIs. However, depending on the services offered and the level of maturity of the DSI, certain KPIs may be not applicable for certain DSIs. (…)[2]

Having the 11 common KPIs as baseline, this proposal suggests how to adapt the KPIs to the eHealth services nature and specificities.


[1] CEF Monitoring and Reporting Framework: https://ec.europa.eu/cefdigital/wiki/display/CEFDIGITAL/About+the+CEF+Monitoring

[2] CEF Reporting and Monitoring Guidebook, V1.2, September 2016


Table of Contents



3. eHDSI - KPIs Dashboard

KPI-1: Uptake

Ref: Monitoring Framework, start measuring 2019 in a quarterly basis

Data last update:  

KPI-1UptakeQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)Total (accumulated)TARGET
20191st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)Total (accumulated)
2019
KPI-1.1
Number of Countries with Operational NCPeH 

2

55667
KPI-1.2
Number of transactions between Countries11.48216.19519.800 17363 (draft)64.84020.000
KPI-1.3
Number of ePrescriptions exchanged3.4523.3354307 3.731 (draft)14.8256.000
KPI-1.4
Number of eDispensations exchanged 1.033

1.669

2.1502.007 (draft)6.859 3.000
KPI-1.5
Number of Patient Summaries exchanged00000N/A
KPI-1.6
Number of Operational Services

2

eP [1 eP-A, 1 eP-B]

PS [0 PS-A, 0 PS-B]

5

eP [1 eP-A, 2 eP-B]

PS [1 PS-A, 1 PS-B]

6

eP [1 eP-A, 2 eP-B]

PS [1 PS-A, 2 PS-B]

7

eP [1 eP-A, 2 eP-B]

PS [2 PS-A, 2 PS-B]

716
20202nd year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)Total (accumulated)2020
KPI-1.1
Number of Countries with Operational NCPeH 7




KPI-1.2
Number of transactions between Countries20285




KPI-1.3
Number of ePrescriptions exchanged8651




KPI-1.4
Number of eDispensations exchanged 3809




KPI-1.5
Number of Patient Summaries exchanged





KPI-1.6 (new, for 2021)
Number of Dispensation Discard operations performed------
KPI-1.7








KPI-1.7.1Countries with eP A Services2





KPI-1.7.2Countries with eP B Services2





KPI-1.7.3Countries with PS A Services3





KPI-1.7.4Countries with PS B Services4




KPI-1.8 (new, for 2021)









KPI-1.8.1Total Number and percentage of Pharmacies enabled with eHDSI services (as Country B)------

KPI-1.8.2Total Number and percentage of Hospitals enabled with eHDSI services (as Country B)------

KPI-1.8.3Total Number and percentage of Other Points of Care enabled with eHDSI services (as Country B)------
KPI-1.9 (new, for 2021)








KPI-1.9.1Number of citizens who are potentially
able to benefit from eHDSI services
------

KPI-1.9.2Number of citizens excluding themselves from eHDSI services------
KPI-1.10 (new, for 2021)
Number of distinct citizens across Europe and nationally who have used the ePrescription service------
KPI-1.11 (new, for 2021)
Number of distinct citizens across Europe and nationally who have used the Patient Summary service------

KPI-2: Quality

Ref: Monitoring Framework, start measuring 2017 Q2, Semester bases before entering operation and quarterly basis after entering operation

KPI-2QualityQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2017
Before routine operationsQ1 & Q2 (Jan -Jun)Q3 & Q4 (Jul - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities 




2018
Before routine operationsQ1 & Q2 (Jan - Jun)Q3 & Q4 (Jul - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities


2019
Routine Operations (Y1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE
KPI-2.1Quality of the information provided to National Contact Point staff, through the eHDSI Communities




2020Routine Operations (Y2)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE

KPI-2.1 

(new, for 2021)

Citizens awareness about eHDSI services-----

KPI-2.2

(new, for 2021)

Citizens perception of eHDSI services value in their lives

-----

KPI-2.3

(new, for 2021)

Health Professionals awareness eHDSI services

-----

KPI-2.4

(new, for 2021)

Health Professionals satisfaction from use of eHDSI

-----

KPI-3: Reliability

Ref: Monitoring Framework, start measuring 2019 (after first NCPeHs enter routine Operations) in a quarterly basis

KPI-3.1 to 3.3Reliability (Core Services)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2019
1st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
KPI-3.1Configuration Services uptime (SMP services, in Production environment)

100%

100%100%100%100%
KPI-3.2Terminology Services uptime (CTS services and portal, in Production environment)100%100%100%100%100%
KPI-3.3

Health Professionals satisfaction from use of eHDSI

99.77%





2020
Routine Operations (Q1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE
KPI-3.1Configuration Services uptime (SMP services, in Production environment)




KPI-3.2Terminology Services uptime (CTS services and portal, in Production environment)




KPI-3.4NCPeHReliability (Generic Services )Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
2019
NCPeH1st year of Routine OperationsQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)AVERAGE
KPI-3.4FIFI - NCPeH uptime


99.985%100%100%99.995%
KPI-3.4EEEE - NCPeH uptime
100%100%98.4%99.46%
KPI-3.4HRHR - NCPeH uptime N/A100%99.99%100.00%99.99%
KPI-3.4CZCZ - NCPeH uptime N/A100%99.9%99.9%99.93%
KPI-3.4LULU - NCPeH uptime N/A100%99.9%99.99%99.96%
KPI-3.3






2020

Routine Operations (Q1)Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)AVERAGE








KPI-4: Milestones

Indicator:

  • KPI-4.1: List of proposed milestones available in AP3

Implications:

  • Approved milestones list; Quarterly updates; Start measuring 2016 Q3

Last update:   (further details available at: https://ec.europa.eu/cefdigital/wiki/x/_EQZAg )

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2017

01

Communication Services Go Live - Release 1

2016-01

2016-03

100%Complete

02

Collaboration Services Go Live - Release 1

2016-01

2016-11

100%Complete

03

1st eHDSI OpenNCP Boot Camp

2016-07

2017-01

100%Complete

04

Communication Services Go Live - Release 2

2016-07

2017-03

100%Complete

05

Interoperability Specifications - Release 1

2016-07

2017-03

100%Complete

06

Configuration Services Go Live - Release 1

2016-01

2017-03

100%Complete

07

Terminology Services Go Live - Release 1

2016-01

2017-03

100%Complete

08

NCPeH reference Implementation Operation ready release (V2.5.x)

2016-01

2017-06

100%Complete

09

1st eHealth Interoperability Test marathon at EC

2017-01

2017-09

2018-02

100%

Complete

10

Completion of the Audit for MS Going live in Wave 1

2016-07

2017-10

2019-04

100%

Complete

11

Approval decision for Wave 1 of NCPeH Go Live

2016-07

2017-11

until 2019-06

100%

Complete

12

Wave 1 of NCPeH Go Live

2016-10

2018-02

until 2019-12

100%  

Complete (Delayed)

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2018

13

Communication Services - Release 3

2017-09

2018-06

--Skipped for next wave

14

Collaboration Services – Release 2

2017-09

2018-06

100%Complete

15

Interoperability Specifications – Release 2

2017-09

2018-06

100%Complete

16

Configuration Services – Release 2

2017-09

2018-06

100%Complete

17

Terminology Services  – Release 2

2017-09

2018-06

100%Complete

18

NCPeH reference Implementation Operation ready release (OpenNCP V2.5.3)

2017-09

2018-06

2018-08

100%Complete

19

2nd eHealth Interoperability Test marathon at EC

  • TESTA (June) PREP (October); Formal (January)
2018-01

2018-09

2019-02

100% 

Complete

20

Completion of the Audit for MS Going live in Wave 2

2018-03

2018-10

2018-10

2019-08

2019-12

100% 

Complete

21

Approval decision for Wave 2 of NCPeH Go Live

2018-10

2019-06

2018-11

2019-11

75%

In progress (Delayed)

22

Wave 2 of NCPeH Go Live

2018-10

2019-06

2019-02

2019-12

50% 

In progress (Delayed)

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2019

23

Communication Services Go Live - Release 4

2019-09

2019-06

0%In Progress

24

Collaboration Services Go Live – Release 3

2019-09

2019-06

100%

Complete

25

Interoperability Specifications – Release 3

2018-09

2019-06

100% 

Complete

26

Configuration Services Go Live – Release 3

2018-09

2019-06

0%Skipped for next wave

27

Terminology Services Go Live – Release 3

2018-09

2019-06

20%

Skipped for next wave

28

NCPeH reference Implementation Operation ready release (V4.0.x)

2018-09

2019-09

100%

Complete

29

3rd eHealth Interoperability Test marathon at EC

2019-06

2020-06

100%

Complete

30

Completion of the Audit for MS Going live in Wave 3

2019-09

2020-08

5%

In Progress 

31

Approval decision for Wave 3 of NCPeH Go Live

2020-06

2020-12

0%Not Started

32

Wave 3 of NCPeH Go Live

2020-07

2020-12

0%Not Started

ID

MILESTONE TITLE

START DATE

END DATE

% COMPLETESTATUS

2020 - 4th RELEASE Cycle

33

Communication Services Go Live - Release 5

2019-09

2020-06

0%Not Started

34

Collaboration Services Go Live - Release 4

2019-09

2020-06

0%Not Started

35

Interoperability Specifications - Release 4

2019-09

2020-06

0%Not Started

36

Configuration Services Go Live - Release 4

2019-09

2020-06

0%Not Started

37

Terminology Services Go Live - Release 4

2019-09

2020-06

0%Not Started

38

NCPeH reference Implementation Operation ready release (V3.2.x)

2019-09

2020-03

0%Not Started


KPI-5: Change Management

Description:

  • KPI-5.1: Number of Changes Requested

  • KPI-5.2: Average time for final decision

implications: Quarterly updates; Start measuring 2016 Q3

more info at: https://ec.europa.eu/cefdigital/wiki/x/GkUZAg


KPI-5.1: Number of Changes Requested

Last update:  


YEARQUARTERCP IDTOTAL NUMBER OF NEW CP/sACCUMULATED NUMBER OF CP/s PER YEAR*

ACCUMULATED NUMBER OF CP/s

AS OF TODAY

CP ID - IMPLEMENTATION IN PROGRESSCP ID - IMPLEMENTED

ACCUMULATED NUMBER OF IMPLEMENTED CP/s

PER QUARTER

2016Q3------------
2016Q41, 2, 3, 5, 6, 7, 8, 9, 10, 12101010----
2017Q113,14, 15, 164414----
2017Q2----4----1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 15, 1613
2017Q317, 14151517--13
2017Q4181616171714
2018Q1No new CPs---1614, 18--14
2018Q219, 202218--14, 1816
2018Q3No new CPs--21819--16
2018Q421, 22, 23, 24, 25, 26682419, 21, 23, 24, 25, 26--16
2019Q127, 28, 29, 30442819, 21, 23, 24, 25, 26, 27, 28, 29--16
2019Q2----4--19, 21, 23, 24, 25, 26, 27, 28, 2919, 21, 25, 26, 27, 28, 2923
2019Q331152923, 2423, 2425
2019Q432, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42 11164032, 33, 34, 354026
2020Q1----04032, 33, 34, 35, 36, 37, 38, 41, 42--26
*Accumulated number of CPs per year (target for 2019: 12 - target for 2020: 7)

KPI-5.2: Average time for final decision

Last update:   


CHANGE PROPOSALSTARTENDTIME FOR DECISION*STATUS
CP-eHealthDSI-001 : Add Evidence Emitter Feature

25COMPLETED
CP-eHealthDSI-002 : Add SMP/SML capabilities

17COMPLETED
CP-eHealthDSI-003 : Clarification on the usage of some RFNA optionality

----REJECTED
CP-eHealthDSI-005 : Implementation of a caching mechanism

----REJECTED
CP-eHealthDSI-006 : Adding message signature

----REJECTED
CP-eHealthDSI-007 : Add the eID functionalities

----REJECTED
CP-eHealthDSI-008 : Adoption of computable CDA Template specifications (HL7 DSTU template exchange format)

14COMPLETED (Wave 1)
CP-eHealthDSI-009 : Formal approval of changes agreed in the EXPAND project for the description of medication

14COMPLETED (Wave 1)
CP-eHealthDSI-010 : Formalization in ART DECOR of the CDA templates used for the transmission of the original Patient Summary and ePrescription

14COMPLETED (Wave 1)
CP-eHealthDSI-012 : Remove all references to the epSOS LSP, work packages information, project management information, requirements and PAC, HCER and MRO services

8COMPLETED
CP-eHealthDSI-013 : MVC update for eHDSI Wave 1 Go Live

11COMPLETED (Wave 1)
CP-eHealthDSI-014 : messaging profile and algorithm binding modification

 

 

20.5

re-Submitted,

COMPLETED

CP-eHealthDSI-015 : Update eHealth DSI PS and eP - NCPeH Deployment Guide : Semantic aspects

--
REJECTED
CP-eHealthDSI-016 : Revision of Semantic Services Specification

7COMPLETED (Wave 1)
CP-eHealthDSI-017 : Change eHDSI connectivity channel from IPSec VPN to TESTA

 

 

7COMPLETED (Wave 1)

CP-eHealthDSI-018 : Remove support for multiple prescription items in a single prescription

 

 

14

COMPLETED (Wave 2)

CP-eHealthDSI-019: Include Marketing Authorization Holder (MAH) in the description of the Medicinal Product

 

 

9COMPLETED (Wave 3)
CP-eHealthDSI-020 : Add Clarification of Dispensation Guidelines to ePrescription Functional Requirements

 

 

9REJECTED
CP-eHealthDSI-021: SAML - Make mandatory the Authentication Method used in Country B

 

 

11.5COMPLETED (Wave 3)
CP-eHealthDSI-022: SAML - Update the values for Purpose of Use to ''Treatment''

 

11.5REJECTED
CP-eHealthDSI-023: SAML - Update the values for the Structural Role of the HP

 

11.5COMPLETED (Wave 3)

CP-eHealthDSI-024: Formalize the ''Description'' element in the eP list

 

 

13COMPLETED (Wave 3)

CP-eHealthDSI-025: Allow the use of ''Original Text'' element to clarify the nationally used unit

 

 

9.5

COMPLETED (Wave 3)

CP-eHealthDSI-026: Extend the value set of Units

 

 

9.5COMPLETED (Wave 3)

CP-eHealthDSI-027: Enhance the communication and presentation of ''Known absence'' and ''No information'' for PS required sections

 

 

5COMPLETED (Wave 3)

CP-eHealthDSI-028: Allow Displaying Concepts in English

 

 

9COMPLETED (Wave 3)

CP-eHealthDSI-029: Allow Multiple Code Systems for a single coded data element

 

 

7.5COMPLETED (Wave 3)

CP-eHealthDSI-030: Confirmation of dispensation by Country A

 

 

7.5REJECTED
CP-eHealthDSI-031: Representation of fractions for the Strength of Medicinal Product

 

 

9COMPLETED (Wave 3)
CP-eHealthDSI-032_Include ICD-10 5 character codes

 

 

10Implementation in progress (Wave 4)
CP-eHealthDSI-033_Enable indicating dosage period length

 

10Implementation in progress (Wave 4)
CP-eHealthDSI-034_Use of ATC based Value Set for Vaccines

 

10Implementation in progress (Wave 4)
CP-eHealthDSI-035_Set as mandatory Nature of Patient Summary

 

10Implementation in progress (Wave 4)
CP-eHealthDSI-036: Enable dispensation discard operation

 

11.5Implementation in progress (Wave 4)
CP-eHealthDSI-037: Update project references to eHDSI in CDA IG and MVC

11.5Implementation in progress (Wave 4)
CP-eHealthDSI-038: Duplicate OID issue

11.5Implementation in progress (Wave 4)
CP-eHealthDSI-039: Next of kin

--Addressed to eHealth Network 
CP-eHealthDSI-040: MVC Hotfix

 

6COMPLETED (Wave 3 and Routine Operations)
CP-eHealthDSI-041: Relax mandatory constraints to external templates (IHE PCC and CCD)

11Implementation in progress (Wave 4)
CP-eHealthDSI-042: Implement International Search Mask

9Implementation in progress (Wave 4)

*Time expressed in weeks, for approval or rejection decision (target for 2019 and 2020: < 12 weeks)


Average time per quarter

Last update:   


YearQuarterChange Proposals ID**AVERAGE TIME FOR DECISION*
2017Q12, 8, 9, 10, 12, 1313
2017Q21, 1616
2017Q3177
2017Q4--
2018Q114, 1817.25
2018Q219, 209
2018Q3

-

-
2018Q421, 22, 2311.5
2019Q124, 25, 26, 279.25
2019Q228, 29, 308
2019Q3319
2019Q432, 33, 34, 35, 409.2
2020Q136, 37, 38, 41, 42 10.9

*Time expressed in weeks, for approval or rejection decision (target for 2019 and 2020: < 12 weeks)

**The CPs which fall in between 2 quarters are considered as part of the quarter when the decision was made.

4. eHDSI CEF BB - Reuse Monitoring

Status of the CEF Building Blocks reuse in eHealth DSI.

Legend: 

Possible status of reuseConditions
No commitmentNo commitment to perform an analyses to evaluate the potential of reuse.
Commitment to analyseAnalysis in progress.
Commitment to reuseIf the analysis' outcome is positive for the reuse
Not going to reuseIf the analysis' outcome is not positive for the reuse
ReuseIf the integration has been completed and the system is operational
Reuse (partial)Integration of some components has been completed and the system is operational


Last update:  

YearReuse MonitoringQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)
2016
CEF Building Block
Q1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sept)Q4 (Oct - Dec)
Preparation (year 1)


eDelivery



Commitment to reuseCommitment to reuse
eSignature

Commitment to analyseCommitment to analyse
eID

Commitment to analyseCommitment to analyse
eTranslation

No commitmentNo commitment
eTranslation

No commitmentNo commitment
2017
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Preparation (year 2)


eDelivery

Commitment to reuse

Commitment to reuseCommitment to reuseCommitment to reuse
eSignatureCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eIDCommitment to analyseCommitment to analyseCommitment to reuseCommitment to reuse
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2018
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Preparation (year 3)



eDelivery

Commitment to reuse

Commitment to reuseCommitment to reuseCommitment to reuse
eSignatureCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eIDCommitment to analyseCommitment to analyseCommitment to analyseCommitment to analyse
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2019
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Year 1 of Routine Operations




eDelivery

Reuse Partial [SMP, PKI]

Commitment to analyse (AS4)

Reuse Partial [SMP, PKI]

Commitment to analyse (AS4)



eSignatureCommitment to analyseCommitment to analyse

eIDCommitment to analyseCommitment to analyse

eTranslationNo commitmentNo commitmentNo commitmentNo commitment
eTranslationNo commitmentNo commitmentNo commitmentNo commitment
2020
CEF Building BlockQ1 (Jan - Mar)Q2 (Apr - Jun)Q3 (Jul - Sep)Q4 (Oct - Dec)
Year 2 of Routine Operations




eDelivery





eSignature



eID



eTranslation



eTranslation









  • No labels

3 Comments

  1. In version 1.4 of the document (current, adopted) KPI 1.5 still refers to eP/eD documents instead of PS documents... (rationale, aimed usage) on page 16.

    1. Hello Stephane SPAHNI

      Thank your for the feedback.

      I will check it and provide an update ASAP.

      i will keep you aware.

      Wish you a great week.

  2. The root cause for unavailability is part of the dataset for KPI 3.4. Are we going to set a catalogue for this causes of unavailability or using free text for this field?.