The aim of the WP is to develop functional production prototypes including setting up procedures and developing technical solutions, to promote and support the collection, processing and analysis of smart meter data for statistical production in the participating NSIs related to statistics on measuring electricity consumption and identifying energy consumption patterns.
To achieve the aim the WP will be relaying on the work done during the previous ESSnet project. The participating NSIs will develop and test detailed procedures and techniques for producing functional prototypes. Where possible the results will be implemented as statistical products. Subject matter experts of the relevant statistical production units of the NSIs will be involved in all phases of the work.
During the project the implementation procedures for the following statistical products will be produced:
- electricity statistics of businesses (EE, DK, SE),
- electricity statistics of households (DK, SE)
- identifying vacant or seasonally vacant dwelling (EE, NO)
The results will be published on the wiki page of the ESSnet as a report. In the report the WP will describe in details what are the infrastructure requirements and steps needed to implement the new data source for production of statistics. The report will be updated in three phases as more knowledge is acquired during the project.
The work is based on the results of the previous ESSnet and results described in Reports 1 – 4 will be used as a starting point to develop functional production prototypes. The implementation phase will be relaying in the tasks - data access, data handling, methodology and techniques and future perspectives - completed during the ESSnet big data and the result of the tasks will be extended to be fitted in the big data modified Generic Statistical Business Process Model (GSBPM). The following results of the ESSnet will be taken into consideration:
- Data access – Description of the current status and future perspectives regarding the availability of smart meters in the partner countries. Overview of data holders/owners in the partner countries. Assessment of legal aspects regarding the access to the data and linking data to other data sources. Technological requirements (software and hardware) needed to deal with this type of data.
- Data handling – Description of the available data (structure, records, attributes). Assessment of the current coverage (households and businesses) and expected coverage in the near future. Processing raw data and transforming it into cleaned datasets to be used in the next tasks.
- Methodology and techniques
- => Linking smart meters data with administrative data – Description of the linking process to different administrative sources. For each data source the following information is given: data source description, linking methodology, problems encountered, linking rate and other quality information. Overview of IT-infrastructure used in the linking process, recommendations based on the experiences.
- => Methodology to produce electricity consumption statistics regarding businesses - Description of the procedure of processing the data from raw data to processed data to final estimates. If feasible, combining smart meters data with survey data to improve survey estimates is studied. Quality assessment of the processed data (processing and measurement errors) and final estimates (validity of the concepts, timeliness, comparability, coherence) is carried out.
- => Methodology to produce electricity consumption statistics regarding households - Description of the procedure of processing the data from raw data to processed data to final estimates. Quality assessment of the processed data (processing and measurement errors) and final estimates (validity of the concepts, timeliness, comparability, coherence).
- => Methodology to produce estimates of seasonally vacant living spaces – Description of the procedure of processing the data from raw data to processed data to final estimates. If feasible, smart meters data in combination with population register data is used to model the person’s main residence indicator.
The results of the ESSnet indicated the possibility to produce statistics from the smart meter data, but the methods and methodologies have to developed further to correspond the requirements of production of official statistics. To broaden the influence and scalability of the planned statistical products the European Commission smart grid standards have to been taken into account. There is a need to describe secure data transfer procedures. Due to the fact that the raw data is different in each country the WP has to define what is the output of preprocessing so that it would be possible to apply the same procedure for the data of each participating country and it would be possible to generate standardized outputs from the smart meter data. This implies the following:
- Developing methodology for geo-spacial linking.
- Defining common quality measurements for linking.
- Developing standard visual outputs for electricity data.
- Validate the methods selected during the ESSNet big data defining requirements for classifying households as vacant or seasonally vacant.
- Other developments to fit the requirements of official statistics.
Description of work
The aim for this WP is to implement smart meter data for producing statistics. This can be as a supplement to other statistics e.g. energy statistics of businesses, census, household costs, tourism seasonality or impact on environment. The implementation will include linking electricity data with other administrative sources for eventually producing statistics of businesses, households and identifying vacant living places or seasonal/temporary occupancy of living places. Work will include the identification of energy consumption patterns in households. Each participating country will produce implementation procedures for statistical products. For conducting the project four face-to-face meetings of participating countries are organised. The first meeting will be the milestone to fix the status of the identification of the data holders and the access to the data. In addition two visits between two countries will be organised for knowledge transfer. At least once a month a WebEx meeting will be organised before the CG meeting.
The following activities will be undertaken:
1. Definition of statistical products
2. Identification of statistical production processes and capabilities that may be affected at national level (covering at least the GSBPM phases Design, Build, Process, Analyse and Dissemination)
3. Definition of a risk plan with mitigation scenarios
4. Definition of a quality management framework to ensure quality of outputs and process, leading to complementary statistics (indicators)
5. Definition of required metadata throughout the process (metadata schema)
6. Development of an adequate Reference Methodological Framework for processing data from various relevant sources, for official statistics
7. Development of data models, data validation and data exchange protocols
8. Development and testing of fully functional prototypes including the adaptation and consolidation of the methodology, the procedures and the tools for collecting, processing and analysis for 8. producing new statistics or enhancing existing ones. Due to handling of the smart meter data does not require comprehensive processing only general descriptions of the algorithms are provided and no software is produced
9. Definition of the implementation requirements of prototypes in the relevant statistical production processes at European and national level
10. Production of statistical output (experimental statistics will be hosted by Eurostat's dedicated section)
11. Transfer of knowledge, methodologies; software development, developer's and user's handbook; testing and maintenance, skills, at European and national level (e.g. Energy Statistics Working Group, Population and Housing Censuses Working Group).
Milestones and deliverables
See here for an overview of available milestones and deliverables.
|DM1||Report on the WP meeting mid-2019||Month 9|
|DM2||Report on the WP meeting mid-2020||Month 20|
|D1||Technical Report Version 1||Month 4||The main structure of the report will be established and the main focus is in activities 1 – 5 for each statistical product. The initial quality and metadata framework will be established.|
|D2||Technical Report Version 2||Month 14||The main focus will be on establishing procedures and techniques to produce statistical products from smart meter data and it covers mainly outputs of activities 4 – 8 for each statistical product. The quality and metadata framework will be updated.|
|D3||Technical Report Version 3||Month 23||The main focus will be producing relevant statistical products and disseminating the results and it mainly covers activities 4, 5, and 9 – 11 for each statistical product. The quality and metadata framework will be updated.|