VALIDATION AND TRANSFORMATION

The process of generating and disseminating official statistics follows a number of key steps. However, these have often been carried out in an ad-hoc, manual way. Now, Eurostat can offer a range of services within what is called "IS4STAT" that largely automate and harmonize the production process in the European Statistical System (ESS).

These IS4STAT services concern format conversion or transformation and validation.

In order to use the services, the data files must be compliant with SDMX.

Format conversion

Data files are SDMX compliant when the structure of the file, the codes and concepts correspond to what has been defined in the relevant Data Structure Definition(s) (DSD). The transformation service can be called upon to convert an SDMX-compliant file from a data provider into the preferred .xml file format. This conversion service is based on the SDMX converter tool.

The conversion to the .xml format means that the validation services can then be used.

Validation

The new validation service infrastructure is built around two components performing validation operations: one for structural validation (called STRUVAL) and one for content validation (called CONVAL).

The services aim to ensure that no incomplete or otherwise incorrect data enters production databases for further processing. In addition, the services are designed to produce full, detailed and descriptive reports and feedback to production stakeholders on the possible presence of validation failures to support effective and efficient corrective response.

In this way, the process of data sets is faster, more transparent and more accurate. By automating the operation of the validation services in a structured way, precious staff resources can be allocated to higher value added tasks. By making this service available within the European Statistical Service it also provides an industry standard that is flexible and scalable to local level operations.

To use the validation services, however, a number of prerequisite activities must have been completed by the business units.