API - Getting started with SDMX3.0 API

Introduction

The SDMX initiative sets standards to facilitate the exchange of statistical data and metadata using modern information technology. SDMX has also been published as an ISO International Standard (ISO 17369).

The operations in this API supports SMDX-3.0 artefacts and implements the SDMX REST API specification v2.0.

To make the most of this guide, a basic knowledge of XML and REST web services is required.

The main elements are referred to as SDMX artefacts. Here is a short definition of some terms used in SDMX:

• Dataset: a collection of related observations, organised according to a predefined structure
• Data Structure Definition (DSD): metadata describing the structure and organisation of a dataset, the statistical concepts and attached to them code lists used within the dataset
• Dimensions: concepts that determine the dataset’s "physical" structure
• Codelist: a code list is a predefined list from which some statistical coded concepts take their values. Each code list has the following properties:
  • identifier (it provides a unique identification within the set of code lists specified by a structural definitions maintenance agency);
  • name (also unique);
  • description (a description of the purpose of the code list); and
  • code value length (either an exact or a maximum number of characters and a type, i.e. numeric or alphanumeric).
• Attributes: give additional information about the concepts used and do not affect the dataset structure itself
• Dataflow: a structure which describes, categorises and constrains the allowable content of a dataset that providers supply for different reference periods
• Concept scheme: the descriptive information for an arrangement or division of concepts into groups based on characteristics, which the objects have in common. A concept scheme is a maintained list of concepts that are used in key family and metadata structure definitions (Definitions from EUROSTAT SDMX info space and OECD Glossary of statistical terms)

For in depth details, check as well the learning section of SDMX.org or the  formal definition of the SDMX information model.

About versioned artefacts

While all SDMX artefacts could be versioned, currently only the following structural artefacts are versioned: Code lists (CL), Concept Schemes (CS), Data structure definitions (DSD).

It means that such artefacts are final and identified by a version number and safe to be copied/cached for further reference.

Other artefacts : Dataflow, ContentConstraint will always have the default version '1.0' and need to be requested again for updates.

Retrieving Structural metadata artefacts

Starting from the online data code of a dataset of choice, it is possible to query the API for detailed metadata on this data

Call in this guide, explicitly request not compressed response but it is recommended to use compression in automated or recurrent calls.

Looking into the metadata of a dataset

in the SDMX Dataflow

Taking the dataset ISOC_CI_ID_H as example, its main information are available in its Dataflow SDMX artefact

Link: https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataflow/ESTAT/ISOC_CI_ID_H/1.0?compress=false 

 This resource is not versioned, so 1.0 or + (meaning latest) can be used interchangeably 

The minimal response would always contains the dataset label and the reference to the versioned DSD currently used by the dataset

Additionally a set of annotations would provide additional information (omitted in previous example, please expand full XML below to see them) 

ISOC_CI_ID_H Dataflow XML 

in the referenced data structure definition (DSD)

From the reference present in the dataflow, it is possible to query for the corresponding SDMX Data Structure Definition

DSD Link:

Https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/datastructure/ESTAT/ISOC_CI_ID_H/28.0?compress=false

 These resources are versioned, so version present in the reference must be used to ensure consistency.

This definition is providing the list of dimensions used in the definition of the time-series of the dataset.

The order of dimensions will help build key filtering in the data query later

For each dimension a reference is provided

1. to the concept holding the dimension label, the concept is one item of a concept scheme.

   In current Eurostat Dissemination Chain, there is one DSD and one Concept Scheme generated for each dataset with identical identifier (but potentially different version). 

   Concept Scheme Link: https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/conceptscheme/ESTAT/ISOC_CI_ID_H/28.0?compress=false

2. to the code list holding the code and labels for the dimension positions

These code lists are reference metadata and may contains more code and labels than the one used by a specific dataset.

To known the list of positions present in the dataset, please refer to the Content Constraint artefact (next section).

Additionally the DSD defines

• the mandatory TIME_PERIOD time-dimension where the value are expressed using ISO 8601 standard
• the primary measure OBS_VALUE holding the statistical value

• the optional value attribute OBS_FLAG holding the statistical status (also referred as flags)

Flag codelist Link:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/codelist/ESTAT/OBS_FLAG/1.31?compress=false

ISOC_CI_ID_H DSD XML 

in the SDMX Data Constraint

Content Constraint Link:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataconstraint/ESTAT/ISOC_CI_ID_H/1.0?compress=false 

In the API, SDMX Data Constraints artefacts are used to define available positions, it lists for each dimension code the list of position code that are used at least once to refer to a statistical observation.

From example below we can see that ISOC_CI_ID_H dataset 

• provides annual data ( freq = A)
• provides data for 14 indicators 
• provides data in 2 units
• provides 17 breakdown and a TOTAL on "Type of Household" [ hhtyp ]
• provides data for EU aggregates and member states + other countries
• provides data from 2002 to 2010 plus 2014

Data Constraint SDMX XML 

retrieve several artefacts in a single response

 It is not necessary to do these calls one by one.

Starting back from the Dataflow it is possible to include the referenced artefacts, at two different level

Also from the DSD, it is possible to include the referenced artefacts :

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/datastructure/ESTAT/ISOC_CI_ID_H/28.0?compress=false&references=children

Special case of Dataset listing

Instead of specifying a dataset code in the dataflow request the special character * can be used to mean ALL and retrieve a list of all Eurostat datasets in one request

 It is recommended to retrieve the compressed version 

ALL DATAFLOWS link

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/structure/dataflow/ESTAT/* 

Special case of Metadata harvesting

Similarly to the request on ALL dataflows it is possible to get the latest version for all artefacts for a specified type

 It is recommended to retrieve the compressed version 

Data query

in SDMX 3.0, the data query requires the dataflow complete identifier : type/agency/ID/version

Data Link:

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/data/dataflow/ESTAT/ISOC_CI_ID_H/1.0?compress=false 

Above link retrieves the complete dataset in SDMX-ML 3.0 Structure Specific Data, that is the default format as Generic Data is removed from SDMX 3.0

The data file is composed of a set of time-series identified by its series-keys containing Observation as the one show below

SDMX Generic Time Series XML Excerpt

It is possible to further customise the query to retrieve only the needed data or to request a different output format.

Filtering data

In SDMX 3.0 REST API, there are two kind of filtering mechanism

• Filtering on series-keys : it is possible to define for each dimension one value or a wildcard to scope the data extraction ;  this filter must follow dimensions order
• Filter by component value : it is possible to define for each dimension a list of value to filter on ; TIME_PERIOD dimension allow range ; this filter is not impacted by dimensions order

These two filtering mechanisms can be combined with the limitation that a dimension specified on the series-keys filter should not be repeated in the component value filter.

While all filtering could be done on component value filter, using series-keys filter can ease readability of the data slices targeted by a data query.

Filtering on series-keys

First level of filtering is about selecting how the dataset could be sliced. It is done by filtering on the series-keys following the dimension order as specified in the DSD

In the example of ISOC_CI_ID_H, the series-keys template is as follow

FREQ.INDIC_IS.UNIT.HHTYP.GEO

with the following syntax:

• '*' is a wildcard for ALL values meaning no filtering is needed for this dimension 
• A specific value can be set for a dimension
• Partial key is supported and assume '*' wildcard value for remaining dimensions, see example below

Filtering on component value

Component value filtering has the following syntax c[dim_id]=value:

• dim_id is the dimension id as found in the DSD 
• value can list several positions separated by a comma
• both dim_id and value are case insensitive

Filtering on time period

Filtering the observations to be returned based on their TIME_PERIOD value is also controlled via component value filtering as other dimension but additionally it supports value range definition prefixing the value by a short operator and a colon ': '

Reusing above single time-series example, it can be restricted on TIME_PERIOD

c[TIME_PERIOD]=ge:2008+le:2010

Supported format 

An additional format parameter allows to request a response in a different semantic format

Compression

An additional compress parameter allows to disable compression. It is recommended to use it only if necessary.

Uncompressed TSV data link

https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/data/dataflow/ESTAT/ISOC_CI_ID_H/1.0?format=tsv&compress=false

Retrieving Navigation artefacts

It is worth to mention secondary navigation artefacts that represent as SDMX artefacts a classification of dataset in categories (also referred as "Navigation Tree")

• Category Scheme : Hierarchy of categories 
• Categorisation :  one categorisation is referencing one dataset into a category of a Category Scheme