WP2 – A generic validation report
To facilitate a multi-step validation process based on a mix of shared, replicated and autonomous services, a common machine-readable validation report is required. The Netherlands are in the process of designing a generic report structure that fulfills all the necessary requirements.
A review of multiple validation reports from different countries and Eurostat yielded an impressive list of diverse information that could be of interest in such a report. After cleaning and categorizing this long list of validation report items, we were able to deduce a few non-trivial requirements for this validation report.
The basic requirement for any validation report is
- Identifiability: A validation report shall convey validation results such that they can be identified with the validation procedure, the validation rules used, and the validated data.
A validation report that does not identify which validation rule failed on which data might as well be a popup saying that “something went wrong.” Another requirement is based on the division of the validation process in multiple services.
- Combinability: Two validation reports shall be combinable in such a way that the result is again a validation report that includes all information that separate reports contained.
If my validation procedure needs to run the data through validation services A and B, the validation reports of both services should be combinable into a single report. The third important requirement is
- Aggregability: A validation report can be aggregated such that the result is again a validation report.
Aggregations play an important role in validation reports. They are the basis for validation metrics and useful for general overview of larger validation procedures. A human-readable version can be obtained by transformation, applying style sheets or other suitable methods.
The main structure of the machine-readable version of the validation report basically allows for any value, metric or method to be included in the report. To fill it with life, we gathered a list of items from brainstorm sessions and validation reports that are currently used in different NSOs.
A pilot implementation of the validation report structure in JSON and a pilot dashboard supporting filtering and aggregation of validation reports can be found on Github.