The concept of the statistical unit is:

• Online job advertisements (OJA): advertisements published on the World Wide Web, revealing an employer’s interest in recruiting workers with certain characteristics for performing certain work. This could be motivated by the employer’s need to fill a current vacancy, by an exploration of potential opportunities, or other reasons. These advertisements normally include data on the characteristics of the job (e.g. occupation and location), characteristics of the employer (e.g. economic activity) and job requirements (e.g. education and skills). Part of this information is available only as natural language textual data. 

The following concepts are used in the WIH-OJA data:

• Occupation (ISCO levels 1-4): the ISCO occupation that can be best identified in the online job advertissement. Occupations are closely related to job titles that are used in job postings to help attracting the right candidates and facilitate easier comparison of applicants' previous jobs and experiences. The list of variables associated to this concept are:
  • occupation4d
  • occupation3d
  • occupation2d
  • occupation1d
• Skill (ESCO levels 3): the skill, as defiined in ESCO ,identified in the online job advertisement. Skill is defined as "the ability to carry out the tasks and duties of a given job". The list of variables associated to this concept are:
  • Skill
• Skill reusability level: How widely a knowledge, skill or competence concept can be applied, according to ESCO. List of variables associated to this concept:
  • esco_v0101_reusetype
• ICT/Digital skill: indicates whether a skill is considered part of the ICT (Information and Communication Technology), or digital skills set. The variable associated to this concept:
  • esco_v0101_ict
• Place of work: city (LAU unit), region (NUTS 1-3) and country (Alpha-2 ISO 3166) of place of work  identified in the text of the OJA. List of variables:
  • City
  • Nuts3
  • Nuts2
  • Nuts1
  • Country
• Advertisement active time: The period of time during which the job advertisement is available on the web. This includes the following two concepts:
  • First active date: the date when the advertisement was first published online. The first active date is determined by the publication date, if provided and extracted from the advertisement, or by the date when the advertisement was first detected in instances where the publication date is absent.
  • Last active date: the date when the advertisement is no longer available on the web.  The last active date is determined by the date when the posting is not found in any of the sources, or its expiration date if mentioned in the advertisement. The list of variables associated to this concept are:
    • first_active_date
    • last_active_date

The main users of OJA data are the following:

• CEDEFOP, for policy making on skills and Vocational Training
• National Statistical Authorities, namely those participating to the Web Intelligence Network
• Eurostat Job Vacancy Statistics and ICT statistics, for producing statistics and indicators on occupations and skills
• Researchers and universities, for various research purposes

Coverage error 

The target population of job portals advertising potential jobs in the European labour markets may not be completely identified. 

Two landscaping studies are carried out to create an inventory of all the websites (job portals, etc.) in scope for the OJA use case. 

Undercoverage error may be present due to this inventory of sources being incomplete. Although possible, undercoverage should not be a significant source of error. The economic viability of job portals depend on their visibility and notability. Eurostat involved national labour market experts and used a harmonised methodology across countries, to discover and identify all relevant job portals, such as web search engines. One possible exception is very specialised job portals, which are active and notable only within a restricted community of experts.

A certain amount of undercoverage error, may be mitigated by the fact that many online job advertisements not retrieved from one source are anyway collected from other websites. This can happen when large job portals (aggregators) cover job advertisements from other (smaller or less relevant) job portals which are not included in the inventory of sources. 

Overcoverage of OJA data may arise due to the fact that more than one source (website) is publishing the same advertisement, which would result to double-counting or multiple listing of certain OJAs in the data. The table below presents some indicators on double-counting of OJAs, for example the average number of sources an advertisement has been reported by more than one source.  

Measurement error

Web scraping of job vacancies on job portals faces measurement errors, that are mainly the result of scraping errors (scraper may download incorrect data from the web page), errors on the web page or 

incorrect data on the web page (e.g. employers may upload incorrect data). 

Non-Response error 

In data from web content, like the WIH-OJA data, non-response errors can happen when one source of data (a job portal website) may either be “down”, i.e. not operating, or may block the access to the data, by either block a crawler from scraping a website, or the block the API). In such cases the nonresponse error can be understood as an “unit non response” errors, where no data is collected from a unit (job portal). 

Such events may happen frequently in a usually small number of sources in the WIH-OJA data. Details in such unit non-response errrors are documented and published with the methodological notes of the data release. 

Processing error 

Dealing with unstructured data text extracted from the web is challenging and errors may ocur in the processing and data preparation. More specifically, processing errors may arise in the OJA data due to the following processes: deduplication, merging, spam filtering, text processing, data cleaning, date detection, language detection.

Deduplication of job advertisements involves the identification and removal of duplicate entries within the dataset. Potential errors may arise during the deduplication of OJA data, specifically in the semantic deduplication stage of the pre-processing phase, which comprises two primary steps:

1. Metadata Matching: Utilizes job portal metadata (e.g., reference ID, page URL) to eliminate duplicate job ads on aggregator websites, following physical deduplication principles.
2. Logical Deduplication (Fuzzy Matching): Examines the content (description) of job ads, deeming them duplicates if the description and location closely match existing ads in the database, as part of logical deduplication.

Afterwards, a third deduplication phase compares structured fields from the information extraction stage. In the final comparison during the processing phase, ads are assessed based on publication date, occupation, place, skills, industry, and other distinctive characteristics. If ads share identical values in these fields, even if from different sources or phrased differently, they are considered duplicates and merged into a single advertisement. This step enhances the identification of duplicates with identical content present in classified variables. 

Errors in detecting the language used in each OJA may introduce inaccuracies in the subsequent text processing and summarizing phases. 

Errors may occur in date detection (first_active_date and last_active_date) using pattern matching techniques if the regular expressions used are not effecively customized.

Errors may arise during noise filtering of the downloaded web content. To detect true job advertisements in online advertisements of job portals from training courses or other non-related job-ad content, the system currently employs two distinct noise filters:

• No Vacancy Filter: designed to exclude content unrelated to job vacancies.
• Spam Filter: aimed at excluding postings that do not contain work opportunities, such as training courses. 

These filters are language-dependent and undergo training on different datasets. 

Sometimes incoherent classification between the label and the code of a given skill may arise due to processing.

Model assumption error 

Various models are used in the WIH-OJA, e.g. to classify the text found on online advertisements to a particular position in a classification (occupation, skill, location).  

Ontology-based models are used to classify advertisements according to the terms in the corresponding ontology. These models are used to search for an exact or similar match of the terms in the ontology and the natural language text in the title or description of the online job advertisement or of the metadata text. 

Machine Learning models (classifiers): If the ontology models provides no results, a pre-trained machine-learning algorithm is used to classify the advertisement.  

The most adopted machine-learning model is Naïve Bayes, which is trained on datasets using unsupervised learning techniques (for example, Word2Vec). Naïve Bayes is a simple technique for constructing classifiers: models that assign class labels to problem instances, represented as vectors of feature values, where the class labels are drawn from a finite set. It is not a single algorithm for training such classifiers, but a family of algorithms based on a common principle. 

The process must be applied separately for each attribute (variable) considered and for each language, as each model must be trained to best fit the characteristic of the domain and the language.

The process starts with the creation of the training set. Training sets are defined separately for each language pipeline, depending on the available vacancies, the size of the country and the composition of vacancies. In general, each training set includes over 70,000 vacancies. 

The training set is then divided into three parts:

• Training (60 %)
• Testing (20%)
• Evaluation, or quality measure (20%)

Training and testing can also be carried out on different kind of models to compare results and choose the best-suited model on a case-by-case basis. In all OJA cases, Naïve Bayes models have proved to be the most suitable, best-performing option.

Once trained, the machine-learning model can be used to classify vacancies in the processing pipeline.

Data undergo revisions for various reasons throughout the data pipeline. These revisions are crucial to maintaining the accuracy and reliability of the OJA data. The primary reasons for data revisions include: 

1. Error Correction: Data is revised monthly to address any errors discovered during processing, such as deduplication issues, code errors, or misclassifications. These revisions are applied retroactively to ensure consistency in historical data.
2. Classification System Changes: Revisions occur when there are changes in the classification system. This involves updating the taxonomy version and potentially modifying models, whether machine learning (ML) or rule-based, to align with the new classification standards. This ensures that the dataset remains up-to-date with the latest classification methodologies.
3. Methodological Improvements: Data revisions may be prompted by enhancements in methodologies to achieve more accurate results. This typically involves refining models, whether through rule adjustments or ML model updates, to improve the overall quality of the dataset.
4. Misclassifications and Model Errors: Corrections are made in response to misclassifications or errors identified during model debugging sessions. This process involves improving variables or the dataset itself to enhance the performance and accuracy of the models.
5. Code Errors: Revisions are made when code errors are detected during debugging sessions in the data pipeline. This includes correcting errors in the source code of the data pipeline itself, as well as changes in the code used for training models.

All changes are meticulously tracked in Git repositories. These can include:

• Changes in the source code of the data pipeline
• Updates to the training set
• Modifications to the rules governing classification
• Adjustments to the taxonomy
• Changes in the code used to train ML models

The version control provided by the repository allows for a thorough review of changes, enabling the navigation through the history of revisions and ensuring transparency in the data revision process.

The data collection process of the OJA data can be described in the following steps, as can be seen in the following image: 

Landscaping 

The purpose of the WIH OJA landscaping exercise was to obtain a comprehensive and representative list of job portals with OJA data of good quality. 

The sources (job portals) selection strategy has been implemented prior the start of the crawling activities, based on work and reports collected from International Country Experts.

To explore the available data on the use of online job-portals in recruitment and job-search in the 28 European Countries, International Country Experts (ICEs) were engaged in the Landscaping Activity.

This process is composed of 4 main steps:

1. Source selection in landscaping
2. Augmentation of source list
3. Agreements with source data providers
4. Coverage assurance 

At the end of the source selection process all the sources have been tested to identify potential issues related to the content of the site, to possible technical problems or to the unavailability of the site.

Data Ingestion 

This phase includes all the tasks needed to obtain and import raw data from web portals and to store them into a database. The key point in the ingestion phase is to ensure a stable data flow preventing potential loss of data due to harvesting issues (i.e. due to incomplete or inaccessible information). The data ingestion phase deals with both structured sources, which store information in structured fields, and unstructured sources, where information is largely extracted from large chunks of text. 

The data ingestion activity includes the collection of data through:

1. Scraping
2. Crawling
3. Direct access (e.g. API)

Deduplication 

Deduplication is a fundamental step of the OJA data processing: the ability to detect duplicate postings is necessary to obtain a high-quality set of results. In order to improve deduplication performance, this task is split in three distinct parts performed during the whole data processing pipeline:

• Physical deduplication
• Semantic deduplication
• Logic deduplication

The data ingestion phase is in charge of the first part (physical deduplication): postings coming from the same URL are considered as duplicates and not passed to subsequent phases. Metadata matching using metadata deriving from job portals to remove job advertisements duplicates on aggregators’ websites (e.g. reference id, page url). The two remaining phases will be described while depicting the whole process. 

The pre-processing phase is in charge of the semantic deduplication: this type of deduplication is based on content and is mainly composed of two phases:

• Logical deduplication or fuzzy matching carried out on the description (or content) part of the job advertisement. An OJA is considered a duplicate if the description (and the job location) is the same (or very similar) of another job ad already present in the database.
• Metadata matching using metadata deriving from job portals to remove job advertisements duplicates on aggregators’ websites (e.g. reference id, page URL) 

After the physical deduplication step (based on the URL or the unique id of the vacancy) and the logical deduplication (based on text similarity) a third deduplication step has been added to compare structured fields resulting from information extraction phase. At the end of the processing phase resulting advertisements are compared by:

• Publication date
• Occupation
• Place
• Skills
• Industry 

Distinct advertisements from the previous 2 phases with the same values for these fields are considered as duplicates and merged into a single advertisement. This additional step allows the identification of duplicates posted on different sources, written in different ways but all containing the same detailed content after the classification phase. 

Pre-processing 

The pre-processing is a critical step of the data processing pipeline. This phase must face some critical issues related to the particular type of data collected:

• Presence of noise in the source (information not directly related to OJA)
• Uniform coverage among all countries and occupations despite different languages and vocabularies
• Non-existence of a common standard to submit information
• Unstructured data (text) 

Moreover, to obtain usable results the following challenges must be faced:

• How to measure the quality of collected data.
• How to increase the quality of the data.
• Which variables affect quality of data.
• Duplicated job ads
• Availability of metadata
• Timeliness and periodicity of the data
• Complexity of data
• How to keep track of quality of data. 

The number of words collected is not considered an interesting measure as it depends mostly on the language characteristics and on the way the OJAs are written. The number and the quality of features extracted is considered much more important and will be monitored and verified step by step. 

The pre-processing phase can be divided into 3 steps:

• Merging
• Cleaning
• Text processing and summarising 

OJA Information Extraction 

The information extraction process is defined through a set of processing pipelines. A pipeline can be defined as a part of information extraction dealing with a specific content (attribute or variable) and with a particular language. Several pipelines can be combined to define the whole information extraction process. During the processing of each pipeline, advertisements are analysed to classify the contents of the pipeline (contract, occupation…) according to the specified language. 

In practice, the information retrieval process is composed of:

• One pipeline for each language considered
• One pipeline for each attribute (statistical variable) to be classified:
• occupation
• skills
• etc.

To check that the data does not display implausible values that are unlikely to reflect genuine statistical variation, a data validation process has been set up for OJA data. 

Data validation is defined as an “activity verifying whether or not a combination of values is a member of a set of acceptable combinations”, but this activity must be intended as a dynamic process (Eurostat’s Methodology for data validation manual).  

Every time that a data release is planned, statisticians check data compliance with the validation rules. When the data does not conform with the rules, one (or more) of three outcomes can occur:

• Data is revised and improved
• Information on the reasons for anomalous data behaviour is collected, contributing to the improvement of meta-data
• The rule is modified or dropped 

Therefore, while the validation process is a fundamental tool for data improvement and data quality control, its rule set is always amenable to change. For this reason, feedback on the validation rules and on the validation process is always welcome from statisticians and data users. 

Validation rules 

The validation rules currently applied to the OJA dataset covers a variety of statistical properties of the data. The most common (i.e., the ones that apply to the largest number of OJA variables) are:

• Consistency with Eurostat’s official code lists (e.g. occupation codes should follows the ISCO08 classification)
• Consistency within hierarchical classifications (e.g. the NUTS3 region of an ad must be contained in the NUTS2 region in which it is classified)
• The distribution of ads within categories of a classification should be reasonably stable over time and across data releases (this is a plausibility rule, meaning that its violation raises a ‘Warning’, which is investigated further)
• When a variable follows a certain classification, the distribution of ads across the categories should not be too unbalanced (e.g. if 50% of all ads were found in a single 4-digit ISCO occupational category, or if no data is found for dozens of occupational categories, this raises a ‘Warning’ which is then investigated further)
• For some variables (e.g. date and country), there should be no missing data
• For string variables, some formatting requirements are usually imposed (e.g. there should be at least one alphabetic character in every company name)

Besides these rules (and other minor ones) operating at the level of the data record, or of the distribution, there are some others that check the consistency of the whole database against basic logical or numeric requirements. These rules (also called structural validation rules) include for example checking for correct naming of datasets and variables, absence of empty fields, etc.

Furthermore, country experts regularly check, on a random set of OJAs, if the text found on OJAs is correctly allocated to the right position in the various classifications used (ISCO for occupations, ESCO for skills, NUTS for regions, etc..).

Additional information on the validation procedure, the validation rules or validation reports the can be found on the following wiki page of the Eurostat’s WIH: Validation rules. The access to this page is nevertheless restricted to a specific community of users. An access grant can be provided upon request by sending an email to the functional mailbox ESTAT-WIH@ec.europa.eu.