EU Science Hub

Language Technology Resources

This page gives you an overview of Linguistic Resources and Tools (multilingual software, parallel corpora, and more) that are available for download from the JRC’s webpages.

The data releases are in line with the general effort of the European Commission to support multilingualism, language diversity and the re-use of Commission information.

The JRC has developed Language Technology (text mining, computational linguistics) tools for more than twenty languages and it has been analysing up to 300,000 online news articles per day since 2004, thus creating valuable meta-data. Some of this software and of the created meta-data have been released publicly, starting in 2006 with the large-scale multilingual parallel corpus JRC-Acquis, covering twenty-two languages. The JRC also helps distribute the linguistic resources produced by other European Union organisations. The most outstanding feature of all these resources is their high multilinguality and the fact that the texts are parallel (i.e. the corpora consist of texts and their manually produced translations). For comparative details, see the journal publication An overview of the European Union’s highly multilingual parallel corpora.

The resources listed below are useful to academia and industry to carry out research and development into highly multilingual text analysis tools and especially into cross-lingual applications. The resources distributed here have already been used to train statistical machine translation, generate dictionaries, evaluate multilingual document summarisers and information extraction software, support librarians in their daily work, help improve name searches in large data repositories, and more.

 

JRC-Acquis

The JRC-Acquis is a multilingual sentence-aligned parallel corpus in 22 languages, containing a total of over 1 billion words.

 

DGT-Acquis

The DGT-Acquis is a multilingual paragraph-aligned parallel corpus in all 23 official EU languages, including documents from the Official Journal’s L and C series since the year 2004.

 

DCEP-Digital Corpus of the European Parliament

The DCEP is a multilingual sentence-aligned parallel corpus in 23 official EU languages (253 language pairs) plus Turkish (altogether 24 languages and 276 language pairs when considering the small number of Turkish documents), consisting of European Parliament texts produced between 2001 and 2012 and containing over 1.3 billion words.

 

DGT-Translation Memory (DGT-TM)

DGT-TM is a Translation Memory of the Acquis Communautaire, i.e. the body of European legislation, including all the treaties, regulations and directives adopted by the European Union (EU) and the rulings of the European Court of Justice.

 

EAC-Translation Memory (EAC-TM)

EAC-TM is a Translation Memory (a collection of sentences and their manually produced translations) in 26 languages focusing on the subject domain of education, training, culture and youth.

 

ECDC-Translation Memory (ECDC-TM)

ECDC-TM is a Translation Memory of the web pages of the European Centre for Disease Prevention and Control (ECDC).

 

JRC-Names - a multilingual named entity resource

JRC-Names is a highly multilingual named entity resource for person and organisation names. It consists of large lists of names and their many spelling variants (up to hundreds for a single person), including across scripts (Latin, Greek, Arabic, Cyrillic, Japanese, Chinese, etc.). It has been compiled by analysing large volumes of multilingual news since 2004, combined with Wikipedia mining. It gets updated daily. As of March 2016, it contains 307,000 person and organisation names plus 333,000 spelling variants written in over 20 different scripts and in many more languages. At its initial release in September 2011, there were 205,000 distinct entities.

 

JRC-Names as Linked Data

JRC-Names as Linked Data is an RDF representation of the JRC-Names resource.
This new edition offers more information compared to the previous JRC-Names resource, including: titles and function names that have been historically found next to the person mentions; information about the time period during which name variants and their titles were found; various frequency counts.
It has links to other linked datasets such as  DBpedia, New York Times Open Data and Talk of Europe.

 

JEX - JRC EuroVoc Indexer

JEX is multi-label classification software that automatically assigns a ranked list of the over six thousand descriptors (classes) from the controlled vocabulary of the EuroVoc thesaurus to new texts. JEX has been trained for twenty-two EU languages.

 

Multilingual summary evaluation data

This is a manually annotated collection of document clusters of parallel texts in seven languages (Arabic, Czech, English, French, German, Russian and Spanish) that can be used to evaluate multi-document, or even single document, summarisation software.

 

Sentiment-annotated set of quotations

This is a set of 1590 English language quotations (reported speech) extracted automatically from the news and annotated manually for the sentiment expressed towards entities (persons or organisations) mentioned inside the quotation.

 

Named entity annotations of a Turkish tweet data set

This resource includes 1322 named entity annotations from a total of 868 Turkish tweets published on July 26 2013 between 12:00 and 13:00 GMT.