What is i-Treasures about?
i-Treasures is a very special research project. Why? Because it brings together the past and the present, or, to be more specific, two areas which traditionally have rarely gone hand in hand: intangible cultural heritage and advanced modern technologies.
When you hear about 'cultural heritage' you probably first associate it with tangible architectural buildings or other works of art, such as paintings. However, an important part of our cultural heritage is intangible traditions and living expressions inherited from our ancestors. These make up the Intangible Cultural Heritage (ICH), which includes performances, practices, knowledge and skills. There are not many people who have the skills to perform traditional dances or songs. Therefore, if we do not cherish this heritage and pass it on to next generations, these artistic expressions could be lost forever. To this end, the i-Treasures project develops innovative tools for the preservation and transmission of our precious intangible cultural heritage with the help of modern technologies. To demonstrate our technologies, we have selected a wide range of intangible cultural heritage domains, in which the human body is dominant. We focus on the following four ICH categories: singing, dancing, traditional craftsmanship (focusing on pottery), and contemporary music composition.
What have we developed?
The main objective of i-Treasures is to develop an open and extendable platform to provide access to ICH resources, enable knowledge exchange between researchers and contribute to the transmission of rare know-how from Living Human Treasures (LHT) to apprentices. Unlike other projects in the cultural heritage domain, i-Treasures goes beyond the mere digitisation of cultural content. Its main contribution is the creation of new knowledge by adopting novel technologies and methodologies for capturing, analyzing and modelling traditional and contemporary ICH expressions.
Capture of ICH performances is based on the use of multi-sensing technologies. In order to investigate the vocal tract articulation during singing, we designed a prototype hyper-helmet, equipped with several lightweight sensors, including an ultrasound miniaturized transducer, an optical camera, a microphone, etc. Dance performances are captured using different motion capture technologies such as depth sensors, inertial sensors and optical mocap systems. In the same fashion, finger, hand and body gestures of wheel-throwing pottery experts are recorded using optical, depth and inertial sensors. Recently, the first version of the i-Treasures ICH dataset was released, containing annotated ICH performance data from different use cases. This dataset can be obtained from the i-Treasures web site, and is free for research purposes.
Based on the in-depth analysis of the various cultural expressions as well as discussions with ICH experts, we have developed several signal processing techniques for the analysis of the captured ICH performances and the extraction of meaningful features. These include modelling and recognition of facial muscle movements, body movements, hand and finger gestures, brain activity, vocal tract operation (including tongue and lips movement), acoustic speech and music sounds.
Data obtained from capturing and analysing ICH, coupled with other cultural resources, is accessible through the i-Treasures web platform, which will allow both knowledge exchanges among researchers as well as education of new apprentices. Using our platform you can learn to dance, sing, make a pottery object or compose music – there are special pedagogical courses which invite interested learners to acquire the rare know-how with the help of a virtual tutor. The platform offers great possibilities of on-line learning, with games, realistic 3D environments and avatars keeping the learner engaged.
Several prototype educational game-like applications have been implemented for Tsamiko dance, Walloon dance, Salsa dance, Human Beat Box singing, Byzantine music, pottery, and contemporary music composition. The games are designed to get input from various sensors and game devices, such as the prototype hyper-helmet (for vocal tract data collection) or off-the-shelf commercial sensors like Kinect. The game scenario involves a user who attempts to learn the basics by observing and imitating expert performances. The learner’s performance is captured by the various sensors, it is analysed and evaluated in real time and the evaluation result is presented to the learner using audio-visual feedback.
Another interesting learning tool developed within the project is a Text-to-Song synthesis system for Sardinian Canto a Tenore, Byzantine music and English pop, which enables the user to enter lyrics and/or notes and produce the equivalent singing voice.
A novel intangible musical instrument for contemporary music composition has also been designed. The intangible instrument offers a novel natural user interface for learning, performing and composing based on gestural and emotional expression. The upper body and finger gestures, as well as the emotional state of the user are captured by multiple sensors and they are implicitly or explicitly mapped to various sound parameters, resulting in real-time sound synthesis. The interface is supported by a novel set-up made of plexiglas, shaped so as to look like a table, over which the user can perform piano-like gestures.
Finally, the platform includes a digital repository where ICH expert performances and corresponding metadata can be stored or accessed. The user can search the digital repository for stored ICH content (e.g. songs, dance performances, etc.) using advanced search functionalities that exploit semantic analysis information. Metadata information is compatible with part of the Europeana Semantic Element Set.
We have also developed a generic framework, which provides an easy way to design and develop simple game-like applications for any kind of dance or other kinds of body activities (e.g. related to physical exercise or martial arts). Finally, we recently released an open source library (MotionMachine) that enables the rapid prototyping, extraction and visualization of motion capture features.
The project will benefit scientists, cultural heritage communities, school workers and educators, students, dancers and singers.
Impact, dissemination and future work
ICH is progressively promoted as a powerful societal and economic driver for sustainable development, at the local, national, European and global level. The long-term goal of i-Treasures is to provide both a methodological and technological framework to local actors (territorial authorities, schools etc.) in order to preserve ICH (e.g. handicraft professions) and promote local development. In Europe, revenues generated by cultural tourism are most significant and i-Treasures can have a strong economic impact on the tourist sector, by promoting intangible heritage which is typically related and connected to certain less known places.
Furthermore, we also introduced our tools and technologies in the cross-disciplinary MA programme “Educational Sciences - Learning Technologies” at the Aristotle University of Thessaloniki during the current spring 2016 semester. The aim was to enable students to use the i-Treasures tools to safeguard expressions of intangible heritage in collaboration with local cultural heritage institutions and primary/secondary schools. The students will submit safeguarding projects making use of the i-Treasures technologies and also relevant ethnographic methods as part of their assessment.
Now more than ever, it is important for us, Europeans, to guard and transmit our cultural heritage and i-Treasures makes an important step towards this direction. It raises awareness of the importance of ICH and contributes to its safeguarding, while maintaining cultural diversity and encouraging intercultural dialogue and respect for other ways of life.
EURONEWS TV documentary on i-Treasures on air 11th of July 2016
During our last project meeting in Thessaloniki, the Euronews film crew filmed procedures for capturing and analyzing different ICH expert performances and evaluating our game-like applications by learners for the Futuris show. You can watch the video on Euronews Futuris programme online.