Sound is poorly served in interactive systems

Interactive computer systems and virtual reality developed historically over Computer Graphics, adding interactivity, toward immersive interactive visual environments.

In visual-based systems and VR, high quality sound is still remarkably rare. When present, sound relies usually on playing fixed sounds, basically triggered by detecting some events in the graphics scene. The main research stream in regards to sound outputs consisted so far in implementing sound spatialisation means over prerecorded sound, with the aim to improve the quality of immersion in the 3D scene.

Conversely, quality sound-powered interaction developed quasi-exclusively so far in the framework of sound and music computing, by targeting digital musical instruments where visual aspects are noticeably absent.

Hence, there exist today research and achievements in visual-based interactive systems, and sound-based interactive systems, but these research developed in parallel over distinct communities that rarely crossed each others.

Goal and impact

A major new step is needed toward quality sound in interactive multisensory systems, in order to enable sound-powered VR systems where sound is deeply integrated into the virtual scene, and all sound qualities are truly tied to the interaction.

Improving sound quality in interactive computerized systems would have impact on all human activities, at a time when digital technology is involved in most our activities: communication, creation, interaction, production, gaming, etc. All of the tasks in which having a deep sensorimotor involvement would benefit from this new step in the “interactive computer”, increasing for example satisfaction, involvement, productivity, creativity or quality. It would trigger opportunity in Europe, for the content-production industries.

Research axis : examples

Achieving sound-powered interactive multisensory systems would require incorporating sound generation algorithms, such as signal-based sound synthesis or physics-based acoustic simulation, but also to intricate them in depth within the scene.

The approaches and methodologies that would to that aim are still unexplored and unknown. Calling sound into VR and interactive systems corresponds to new bottlenecks and challenges.

Would we need only to let various models (sound, visual, haptics models, for example) cohabitate, and simply exchange some data? In such an approach, which type of data should be exchanged between the models, and how? What are the time and latency constraints? Should we consider that one of the several models is a “master model”, from which the others are controlled? Otherwise, how should we set up the cohabitation of these models, to enable a coherent generation of all the phenomena, for each of the involved sensori-motor channels?

When physics-based modelling and algorithms are called within the scene, should we set up various physics models (for example one per sensory modality)? In this case, how could we guaranty, and to which level, the coherency of the mechanical (visual, force feedback) and acoustic physical phenomena generated?

Conversely, would we need to reconsider the core formalisms from which the scenes are built, and the algorithms run, toward new truly ‘multisensory’ modelling and simulation formalisms, which would enable setting up global models able to generate in one stream all of the dynamic phenomena in response to the interaction?

Hence, introducing quality sound cannot be achieved by naively interconnecting the achievements of sound and music computing into those of computer graphics and virtual reality.

It requires reconsidering in depth the modelling formalisms, the computation algorithms and their implementations, the latency constraints, the time loops, the computation hardware and roles of involved co-procesors, and possibly the transducers, of the today’s interactive computerized systems.

Means

In such research, specialists of sound and image computing and virtual reality could be involved, and work together. Steering applications fields, such as art, creation, gaming, education, and human and cognitive sciences, should also be involved.

Europe may also trigger the emergence of new community meetings (workshops, conferences…) where, beyond existing structured scientific domains such as computer graphics, sound and music computing, haptics or human computer interaction, all of the sensori-motor human skills are altogether considered and targeted.