New methods for perceptual evaluation of next generation sound systems.

As part of SenseLab’s ongoing participation within the International Telecommunication Union (ITU), Nick Zacharov, Lead Technologist at FORCE Technology's SenseLab, was nominated to chair the rapporteur group (RG25, 2015-2016) on audio quality assessment within the Radio Communication Sector.

This group oversees all ITU Recommendations relating to audio quality matters, including the ITU-R BS.1534 and ITU-R BS.1116 Recommendations. Both of these recommendations are widely used throughout the audio industry for the selection of audio codec, watermark technologies and so forth, through subjective listening tests.

Next generation sound systems

Presently there is a big move across the audio, broadcast and cinematographic industries to introduce so-called next generation sound systems (also known as advanced sound systems). Such systems allow for a scalable reproduction of the sound into a user’s environment whether they are using a 2-channel stereo system (headphone or loudspeaker) or a large number of loudspeaker channels, e.g. 5.1, 10.2, 22.2 channels of loudspeakers. This is achieved using a so-called channel-based or object-based technology, whereby the sound field is rendered according to the nature of the loudspeaker configuration (number, loudspeakers location, etc.).

With the advent of such next generation advanced sound systems come a wide range of new quality characteristics that are hard to capture with the existing listening test recommendations. The need to developed new means for the assessment of audio quality such highly immersive spatial sound applications has thus emerged.

Novel methods for perceptual evaluation of sound systems

As part of this effort, SenseLab has worked on the development of novel methods for the perceptual evaluation of next generation sound systems. The Multiple Stimulus Ideal Profile Method has been developed with the aim of efficiently evaluating overall quality of such technologies as well as characterising the perceptual nature of the technologies.

In order to validate the method, joint evaluation tests were performed in collaboration with BBC Research and Development (Salford UK) and QI Statistics (UK). The excellent results of this work were contributed to the ITU-R WP6C RG25 group and have resulted in a work plan to further refine the method with the hope of approving a new ITU-R Recommendation. Presently the method is in a “preliminary draft new recommendation”.

In order to get further input and review of the method, a conference paper was prepared [1] for the Quality of Multimedia Experience (QoMEX) conference held in Lisbon. The paper was well received and led to good discussion and further received the best paper award from the conference (see figure 1).
Figure 1. QoMEX 2016, best paper award.

The invention of the sound quality wheel

In addition to this work, SenseLab has been working on unifying the sound quality terminology used in perceptual evaluation. Whilst the wine aroma wheel was already formalised in the early 80’s for the wine industry, no equivalent lexicon has existed in the field of audio until now.

The sound quality wheel (see figure 2) has been in recent development and published, providing a lexicon for generic reproduced sound quality evaluation. This work has been progressing within SenseLab and has been now been published [2,3]. The sound attribute wheel has also been contributed into the ITU-R WP6C RG25 for consideration as part of new methods for perceptual audio evaluation.
Figure 2. Version 2.0 of the sound quality wheel, Audio Engineering Society, 2016 [2].
The work continues in the development of such a new recommendation with the hope of finalisation within the next 12-18 months. The next meeting of the group is in October 2017, where we hope to see progress towards a new Draft Recommendation.

Virtual reality and broadcasting

ITU-R is studying the important of VR for broadcasting and has found rapporteurs to study what aspects of VR will benefit from ITU-R Recommendations. On 17 October 2016, the virtual reality (VR) workshop “Seminar on Virtual Reality and 360 in Broadcasting” was held at ITU’s Headquarters in Geneva. Presentations from the seminar can be found here.

Further information

For further information about ITU rapporteur group (RG25) and the new methods for perceptual evaluation of sound systems, please contact Lead Technologist Nick Zacharov, tel. +45 43 25 15 88.

[1] Nick Zacharov, Chris Pike, Frank Melchior and Thierry Worch. “Next Generation Audio System Assessment using the Multiple Stimulus Ideal Profile Method”, Quality of Multimedia, Experience (QoMEX), Lisbon, Portugal, 2016.

[2] Pedersen, Torben H., Zacharov, Nick. “The Development of a Sound Wheel for Reproduced Sound”, Audio Engineering Society 138th Convention, May, 2015.

[3] Zacharov, Nick, Pedersen, Torben H. “Spatial Sound Attributes — Development of a Common Lexicon”, Audio Engineering Society 139th Convention, October, 2015.