Two Empirical Studies on Audiovisual Semiotics of Uncertainty

There exists limited theoretical guidance on integrating visualization and sonification. In this paper, we address this gap by investigating audiovisual semiotics for uncertainty representation: joining uncertainty visualization and sonification to combine audiovisual channels for enhancing users' perception of uncertainty. We conducted two preregistered crowd-sourced user studies. First, we assessed suitable audio/visual pairs. Then, we investigated audiovisual mappings of uncertainty. Here, we use probability as it is an easily communicated aspect of uncertainty. We analyzed the participants' preferences and reaction times in both user studies. Additionally, we explored the strategies employed by participants through qualitative analysis. Our results reveal audiovisual mappings that lead to particularly strong preferences and low reaction times. Furthermore, we found that preferred audio/visual pairs are not necessarily suitable audiovisual mappings of uncertainty. For example, while pitch paired with brightness was preferred as a pair, it was not well suited as a mapping for uncertainty. We recommend audiovisual mappings of uncertainty that lead to low reaction times and high preferences in both user studies. This paper presents guidelines to anyone seeking to employ audiovisual representations for uncertainty, contributing to enhancing the perception of uncertainty.