Multimodal Systems: Taxonomy, Methods, and Challenges

Naturally, humans use multiple modalities to convey information. The modalities are processed both sequentially and in parallel for communication in the human brain, this changes when humans interact with computers. Empowering computers with the capability to process input multimodally is a major domain of investigation in Human-Computer Interaction (HCI). The advancement in technology (powerful mobile devices, advanced sensors, new ways of output, etc.) has opened up new gateways for researchers to design systems that allow multimodal interaction. It is a matter of time when the multimodal inputs will overtake the traditional ways of interactions. The paper provides an introduction to the domain of multimodal systems, explains a brief history, describes advantages of multimodal systems over unimodal systems, and discusses various modalities. The input modeling, fusion, and data collection were discussed. Finally, the challenges in the multimodal systems research were listed. The analysis of the literature showed that multimodal interface systems improve the task completion rate and reduce the errors compared to unimodal systems. The commonly used inputs for multimodal interaction are speech and gestures. In the case of multimodal inputs, late integration of input modalities is preferred by researchers because it allows easy update of modalities and corresponding vocabularies.