Creative Uses of AI Systems and their Explanations: A Case Study from Insurance

Recent works have recognized the need for human-centered perspectives when designing and evaluating human-AI interactions and explainable AI methods. Yet, current approaches fall short at intercepting and managing unexpected user behavior resulting from the interaction with AI systems and explainability methods of different stake-holder groups. In this work, we explore the use of AI and explainability methods in the insurance domain. In an qualitative case study with participants with different roles and professional backgrounds, we show that AI and explainability methods are used in creative ways in daily workflows, resulting in a divergence between their intended and actual use. Finally, we discuss some recommendations for the design of human-AI interactions and explainable AI methods to manage the risks and harness the potential of unexpected user behavior.