Towards Human-centered Design of Explainable Artificial Intelligence (XAI): A Survey of Empirical Studies

With the advances of AI research, AI has been increasingly adopted in numerous domains, ranging from low-stakes daily tasks such as movie recommendations to high-stakes tasks such as medicine, and criminal justice decision-making. Explainability is becoming an essential requirement for people to understand, trust and adopt AI applications. Despite a vast collection of explainable AI (XAI) algorithms produced by the AI research community, successful examples of XAI are still relatively scarce in real-world AI applications. This can be due to the gap between what the XAI is designed for and how the XAI is actually perceived by end-users. As explainability is an inherently human-centered property, in recent years, the XAI field is starting to embrace human-centered approaches and increasingly realizing the importance of empirical studies of XAI design by involving human subjects. To move a step towards a systematic review of empirical study for human-centered XAI design, in this survey, we first brief the technical landscape of commonly used XAI algorithms in existing empirical studies. Then we analyze the diverse stakeholders and needs-finding approaches. Next, we provide an overview of the design space explored in the current human-centered XAI design. Further, we summarize the evaluation metrics based on evaluation goals. Afterward, we analyze the common findings and pitfalls derived from existing studies. For each chapter, we provide a summary of current challenges and research opportunities. Finally, we conclude the survey with a framework for human-centered XAI design with empirical studies.