Collective Risk Minimization via a Bayesian Model for Statistical Software Testing

In the last four years, the number of distinct autonomous vehicles platforms deployed in the streets of California increased 6-fold, while the reported accidents increased 12-fold. This can become a trend with no signs of subsiding as it is fueled by a constant stream of innovations in hardware sensors and machine learning software. Meanwhile, if we expect the public and regulators to trust the autonomous vehicle platforms, we need to find better ways to solve the problem of adding technological complexity without increasing the risk of accidents. We studied this problem from the perspective of reliability engineering in which a given risk of an accident has severity and probability of occurring. Timely information on accidents is important for engineers to anticipate and reuse previous failures to approximate the risk of accidents in a new city. However, this is challenging in the context of autonomous vehicles because of the sparse nature of data on the operational scenarios (driving trajectories in a new city). Our approach was to mitigate data sparsity by reducing the state space through monitoring of multiple-vehicles operations. We then minimized the risk of accidents by determining proper allocation of tests for each equivalence class. Our contributions comprise (1) a set of strategies to monitor the operational data of multiple autonomous vehicles, (2) a Bayesian model that estimates changes in the risk of accidents, and (3) a feedback control-loop that minimizes these risks by reallocating test effort. Our results are promising in the sense that we were able to measure and control risk for a diversity of changes in the operational scenarios. We evaluated our models with data from two real cities with distinct traffic patterns and made the data available for the community.