Using causal inference and Bayesian statistics to explain the capability of a test suite in exposing software faults

Test effectiveness refers to the capability of a test suite in exposing faults in software. It is crucial to be aware of factors that influence this capability. We aim at inferring the causal relationship between the two factors (i.e., Cover/Exec) and the capability of a test suite to expose and discover faults in software. Cover refers to the number of distinct test cases covering the statement and Exec equals the number of times a test suite executes a statement. We analyzed 459166 software faults from {12} Java programs. Bayesian statistics along with the back-door criterion was exploited for the purpose of causal inference. Furthermore, we examined the common pitfall measuring association, the mixture of causal and noncausal relationships, instead of causal association. The results show that Cover is of more causal association as against \textit{Exec}, and the causal association and noncausal one for those variables are statistically different. Software developers could exploit the results to design and write more effective test cases, which lead to discovering more bugs hidden in software.