Exploring Microtask Crowdsourcing as a Means of Fault Localization

Microtask crowdsourcing is the practice of breaking down an overarching task to be performed into numerous, small, and quick microtasks that are distributed to an unknown, large set of workers. Microtask crowdsourcing has shown potential in other disciplines, but with only a handful of approaches explored to date in software engineering, its potential in our field remains unclear. In this paper, we explore how microtask crowdsourcing might serve as a means of fault localization. We particularly take a first step in assessing whether a crowd of workers can correctly locate known faults in a few lines of code (code fragments) taken from different open source projects. Through Mechanical Turk, we collected the answers of hundreds of workers to a pre-determined set of template questions applied to the code fragments, with a replication factor of twenty answers per question. Our findings show that a crowd can correctly distinguish questions that cover lines of code that contain a fault from those that do not. We also show that various filters can be applied to identify the most effective subcrowds. Our findings also presented serious limitations in terms of the proportion of lines of code selected for inspection and the cost to collect answers. We describe the design of our experiment, discuss the results, and provide an extensive analysis of different filters and their effects in terms of speed, cost, and effectiveness. We conclude with a discussion of limitations and possible future experiments toward more full-fledged fault localization on a large scale involving more complex faults.