A Model-Based Approach for Specifying Changes in Replications of Empirical Studies in Computer Science

Context: The need of replicating empirical studies in Computer Science (CS) is widely recognized among the research community to consolidate acquired knowledge generalizing results. It is essential to report the changes of each replication to understand the evolution of the experimental validity across a family of studies. Unfortunately, the lack of proposals undermines these objectives. Objective. The main goal of our work is to provide researchers in CS, and in other areas of research, with a systematic, tool-supported approach for the reporting of changes in the replications of their empirical studies. Method: Applying DSR, we have developed and validated a composite artifact consisting of (i) a metamodel of the relevant concepts of replications and their changes; (ii) templates and linguistic patterns for reporting those concepts; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study including 9 families of empirical studies from CS and Agrobiology. The 9 families encompass 23 replication studies and 92 replication changes, for which we have analyzed the suitability of our proposal. Results: The multiple case study revealed some initial limitations of our approach related to threats to experimental validity or context variables. After several improvement iterations, all the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity across the family of experiments and its corresponding visualization. Conclusions: Our proposal for the specification of replication changes seems to fit the needs not only of replications in CS, but also in other research areas. Nevertheless, further research is needed to improve it and to disseminate its use among the research community.