An experimental evaluation of continuous testing during development

TitleAn experimental evaluation of continuous testing during development
Publication TypeConference Paper
Year of Publication2004
AuthorsSaff D, Ernst MD
Conference NameISSTA 2004, Proceedings of the 2004 International Symposium on Software Testing and Analysis
Date or Month PublishedJuly 12–14
Conference LocationBoston, MA, USA
Abstract<p>Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It is intended to reduce the time and energy required to keep code well-tested and prevent regression errors from persisting uncaught for long periods of time. This paper reports on a controlled human experiment to evaluate whether students using continuous testing are more successful in completing programming assignments. We also summarize users' subjective impressions and discuss why the results may generalize. </p> <p> The experiment indicates that the tool has a statistically significant effect on success in completing a programming task, but no such effect on time worked. Participants using continuous testing were three times as likely to complete the task before the deadline. Participants using continuous compilation were twice as likely to complete the task, providing empirical support to a common feature in modern development environments. Most participants found continuous testing to be useful and believed that it helped them write better code faster, and 90% would recommend the tool to others. The participants did not find the tool distracting, and intuitively developed ways of incorporating the feedback into their workflow.</p>
DownloadsEclipse plug-in PDF PostScript
Citation KeySaffE2004:ISSTA
Last changed Mon, 2013-06-03 10:27