| Title | Empirically revisiting the test independence assumption |
| Publication Type | Conference Paper |
| Year of Publication | 2014 |
| Authors | Zhang S, Jalali D, Wuttke J, Muşlu K, Lam W, Ernst MD, Notkin D |
| Conference Name | ISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis |
| Pagination | 385–396 |
| Date or Month Published | July |
| Conference Location | San Jose, CA, USA |
| Abstract | In a test suite, all the test cases should be independent: no test should affect any other test's result, and running the tests in any order should produce the same test results. Techniques such as test prioritization generally assume that the tests in a suite are independent. Test dependence is a little-studied phenomenon. This paper presents five results related to test dependence. \par First, we characterize the test dependence that arises in practice. We studied 96 real-world dependent tests from 5 issue tracking systems. Our study shows that test dependence can be hard for programmers to identify. It also shows that test dependence can cause non-trivial consequences, such as masking program faults and leading to spurious bug reports. \par Second, we formally define test dependence in terms of test suites as ordered sequences of tests along with explicit environments in which these tests are executed. We formulate the problem of detecting dependent tests and prove that a useful special case is NP-complete. \par Third, guided by the study of real-world dependent tests, we propose and compare four algorithms to detect dependent tests in a test suite. \par Fourth, we applied our dependent test detection algorithms to 4 real-world programs and found dependent tests in each human-written and automatically-generated test suite. \par Fifth, we empirically assessed the impact of dependent tests on five test prioritization techniques. Dependent tests affect the output of all five techniques; that is, the reordered suite fails even though the original suite did not. |
| Downloads | https://homes.cs.washington.edu/~mernst/pubs/test-independence-issta2014... PDF
|
| Citation Key | ZhangJWMLEN2014 |
