| Abstract | In a test suite, all the test cases should be independent: no test should affect any other test's result, and running the tests in any order should produce the same test results. Test independence is important so that tests behave consistently as intended by the developers. In addition, techniques such as test prioritization assume that the tests in a suite are independent, but they do not justify that assumption. Test dependence is a little-studied phenomenon. This paper presents five results related to test dependence. \par First, we characterize the test dependence that arises in practice. We studied 96 real-world dependent tests from 5 issue tracking systems. Our study shows that test dependence can be hard for programmers to identify. It also shows that test dependence can cause non-trivial consequences, such as masking program faults and leading to spurious bug reports. \par Second, we formally define test dependence in terms of test suites as ordered sequences of tests along with explicit environments in which these tests are executed. We formulate the problem of detecting dependent tests and prove that a useful special case is NP-complete. \par Third, guided by the study of real-world dependent tests, we propose and compare three algorithms to detect dependent tests in a test suite. \par Fourth, we applied our dependent test detection algorithms to 4 real-world programs and found dependent tests in each human-written and automatically-generated test suite. \par Fifth, we empirically assessed the impact of dependent tests on five test prioritization techniques and found that dependent tests affect the output of \textitall five techniques. |
