| Title | Comparing developer-provided to user-provided tests for fault localization and automated program repair |
| Publication Type | Conference Paper |
| Year of Publication | 2018 |
| Authors | Just R, Parnin C, Drosos I, Ernst MD |
| Conference Name | ISSTA 2018, Proceedings of the 2018 International Symposium on Software Testing and Analysis |
| Pagination | 287-297 |
| Date or Month Published | July |
| Conference Location | Amsterdam, Netherlands |
| Abstract | To realistically evaluate a software testing or debugging technique, it must be run on defects and tests that are characteristic of those a developer would encounter in practice. For example, to determine the utility of a fault localization or automated program repair technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fixes. Although such a methodology uses real tests, it may not use tests that are characteristic of the information a developer or tool would have in practice. The tests that a developer commits \emphafter fixing a defect may encode more information than was available to the developer when initially diagnosing the defect. \par This paper compares, both quantitatively and qualitatively, the developer-provided tests committed along with fixes (as found in the version control repository) versus the user-provided tests extracted from bug reports (as found in the issue tracker). It provides evidence that developer-provided tests are more targeted toward the defect and encode more information than user-provided tests. For fault localization, developer-provided tests overestimate a technique's ability to rank a defective statement in the list of the top-n most suspicious statements. For automated program repair, developer-provided tests overestimate a technique's ability to (efficiently) generate correct patches–-user-provided tests lead to fewer correct patches and increased repair time. This paper also provides suggestions for improving the design and evaluation of fault localization and automated program repair techniques. |
| Downloads | https://docs.google.com/presentation/d/1vdkSkkue-jsk5VdUtcXwYIZ0_2bJlfBv... slides (Google Slides)
https://github.com/rjust/defects4j data
|
| Citation Key | JustPDE2018 |
