Automatically Generating Precise Oracles from Structured Natural Language Specifications
by Manish Motwani, Yuriy Brun
Abstract:

Software specifications often use natural language to describe the desired behavior, but such specifications are difficult to verify automatically. We present Swami, an automated technique that extracts test oracles and generates executable tests from structured natural language specifications. Swami focuses on exceptional behavior and boundary conditions that often cause field failures but that developers often fail to manually write tests for. Evaluated on the official JavaScript specification (ECMA-262), 98.4% of the tests Swami generated were precise to the specification. Using Swami to augment developer-written test suites improved coverage and identified 1 previously unknown defect and 15 missing JavaScript features in Rhino, 1 previously unknown defect in Node.js, and 18 semantic ambiguities in the ECMA-262 specification.

Citation:
Manish Motwani and Yuriy Brun, Automatically Generating Precise Oracles from Structured Natural Language Specifications, in Proceedings of the 41st International Conference on Software Engineering (ICSE), 2019, pp. 188–199.
Bibtex:
@inproceedings{Motwani19icse,
  author = {Manish Motwani and Yuriy Brun},
  title =
  {\href{http://people.cs.umass.edu/brun/pubs/pubs/Motwani19icse.pdf}{Automatically 
  Generating Precise Oracles from Structured Natural Language Specifications}},
  booktitle = {Proceedings of the 41st International Conference on Software
  Engineering (ICSE)},
  venue = {ICSE},
  pages = {188--199},
  address = {Montreal, QC, Canada},
  month = {May},
  date = {29--31},
  year = {2019},
  doi = {10.1109/ICSE.2019.00035},

  accept = {$\frac{109}{529} \approx 21\%$},
  note = {ACM artifact badges granted: 
  \href{https://www.acm.org/publications/policies/artifact-review-and-badging-current}{\raisebox{-.75ex}{\includegraphics[height=2.5ex]{ACMArtifactAvailable}}~Artifact Available, 
  \raisebox{-.75ex}{\includegraphics[height=2.5ex]{ACMArtifactReusable}}~Artifact Reusable}. 
  \href{https://doi.org/10.1109/ICSE.2019.00035}{DOI: 10.1109/ICSE.2019.00035}},

  abstract = {<p>Software specifications often use natural language to describe the desired
  behavior, but such specifications are difficult to verify automatically. We
  present Swami, an automated technique that extracts test oracles and
  generates executable tests from structured natural language specifications.
  Swami focuses on exceptional behavior and boundary conditions that often
  cause field failures but that developers often fail to manually write tests
  for. Evaluated on the official JavaScript specification (ECMA-262), 98.4% of
  the tests Swami generated were precise to the specification. Using Swami to
  augment developer-written test suites improved coverage and identified 1
  previously unknown defect and 15 missing JavaScript features in Rhino, 1
  previously unknown defect in Node.js, and 18 semantic ambiguities in the
  ECMA-262 specification.
</p>},

  fundedBy = {NSF CCF-1453474, NSF CNS-1513055, NSF CCF-1763423},
}