Our Project

Unit testing is a software testing methodology that allows programmers to verify the behavior of their code. A typical unit test provides an input to the individual units of code, and ensures the software behaves as it should. Unit testing suffers from a major flaw: programmers are likely to test the conditions they expect to cause failure. The expectations of a programmer may not reflect system behavior in production.

Enter DeepState, a software testing framework for C/C++ that allows users to write unit tests containing generic values. Instead of writing unit tests that test at most a handful of values, DeepState generates thousands of values to test the functions of software. This saves programmers time writing tests, and also results in more robust, reliable software.

However, existing DeepState tests are reliant upon the DeepState runtime library and files that contain the values that were generated by DeepState. Thus, distributing unit tests written in DeepState requires the distribution of (i) DeepState itself, which most programmers will not have on their system; and (ii) the dozens of files that DeepState generated for the unit tests.

Because of these limitations, we have been tasked with creating an extension to the DeepState command-line interface that enables the generation of standalone unit tests. These unit tests must be truly standalone, meaning all references to the DeepState runtime must be replaced with the syntax of a testing framework that the user chooses, and the values that are generated by DeepState must be inserted into the source code of the test itself.

What Our Software Does

In September and October of 2019, we sat down with our client, NAU’s Professor Alex Groce, to figure out what was required. From these meetings, we have designed a software product that must:

  • Be an extension to the existing DeepState command-line interface that allows the user to specify the unit test they want to be translated.
  • Allow users to specify the unit testing framework they wish to translate their unit tests to.
  • Translate unit tests to the testing framework specified by the user. This translated test will contain no references to the DeepState library.
  • Insert the values generated by DeepState into the source of the standalone unit test

Over the course of the past 6 months, we have created software that meets these requirements. The following figure provides an overview of the input given to our system, the output it produces, and the technologies our system uses to perform the translation.

System Overview

Our Solution

Our software is an extension to the DeepState command-line interface that enables the creation of standalone tests. To implement implement this solution, we have designed 3 modules that facilitate the creation of these standalone tests: the Translation Engine, the Binary Parser, and the File Assembler.

Translation Engine

The Translation Engine’s job is to deconstruct the DeepState test file. The Translation Engine will take in the file to be translated, then break it down into a parse tree utilizing ANTLR4. This parse tree will then be passed to the File Assembler for reassembly of the standalone test file.

Binary Parser

The Binary Parser’s job is to make sense of the unstructured binary data generated by DeepState. The Binary Parser will take in an unstructured binary data file generated by DeepState and read the data contained in that file. The File Assembler will then utilize the Binary Parser to “fill in the blanks” where symbolic values once were.

File Assembler

The File Assembler’s job is to reconstruct the file using the parse tree given by the Translation Engine, while also inserting values from the Binary Parser. This reconstructed file will then be outputted as a C++ file. The standalone file will be compilable and contain the generated values from DeepState.

The following image illustrates the integration of our modules.

System Integration

Technologies

Our project uses a few key technologies:

  • DeepState: Because DeepState offers methods for creating and running unit tests, and reading values from the binary files that DeepState generates, GenTest uses the the DeepState API to interact with fuzzers and to retrieve the binary files generated by DeepState.

  • ANTLR4: We use ANTLR4 to parse the input C++ files containing unit tests that contain functions built for the DeepState testing framework. Instead of creating a custom parser that would be error-prone and difficult to maintain, we generate a custom grammar that ANTLR4 uses to generate a parser. This alleviates the burden of writing a custom parser, and provides our softwre with the reliability of an existing, tested parser generator.

Code Base

Our code base is still under development, but an archive containing our complete code base and documentation will be located here upon completion of our project.