The new system for running C++ Unit Tests added to VisualGDB 5.2 has out-of-the-box support for CppUTest and GoogleTest framework. But it’s also designed to be easily extendable to support any other test framework through a flexible set of rules defined in XML files. This post will describe how the test frameworks are defined and what can be customized in the framework definitions to support more frameworks.

The test frameworks are located in %LOCALAPPDATA%\VisualGDB\TestFrameworks. Each framework consists of the source files and a TestFramework.xml file describing the framework. To configure VisualGDB to work with any other test framework, you simply need to define the following things in a TestFramework.xml file:

• How to discover unit tests inside the built ELF file
• How to tell the program which unit tests to run
• How to receive the unit test output stream from the program
• How to detect that a test has just reported a failure

Those settings are configured by editing various elements under the Common, Embedded, Linux, AndroidCommandLine or AndroidApp elements in the TestFramework.xml file (settings from the Common element are automatically added to the platform-specific ones).

Discovering Tests

You can tell VisualGDB how to locate the tests inside your binary by editing the TestDiscoverers element. Let’s look into the test discoverer defined for the CppUTest framework:

<TestDiscoverers>
 <TestDiscoverer xsi:type="SymbolBasedScopeDiscoverer">
 <TargetDemangledNames>false</TargetDemangledNames>
 <Discoverers>
 <TestDiscoverer xsi:type="SymbolBasedScopedTestDiscoverer">
 <TargetDemangledNames>true</TargetDemangledNames>
 <TestSymbolRegex>^TEST_$(Scope)_(.*)_TestShell_instance$</TestSymbolRegex>
 <UniqueNameFormat>{0}::{2}</UniqueNameFormat>
 <UserFriendlyNameFormat>{0}::{2}</UserFriendlyNameFormat>
 <LocationSymbolFormat />
 </TestDiscoverer>
 </Discoverers>
 <ScopeSymbolRegex>^externTestGroup(.*)$</ScopeSymbolRegex>
 <ScopeSymbolGroup>1</ScopeSymbolGroup>
 <ExpectedSymbolType>1</ExpectedSymbolType>
 </TestDiscoverer>
</TestDiscoverers>

The two-level structure (SymbolBasedScopedTestDiscoverer inside SymbolBasedScopeDiscoverer) means that VisualGDB will first try to discover scopes (i.e. test groups) and then discover tests inside each test group. For frameworks with flat test structure (e.g. Googletest) only the SymbolBasedScopedTestDiscoverer discoverer is needed.

The discoverers look through the symbols (i.e. functions, methods & global variables) defined inside an ELF file and match them against the regular expressions defined in the ScopeSymbolRegex and TestSymbolRegex elements. Once a matching symbol is found, VisualGDB treats it as a definition of a test and uses the UniqueNameFormat and UserFriendlyNameFormat elements to derive the test name from it. The format is a .Net composite format string where {0} corresponds to the scope name, {1} denotes the entire test symbol name and {2}, {3} and so on denote the groups from the TestSymbolRegex.

Selecting Tests to Run

VisualGDB supports several ways of selecting which tests to run. The most simple one is called ArgumentBasedTestSelection. Let’s look at the definition from the GoogleTest framework:

<TestSelection xsi:type="ArgumentBasedTestSelection">
    <GlobalArgumentTemplate>--gtest_filter=$(PerTestArguments)</GlobalArgumentTemplate>
    <PerTestArgumentTemplate>$(1).$(2)</PerTestArgumentTemplate>
    <PerTestArgumentSeparator>:</PerTestArgumentSeparator>
    <TestIDRegex>(.*)_(.*)</TestIDRegex>
</TestSelection>

When using this method, VisualGDB will simply pass the list of tests to run on the command line and will expect the test program to parse it. The general format for the test selection argument is specified in the GlobalArgumentTemplate element. The $(PerTestArguments) expression will be replaced by a list of tests to run separated by PerTestArgumentSeparator. The TestIDRegex and PerTestArgumentTemplate elements specify how to transform the unique test ID when passing it on the command line.

If the tests are running in a container that does not have a concept of a command line (e.g. Android App or an embedded project), VisualGDB can instead set a breakpoint at a given location and edit the list of tests to run:

<TestSelection xsi:type="HookBasedTestSelection">
    <HookedEntry>SysprogsTestHook_SelectTests</HookedEntry>
    <TestCountExpression>testCount</TestCountExpression>
    <AddressOfTestArrayExpression>pTests</AddressOfTestArrayExpression>
</TestSelection>

Once the test selection breakpoint hits, VisualGDB will expect the framework to have an array of pointers to all tests available in a place pointed by the AddressOfTestArrayExpression expression. It will keep the tests selected for running untouched and will replace the other ones with NULLs. This will only take 2 memory reads and 1 memory write, so it should not take much time even on slow target connections.

The test pointers should EXACTLY match the addresses of the symbols used to discover the tests by the TestDiscoverer so that VisualGDB can identify them without doing any more memory reads. If this is not possible (e.g. the tests are inside a dynamic library), you can use a variation of the hook-based selection that extracts test names:

<TestSelection xsi:type="HookBasedTestSelectionViaNames">
    <HookedEntry>SysprogsTestHook_SelectTests</HookedEntry>
    <TestCountExpression>testCount</TestCountExpression>
    <AddressOfTestArrayExpression>pTests</AddressOfTestArrayExpression>
    <TestNameExpression>{((testing::TestInfo*){&amp;})-&gt;test_case_name_-&gt;_M_dataplus-&gt;_M_p:s}_{((testing::TestInfo*){&amp;})-&gt;name_-&gt;_M_dataplus-&gt;_M_p:s}</TestNameExpression>
</TestSelection>

The TestNameExpression uses C# interpolated string syntax where “{&}” denotes the address of the test and “{#}” denotes the index of the test inside the test pointer array. Let’s look into the different parts of the experssion from the example above: {((testing::TestInfo*){&})->test_case_name_->_M_dataplus->_M_p:s}_{((testing::TestInfo*){&})->name_->_M_dataplus->_M_p:s}.

Obtaining the test output

Once VisualGDB has selected the tests to run, one last thing to do is to define how to read the test output. The test framework should report the test results using a binary protocol described later in this post and the TestFramework.xml file defines how to obtain the test results in that protocol. The easiest way to do that is used by the embedded projects:

 <TestReportChannel xsi:type="SemihostingBasedTestChannel"/>

It simply means that VisualGDB should read the test output data using the Sysprogs Fast Semihosting mechanism on the default channel 0x43.

Linux projects can report the test via pipes (created with the mkfifo command):

<TestReportChannel xsi:type="PipeBasedTestChannelWithEnv">
    <PipeTimestampVariableName>g_SysprogsTestReportTimestamp</PipeTimestampVariableName>
    <EnvironmentVariableName>SYSPROGS_TEST_REPORTING_PIPE</EnvironmentVariableName>
</TestReportChannel>

VisualGDB will create a pipe automatically and pass its path to the application via an environment variable. The PipeTimestampVariableName element defines the name of a global variable used to ensure that all prior output was read from the pipe once a breakpoint in the code is triggered.

Android tests use the Unix local domain sockets instead of pipes to report the test output:

<TestReportChannel xsi:type="LocalSocketTestChannelWithEnv">
    <PipeTimestampVariableName>g_SysprogsTestReportTimestamp</PipeTimestampVariableName>
    <EnvironmentVariableName>SYSPROGS_TEST_REPORTING_SOCKET</EnvironmentVariableName>
    <SocketReadyHook>SysprogsTestHook_ReportingSocketReady</SocketReadyHook>
</TestReportChannel>

VisualGDB will pass the socket name to the Android code and will expect it to open the local domain socket and call the function defined in SocketReadyHook once the socket is ready.

Sysprogs Test Reporting Protocol

The test frameworks should report the test progress and results using a simple binary protocol described below. Each message starts with either 1-byte length (for short packets) or 0xff followed by a 4-byte length (for long packets). The first byte of each packet body (after the packet length) contains the packet type:

The strpTestFailed message can contain zero or more detail objects of the following types:

Normally the test framework only reports error summary or error details and VisualGDB automatically attaches the call stack from the GDB, however the framework can also report the stack explicitly (e.g. if it’s using interpreted languages).

Reference implementation

All test frameworks used by VisualGDB consist of a file called SysprogsTestHooks.cpp and the original test framework with minimal changes that invoke the hooks from that file. If you are modifying your own test framework to work with VisualGDB, you can reuse the SysprogsTestHooks.cpp file to avoid reimplementing the Sysprogs Test Reporting Protocol.  The table below summarizes the functions defined in that file:

You can download the latest VisualGDB version with unit test support on the download page. The sources for the modified test frameworks can be found in our GitHub repository.