{"id":8591,"date":"2016-07-22T04:20:57","date_gmt":"2016-07-22T03:20:57","guid":{"rendered":"https:\/\/sysprogs.com\/w\/?p=8591"},"modified":"2016-07-22T04:20:57","modified_gmt":"2016-07-22T03:20:57","slug":"extending-the-visualgdb-test-system","status":"publish","type":"post","link":"https:\/\/sysprogs.com\/w\/extending-the-visualgdb-test-system\/","title":{"rendered":"Extending the VisualGDB Test System"},"content":{"rendered":"<p>The\u00a0<a href=\"https:\/\/sysprogs.com\/w\/the-new-unit-test-support-in-visualgdb-5-2\/\">new system for running C++ Unit Tests<\/a> added to VisualGDB 5.2 has out-of-the-box support for CppUTest and GoogleTest framework. But it&#8217;s also designed to be easily extendable to support any other test framework through a flexible set of rules defined in XML files.\u00a0This post\u00a0will describe\u00a0how the test frameworks are defined and\u00a0what can be customized in the framework definitions to support more frameworks.<\/p>\n<p><!--more--><\/p>\n<p>The\u00a0test frameworks are located in %LOCALAPPDATA%\\VisualGDB\\TestFrameworks. Each framework consists of the source files and a <strong>TestFramework.xml<\/strong> file describing the\u00a0framework.\u00a0To\u00a0configure VisualGDB to work with any other test framework, you simply need to define the following things in a TestFramework.xml file:<\/p>\n<ul>\n<li>How to\u00a0discover unit tests inside the\u00a0built ELF file<\/li>\n<li>How to tell the program which unit tests to run<\/li>\n<li>How to receive the unit test output stream from the program<\/li>\n<li>How to detect that a test has just reported\u00a0a failure<\/li>\n<\/ul>\n<p>Those settings are configured by editing various\u00a0elements under the <strong>Common<\/strong>, <strong>Embedded<\/strong>, <strong>Linux<\/strong>, <strong>AndroidCommandLine<\/strong> or <strong>AndroidApp<\/strong> elements in the <strong>TestFramework.xml<\/strong> file (settings from\u00a0the Common element are automatically added to the platform-specific ones).<\/p>\n<h3>Discovering Tests<\/h3>\n<p>You can tell VisualGDB how to locate the tests inside your binary by editing the TestDiscoverers element. Let&#8217;s look into\u00a0the test discoverer defined for\u00a0the CppUTest framework:<\/p>\n<pre class=\"\">&lt;TestDiscoverers&gt;\r\n &lt;TestDiscoverer xsi:type=\"SymbolBasedScopeDiscoverer\"&gt;\r\n &lt;TargetDemangledNames&gt;false&lt;\/TargetDemangledNames&gt;\r\n &lt;Discoverers&gt;\r\n &lt;TestDiscoverer xsi:type=\"SymbolBasedScopedTestDiscoverer\"&gt;\r\n &lt;TargetDemangledNames&gt;true&lt;\/TargetDemangledNames&gt;\r\n &lt;TestSymbolRegex&gt;^TEST_$(Scope)_(.*)_TestShell_instance$&lt;\/TestSymbolRegex&gt;\r\n &lt;UniqueNameFormat&gt;{0}::{2}&lt;\/UniqueNameFormat&gt;\r\n &lt;UserFriendlyNameFormat&gt;{0}::{2}&lt;\/UserFriendlyNameFormat&gt;\r\n &lt;LocationSymbolFormat \/&gt;\r\n &lt;\/TestDiscoverer&gt;\r\n &lt;\/Discoverers&gt;\r\n &lt;ScopeSymbolRegex&gt;^externTestGroup(.*)$&lt;\/ScopeSymbolRegex&gt;\r\n &lt;ScopeSymbolGroup&gt;1&lt;\/ScopeSymbolGroup&gt;\r\n &lt;ExpectedSymbolType&gt;1&lt;\/ExpectedSymbolType&gt;\r\n &lt;\/TestDiscoverer&gt;\r\n&lt;\/TestDiscoverers&gt;<\/pre>\n<p>The two-level structure (<strong>SymbolBasedScopedTestDiscoverer<\/strong> inside\u00a0<strong>SymbolBasedScopeDiscoverer<\/strong>) means that VisualGDB will first try to discover scopes (i.e. test groups) and then discover tests inside each test group. For frameworks with flat test structure (e.g. Googletest) only the\u00a0<strong>SymbolBasedScopedTestDiscoverer\u00a0<\/strong>discoverer is needed.<\/p>\n<p>The discoverers look through the symbols (i.e. functions, methods &amp; global variables) defined inside an ELF file and match them against the regular expressions defined in the <strong>ScopeSymbolRegex<\/strong> and <strong>TestSymbolRegex<\/strong> elements. Once a matching symbol is found, VisualGDB\u00a0treats it as a definition of a test and uses the\u00a0<strong>UniqueNameFormat<\/strong> and <strong>UserFriendlyNameFormat<\/strong>\u00a0elements to derive the test name from it. The format is a .Net composite format string where {0} corresponds to the scope name, {1} denotes\u00a0the entire test symbol name and {2}, {3} and so on denote the groups from the <strong>TestSymbolRegex<\/strong>.<\/p>\n<h3>Selecting\u00a0Tests to Run<\/h3>\n<p>VisualGDB supports several ways of\u00a0selecting\u00a0which tests to run. The\u00a0most simple one is called ArgumentBasedTestSelection. Let&#8217;s\u00a0look at the definition from the GoogleTest framework:<\/p>\n<pre class=\"\">&lt;TestSelection xsi:type=\"ArgumentBasedTestSelection\"&gt;\r\n    &lt;GlobalArgumentTemplate&gt;--gtest_filter=$(PerTestArguments)&lt;\/GlobalArgumentTemplate&gt;\r\n    &lt;PerTestArgumentTemplate&gt;$(1).$(2)&lt;\/PerTestArgumentTemplate&gt;\r\n    &lt;PerTestArgumentSeparator&gt;:&lt;\/PerTestArgumentSeparator&gt;\r\n    &lt;TestIDRegex&gt;(.*)_(.*)&lt;\/TestIDRegex&gt;\r\n&lt;\/TestSelection&gt;<\/pre>\n<p>When using\u00a0this method, VisualGDB will simply\u00a0pass the list of tests\u00a0to run on the command line and\u00a0will expect the test\u00a0program to parse it. The general format for the test\u00a0selection argument is specified in the\u00a0<strong>GlobalArgumentTemplate<\/strong> element. The\u00a0<strong>$(PerTestArguments)<\/strong>\u00a0expression will\u00a0be replaced by a list of tests to run separated by\u00a0<strong>PerTestArgumentSeparator<\/strong>.\u00a0The <strong>TestIDRegex<\/strong> and <strong>PerTestArgumentTemplate<\/strong> elements specify how to\u00a0transform the unique test ID when\u00a0passing it on the command line.<\/p>\n<p>If the\u00a0tests are running in a container that does not have a concept of a command line (e.g. Android App or an embedded project), VisualGDB can\u00a0instead set a breakpoint at a given location and edit the list of tests to run:<\/p>\n<pre class=\"\">&lt;TestSelection xsi:type=\"HookBasedTestSelection\"&gt;\r\n    &lt;HookedEntry&gt;SysprogsTestHook_SelectTests&lt;\/HookedEntry&gt;\r\n    &lt;TestCountExpression&gt;testCount&lt;\/TestCountExpression&gt;\r\n    &lt;AddressOfTestArrayExpression&gt;pTests&lt;\/AddressOfTestArrayExpression&gt;\r\n&lt;\/TestSelection&gt;<\/pre>\n<p>Once the test selection breakpoint hits, VisualGDB will expect the framework to\u00a0have an array of pointers to all tests\u00a0available in a place pointed by the\u00a0<strong>AddressOfTestArrayExpression<\/strong> expression. It will keep the tests selected for running untouched and will replace the other ones with NULLs.\u00a0This will only take 2 memory reads and 1 memory write, so\u00a0it should not take much time even on slow target\u00a0connections.<\/p>\n<p>The test pointers should EXACTLY match the addresses of the symbols used to discover the tests by the\u00a0<strong>TestDiscoverer<\/strong> so that VisualGDB can\u00a0identify them without\u00a0doing any more memory reads. If\u00a0this is not possible (e.g. the tests are inside a dynamic library), you can use a variation of the hook-based selection that\u00a0extracts test names:<\/p>\n<pre class=\"\">&lt;TestSelection xsi:type=\"HookBasedTestSelectionViaNames\"&gt;\r\n    &lt;HookedEntry&gt;SysprogsTestHook_SelectTests&lt;\/HookedEntry&gt;\r\n    &lt;TestCountExpression&gt;testCount&lt;\/TestCountExpression&gt;\r\n    &lt;AddressOfTestArrayExpression&gt;pTests&lt;\/AddressOfTestArrayExpression&gt;\r\n    &lt;TestNameExpression&gt;{((testing::TestInfo*){&amp;amp;})-&amp;gt;test_case_name_-&amp;gt;_M_dataplus-&amp;gt;_M_p:s}_{((testing::TestInfo*){&amp;amp;})-&amp;gt;name_-&amp;gt;_M_dataplus-&amp;gt;_M_p:s}&lt;\/TestNameExpression&gt;\r\n&lt;\/TestSelection&gt;<\/pre>\n<p>The TestNameExpression uses\u00a0C# interpolated string syntax\u00a0where &#8220;{&amp;}&#8221; denotes the address of the test and &#8220;{#}&#8221; denotes the index of the test inside the test pointer array. Let&#8217;s look into the different parts of the\u00a0experssion from the example above:\u00a0<span style=\"color: #ff0000;\">{((testing::TestInfo*)<span style=\"color: #3366ff;\">{&amp;}<\/span>)-&gt;test_case_name_-&gt;_M_dataplus-&gt;_M_p<span style=\"color: #339966;\">:s<\/span>}<\/span>_<span style=\"color: #ff0000;\">{((testing::TestInfo*)<span style=\"color: #3366ff;\">{&amp;}<\/span>)-&gt;name_-&gt;_M_dataplus-&gt;_M_p<span style=\"color: #339966;\">:s<\/span>}<\/span>.<\/p>\n<table>\n<tbody>\n<tr>\n<td>Part<\/td>\n<td>Meaning<\/td>\n<\/tr>\n<tr>\n<td>{&amp;}<\/td>\n<td>Will be replaced by the address of\u00a0the test which name is being queried<\/td>\n<\/tr>\n<tr>\n<td>:s<\/td>\n<td>Means that the\u00a0expression before will be formatted as a string<\/td>\n<\/tr>\n<tr>\n<td><span style=\"color: #ff0000;\">{&#8230;}<\/span>_<span style=\"color: #ff0000;\">{&#8230;}<\/span><\/td>\n<td>Defines the overall format\u00a0for building the test name<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Obtaining the test output<\/h3>\n<p>Once\u00a0VisualGDB has\u00a0selected\u00a0the\u00a0tests to run,\u00a0one last thing to do is to define how to read the test output. The\u00a0test framework should report the test results using a binary protocol described later in this post and the TestFramework.xml file defines how to obtain the test results in that protocol. The easiest way\u00a0to do that is used by the embedded projects:<\/p>\n<pre class=\"\"> &lt;TestReportChannel xsi:type=\"SemihostingBasedTestChannel\"\/&gt;<\/pre>\n<p>It simply means that VisualGDB should read the test output data using the Sysprogs Fast Semihosting mechanism on the default channel 0x43.<\/p>\n<p>Linux projects can report the test via pipes (created with the <strong>mkfifo<\/strong> command):<\/p>\n<pre class=\"\">&lt;TestReportChannel xsi:type=\"PipeBasedTestChannelWithEnv\"&gt;\r\n    &lt;PipeTimestampVariableName&gt;g_SysprogsTestReportTimestamp&lt;\/PipeTimestampVariableName&gt;\r\n    &lt;EnvironmentVariableName&gt;SYSPROGS_TEST_REPORTING_PIPE&lt;\/EnvironmentVariableName&gt;\r\n&lt;\/TestReportChannel&gt;<\/pre>\n<p>VisualGDB will create a pipe automatically and\u00a0pass its path to the application via an environment variable. The\u00a0<strong>PipeTimestampVariableName<\/strong> element defines\u00a0the name of a global variable used to\u00a0ensure that all prior\u00a0output was read from the pipe once a breakpoint in the code is triggered.<\/p>\n<p>Android\u00a0tests use the Unix\u00a0local domain sockets\u00a0instead of pipes to report the test output:<\/p>\n<pre class=\"\">&lt;TestReportChannel xsi:type=\"LocalSocketTestChannelWithEnv\"&gt;\r\n    &lt;PipeTimestampVariableName&gt;g_SysprogsTestReportTimestamp&lt;\/PipeTimestampVariableName&gt;\r\n    &lt;EnvironmentVariableName&gt;SYSPROGS_TEST_REPORTING_SOCKET&lt;\/EnvironmentVariableName&gt;\r\n    &lt;SocketReadyHook&gt;SysprogsTestHook_ReportingSocketReady&lt;\/SocketReadyHook&gt;\r\n&lt;\/TestReportChannel&gt;<\/pre>\n<p>VisualGDB will pass the socket name to the Android code\u00a0and will expect it to open the local domain socket and call the function defined in\u00a0<strong>SocketReadyHook<\/strong> once the socket is ready.<\/p>\n<h3>Sysprogs Test Reporting Protocol<\/h3>\n<p>The test frameworks should report the test progress and results using a simple binary protocol described\u00a0below. Each message starts with either 1-byte length (for short packets) or 0xff followed by a 4-byte length (for long packets). The first byte of each packet body (after the packet length)\u00a0contains the packet type:<\/p>\n<table>\n<tbody>\n<tr>\n<td>Packet type<\/td>\n<td>Followed by<\/td>\n<td>Meaning<\/td>\n<\/tr>\n<tr>\n<td>strpTestStartingByID (1)<\/td>\n<td>&lt;32-bit or 64-bit test ID&gt;<\/td>\n<td>A test is about to start. The ID should match the symbol address discovered by the test discoverer.<\/td>\n<\/tr>\n<tr>\n<td>strpTestStartingByName (2)<\/td>\n<td>Test name (UTF8)<\/td>\n<td>A test is about to start. The test name should match the unique test ID discovered by the test discoverer.<\/td>\n<\/tr>\n<tr>\n<td>strpTestEnded (3)<\/td>\n<td>Nothing<\/td>\n<td>A test has just ended (a test is considered succeeded\u00a0unless it reported a failure before\u00a0the end)<\/td>\n<\/tr>\n<tr>\n<td>strpOutputMessage (4)<\/td>\n<td>&lt;1-byte severity&gt; &lt;message text&gt;<\/td>\n<td>An arbitrary text message from the test.\u00a0The severity can be either 0 (Info), 1 (Warning) or 2 (Error). A value of 2 automatically flags the test as failed.<\/td>\n<\/tr>\n<tr>\n<td>strpTestFailed (5)<\/td>\n<td>Zero or more test objects (see below)<\/td>\n<td>The packet provides structured information on a test failure<\/td>\n<\/tr>\n<tr>\n<td>strpTimestamp (6)<\/td>\n<td>&lt;32-bit or 64-bit timestamp&gt;<\/td>\n<td>Synchronizes the test output with the current program state. When a\u00a0breakpoint inside the program\u00a0is triggered, VisualGDB will\u00a0read the value of the <strong>PipeTimestampVariableName <\/strong>variable and wait for a timestamp packet with the same value before it\u00a0determines\u00a0which test is the current one.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The <strong>strpTestFailed<\/strong> message can\u00a0contain\u00a0zero\u00a0or more detail objects of the following types:<\/p>\n<table>\n<tbody>\n<tr>\n<td>Test object type<\/td>\n<td>Followed by<\/td>\n<td>Meaning<\/td>\n<\/tr>\n<tr>\n<td>totErrorSummary (0)<\/td>\n<td>NULL-terminated UTF-8\u00a0message<\/td>\n<td>Summary of the error<\/td>\n<\/tr>\n<tr>\n<td>totErrorDetails (1)<\/td>\n<td>NULL-terminated UTF-8 message<\/td>\n<td>Detailed description of the error<\/td>\n<\/tr>\n<tr>\n<td>totCodeLocation (2)<\/td>\n<td>NULL-terminated file path, 32-bit line number<\/td>\n<td>Location of the error<\/td>\n<\/tr>\n<tr>\n<td>totCallStack (3)<\/td>\n<td>&lt;8-bit frame count&gt; array of (&lt;NULL-terminated path&gt;, &lt;32-bit line number&gt;)<\/td>\n<td>Call stack of the error<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Normally the test\u00a0framework only reports error summary or error details and VisualGDB automatically attaches the call\u00a0stack from\u00a0the GDB, however the framework can also report the stack explicitly (e.g. if it&#8217;s using interpreted languages).<\/p>\n<h3>Reference implementation<\/h3>\n<p>All\u00a0test frameworks used by VisualGDB\u00a0consist of\u00a0a file called SysprogsTestHooks.cpp\u00a0and the original test framework with minimal changes that invoke the hooks from that file.\u00a0If you are modifying your own test framework to work with VisualGDB, you can reuse the SysprogsTestHooks.cpp file to avoid reimplementing the Sysprogs Test Reporting Protocol. \u00a0The table below summarizes the functions defined in that file:<\/p>\n<table>\n<tbody>\n<tr>\n<td>Function<\/td>\n<td>Meaning<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_SelectTests()<\/td>\n<td>Used together with\u00a0the hook-based test selection procedure. Not needed if the test framework can parse command-line arguments.<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_TestStarting()<\/td>\n<td>Reports that a test is starting (if its numeric ID is known).<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_TestStartingEx()<\/td>\n<td>Reports that a test is starting (uses the name instead of the ID).<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_TestEnded()<\/td>\n<td>Reports that a test has just ended.<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_OutputMessage()<\/td>\n<td>Outputs an arbitrary text message.<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_TestFailed()<\/td>\n<td>Reports a test failure and provides details on it.<\/td>\n<\/tr>\n<tr>\n<td>SysprogsTestHook_TestsCompleted()<\/td>\n<td>Called by the test framework when ALL tests are completed. VisualGDB will\u00a0terminate the debugged process in response.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>You can download\u00a0the latest\u00a0VisualGDB version with unit test support\u00a0on the <a href=\"http:\/\/visualgdb.com\/download\/\">download page<\/a>. The sources for the modified test frameworks can be found in <a href=\"https:\/\/github.com\/sysprogs\/BSPTools\/tree\/master\/TestFrameworkGenerator\">our GitHub repository<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The\u00a0new system for running C++ Unit Tests added to VisualGDB 5.2 has out-of-the-box support for CppUTest and GoogleTest framework. But it&#8217;s also designed to be easily extendable to support any other test framework through a flexible set of rules defined in XML files.\u00a0This post\u00a0will describe\u00a0how the test frameworks are defined and\u00a0what can be customized in &hellip; <a href=\"https:\/\/sysprogs.com\/w\/extending-the-visualgdb-test-system\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Extending the VisualGDB Test System<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[1],"tags":[216],"_links":{"self":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/8591"}],"collection":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/comments?post=8591"}],"version-history":[{"count":6,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/8591\/revisions"}],"predecessor-version":[{"id":8689,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/posts\/8591\/revisions\/8689"}],"wp:attachment":[{"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/media?parent=8591"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/categories?post=8591"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sysprogs.com\/w\/wp-json\/wp\/v2\/tags?post=8591"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}