Unittest support for imported projects

[Fabian]

Hi,

we’re using VisualGDB for imported projects using the “Import a CMake project with Advanced CMake Project Subsystem” method.

Is it in principle possible to use the Test Explorer to detect GoogleTest unittests using this method?

If yes, how would we achieve this? We’re currently not using the find_bsp/ find_test_framework functions.

 

Thanks and Regards

[support]

Hi,

Sure, you can do that. In principle, you would need to have 3 things:

1. The project should reference the unit test framework via VisualGDB Project Properties -> Unit Tests. Even if you don’t end up using the actual framework, simply having it referenced there will make VisualGDB look for tests in the project.
2. The binary produced by the project should actually contain the test symbols usually produced by GoogleTest. You can search for TestDiscoverers  in %LOCALAPPDATA%\VisualGDB\TestFrameworks\com.sysprogs.unittest.googletest\TestFramework.xml to get a basic idea of what VisualGDB is looking for.
3. The project’s main() function should actually pass control to the unit test framework, and the framework should contain our patches for selecting the tests via debugger, and for reporting test results via semihosting.

The easiest way to do it without using find_bsp/find_test_framework is to create a simple test project using the wizard, see what exact files (with what parameters) are included in it, and then reconstruct it in your own project.

[Fabian]

Hi,

thanks for your reply.

I managed to get a minimal cmake project running by including the %LOCALAPPDATA%\VisualGDB\TestFrameworks\com.sysprogs.unittest.googletest\Platforms sources into the build.

Unfortunately the Test Explorer cannot evaluate the test result (see attachment).

The pipe within TestOutputPipe is created successfully.

Any additional code I should include?

 

Thanks and Regards

 

Attachments:

You must be logged in to view attached files.

[support]

Hi,

If the main() function calls the same entry point from the test framework, it should work just fine.

You should be able to get more details by opening View->Other Windows->VisualGDB Diagnostics Console, or by running test manually via VisualGDB.exe /runtests with the following environment variables set:

• VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT=1
• VISUALGDB_VERBOSE_OUTPUT=1

If nothing helps, you can also try a black box approach:

1. Disable automatic project rebuilds.
2. Get 2 very similar proof-of-concept projects, one using our framework properly and another one having the files added manually.
3. Make sure the first one works.
4. Replace the binary file produced by the first project with the second one and retry running it.
5. If it still works, the problem is caused by something in the project settings. You can try comparing the .vgdbsettings files from both projects.
6. If the second binary copied in the first project doesn’t work, something inside the binary is off. You can try running objdump -t on both binaries to get the list of all symbols, sorting them by names, and comparing them via a diff tool. Any obvious differences (e.g. missing symbols or significantly different function sizes) should point to the root cause.

[Fabian]

Hi,

I just realized that you patched the gtest/gmock library to include the test hooks.

After using the gtest/gmock version from %LOCALAPPDATA%\VisualGDB\TestFrameworks\com.sysprogs.unittest.googletest\ test execution reports correctly to the Test Explorer.

Is there a way avoid having to include the patched gtest/gmock library and use a vanilla instead?

Our current just use case just involves a “Simulation” (Windows-based) platform, no on-target testing.

 

Another question I noticed working with the current VisualGDB version (v6.0R3):

Configuration of a CMake based VisualGDB project using CMake Presets currently doesn’t allow me to create a “Simulation” platform.
Of course, a native CMake Preset defining the Host as target platform results in the expected compilation result (host binaries), but from VisualGDB’s perspective it’s still an Embedded target.
Debugging of such an executable is not working, assuming a target debugging would be involved.

Will the CMake Presets support extended in the future or should we create a native C++/CMake VisualStudio project instead in this case?

Thanks and Regards

[Fabian]

Another thing:

Trying to integrate the unittests from the minimal cmake project (where test discovery and execution worked) into our main project did not work directly.
I followed your suggestions and compared the binaries and .vgdbcmake files, but they matched.
What was different was the SelectedCMakeTarget in .visualgdb\VisualGDBCache\xyz-Debug-Simulation\xyz.vgdbtestcontainer.

It seems only the unittests of the target marked as “Startup Target” are being discovered successfully.

In our project, we define multiple targets containing unitests. Having to select every target on its own for test execution is not very convenient.

[support]

Hi,

The VisualGDB’s unit test logic was designed to handle many different device types that don’t always have an stdout stream. So, we patched the test frameworks to output the test results in a machine-readable format into a separate channel, independent from the main output, if any. The task of translating the test results into a machine-readable format is done on the target by the code in our patch. If you would like to eliminate the patch, you would need to create a separate proxy application that would read the test results from stdout, parse them via regexes, and forward them back to VisualGDB via the VisualGDB-specific format. If you are willing to implement it, we can add a setting for running an external test result translator, but using the patched versions of the test frameworks could be just easier. The patches are very straight-forward, can be easily applied to other test framework versions, and don’t change the test framework license.

You can use the CMake presets together with the simulation platforms, however it would involve manually editing some configuration files. First of all, please try adding a simulation platform to a regular Advanced CMake project and take a note of the changes it does to the .vgdbcmake file: it will add a new XML node in Configurations/VisualGDBConfiguration[PlatformName=”Simulation”] overriding the project settings (Embedded -> MinGW) and Debug Settings (Embedded Debug -> MinGW Debug). So, whenever you select the Simulation platform, VisualGDB will replace the project and debug settings with the ones from the Configuration node.

When using CMake presets, the Platform is fixed to the default one, and the preset names are mapped to configuration names. So, you can copy the Configuration element from the non-preset project into the same location in the preset-based .vgdbcmake file, and replace the <PlatformName>Simulation</PlatformName> with <Name>YourPresetName</Name>. This will override the settings for that preset just like it previously did for the Simulation platform.

The multiple unit test targets are indeed not supported yet. We will look into adding them and will post an update here in the next couple of weeks.

[Fabian]

Hi,

thank you for the explanations.

Sticking with patching the test framework for the moment seems to be most efficient.

We’ll be looking forward for multiple test target support.

 

Best Regards

[support]

Hi,

OK, we have rechecked the test discovery logic for the simulation platform and it should be possible to get multiple test targets working without any modifications on the VisualGDB side.

First of all, please try cloning this example and duplicating the add_bsp_based_executable() statement to add another target with a different name, but same sources. Then build the solution (simulation platform). Test Explorer should now show 2 sets of tests.

To replicate it in your solution, you would need to manually define a couple of cache variables via set(… CACHE INTERNAL “”)

1. For each executable, set <executable name>_TEST_FRAMEWORK_TARGET to the name of the static library target that corresponds to the test framework.
2. For the actual test framework, set <static library name>_TEST_FRAMEWORK_ID to com.sysprogs.unittest.googletest

You can check the variables defined by our example project by viewing the cache-v2-<id>.json file in <project directory>\.visualgdb\VisualGDBCache\EmbeddedSimulationDemo-Debug-Simulation\CodeModelCache. If your project manages to define the same cache variables, VisualGDB should be able to locate and discover all test targets and not just the startup one.

[Fabian]

Hi,

that solution is working for us. All test targets are listed in Test Explorer now.

Thank you!

 

Best Regards

• This reply was modified 1 month, 1 week ago by Fabian.
• This reply was modified 1 month, 1 week ago by Fabian.

[Author]

Posts