Unit Testing

[bbr]

Hello,

we are currently evaluating how VisualGDB could support us doing unit tests for our next embedded project. In the past we have used Ceedling which works good but lacks a IDE integration. What we really want to keep is one test executable per C module while mocking all dependencies of that module. On the other hand we are willing to go without some other features of Ceedling, like the automated mock generation.

Like we did in the past, we are not interested in running any tests on real hardware. Therefore I tried the Creating Simulation Platforms to Test Embedded Code without Hardware tutorial. The tutorial worked well but I still have some concerns. First of all the Test Explorer window does refresh only once after loading the solution. Which makes adding or removing tests as well as editing test names unbearable. Closing and opening the solution just to add a unit test is an absolute show stopper.

Also when configuring unit tests only for simulation like it is shown in the tutorial you always have to switch between configurations to use the application or the tests. This feels quite unintuitive and annoying. Is there a way to avoid the necessity of the configuration changes?

The next concern is related to the supported test frameworks. All of them are in C++ while we would like the have our tests in C like the application. (I know it is possible to get it working by extern “C”.) Also I do not understand why not all test frameworks are not supported for embedded projects, even if tests are not run on hardware. Maybe you have some background information how the unit test frameworks work together with VisualGDB and if there is a possibility to integrated other test frameworks? Would it be even possible to integrate Ceedling somehow into a VisualGDB solution?

As far as I am aware is that when doing tests with cmake you can use ctest which seems already supported by Visual Studio. I am sure there is a good reason why VisualGDB is not using ctest and instead going into another direction?

The last point is not directly related to unit tests but I am still curious. When using an embedded advanced cmake project it is not possible to create a library by “add_library” even if the library does not consume any hardware related stuff, e.g. the BSP. This seems to prevent easy sharing of libraries between embedded and Linux/MinGW projects. Wouldn’t this enable to have an embedded project with the application and a MinGW project for testing of the embedded project?

 

Regards,

Bernhard

[support]

Hi,

No problem, we will try to help you.

VisualGDB loads the list of unit tests by scanning the binaries (either running them, or looking for specific symbols). Hence, in order for a newly added test to appear in the Test Explorer, you need to build the solution. If it doesn’t help, please let us know what happens after the solution build is complete. Does the Test Explorer window show the progress indicator? Does the View->Output->Test window show anything new?

Switching between the different platforms is necessary because VisualGDB (and CMake itself) defines the toolchain settings (i.e. the compiler to use for all targets) for the entire project. Hence, if you would like to run the tests on the Windows machine, you need to build the platform/configuration targeting the Windows machine. Building embedded code would require switching back to the platform/configuration that uses the ARM toolchain.

That said, you can work around it by having 1 solution with 2 VisualGDB projects around the same CMake codebase. The first project would load the top-level CMakeLists.txt file with the PLATFORM=Hardware macro, while the second project would load the same file with PLATFORM=Simulation (see CMake Command lines to understand what exactly gets passed to CMake). From Visual Studio and VisualGDB perspective, these will be 2 independent projects, one of them producing the ARM binaries, and another one producing Windows binaries. From CMake perspective, it will be the same set of CMake scripts reused between the builds.

Note that it will result in some cross-talk between the projects. E.g. adding a source file to a target within one project will not automatically refresh the same target shown in the other project until you manually refresh the second project via context menu.

Regarding the test frameworks, normally, VisualGDB would run the tests on the hardware. In this mode, it supports GoogleTest, CppUTest and TinyEmbeddedTest. As we have recently added support for simulation platforms (i.e. running tests on Windows), we have ported TinyEmbeddedTest and GoogleTest to support this configuration, but not CppUTest yet.

Ceedling looks interesting, however it appears to be very niche. We would be happy to fully integrate it with VisualGDB, however we would have to bill it as a custom paid feature unless sufficient number of our paid users shows interest in using it.

The regular CTest integration works by scanning the source files for test macros. This is less reliable than the binary-level analysis and does not work when running tests on the hardware (since it involves selecting specific test cases by modifying variables during program startup). That said, VisualGDB support for test frameworks is implemented via interchangeable plugins, so you can try tweaking it to support any case that is not supported directly.

You can add libraries to embedded projects via add_library(), however you would need to manually take care of all the fine details done by add_bsp_based_library(), such as automatic handling of code coverage, whole archive linking, handling of differences between GCC, IAR and Keil, and so on. A better approach would be to define a dummy implementation of add_bsp_based_library() to your Linux configuration and use it instead of the original one.

[bbr]

Hi,

Thank you for your explanations. I will try them out and test if they could work for us.

Regarding the test explorer: The progress indicator is shown when the build starts and disappears before the build is finished. The Output->Test view does only show the test discovery after opening the solution. During or after the build Output->Test is not updated. Please let me know if I can provide any further information on this.

 

Thanks,

Bernhard

[support]

No problem. Please let us know what Visual Studio version you are using.

Please also try the following steps to narrow down the test problem:

1. Open the solution in Visual Studio.
2. Open the View->Output window and switch to the Tests view.
3. Clear the Tests output.
4. Build the project and wait for it to finish.
5. Make sure the Test Explorer window is visible and wait until it doesn’t show any progress indication.
6. Open the View->Output window again, copy the entire output of the Tests view and post it here.

[bbr]

Hi,

I am using Visual Studio Professional 2019 version 16.8.2.

I did the steps you recommend but unfortunately the View->Output->Tests keeps empty after clearing it.

 

Regards,

Bernhard

[support]

Thanks for the update. We have rechecked everything and found a possible cause for the issue. Please try this build: VisualGDB-5.5.103.3919.msi

[bbr]

Thanks, this build solves the issue.

[Author]

Posts