Code Coverage for CMake Build

[Waleed]

Hi! I’m trying to use VisualGDB to generate a coverage report of the code under test as part of the build process with CMake. The “build code coverage reports” checkbox is enabled in project properties which adds the following to the CMakeLists.txt:

bsp_configure_code_coverage(ENABLED 1)

The build fails during the pre-link patching stage, where the pre-link.bat script is executed. This batch script gets generated as part of the build process and it’s quite large (approximately 304,235 characters) because it contains all the object files compiled in the project.

My initial assumption is that, since VisualGDB invokes the pre-link step through cmd.exe, the generated batch script may be exceeding the maximum command-line length which causes part of the command to get truncated, resulting in the observed build failure. However, I’m not certain whether this is the actual cause. If this is true, what would be the recommended workaround? If not, what is the most likely cause of the pre-link patching failure?

Any guidance or suggestions would be greatly appreciated. Thanks in advance!

I’ve attached a screenshot as well captured during build to show failure logs.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

This could indeed be caused by the command-line length limit. VisualGDB’s code coverage support relies on patching all object files before the final linking takes place. This is defined in the coverage.cmake file under the VisualGDB\BSP directory:

add_custom_command(TARGET ${target_name} PRE_LINK COMMAND "$ENV{VISUALGDB_DIR}/VisualGDB.exe" ARGS /decover $<TARGET_FILE:${target_name}> ${_effective_sources})

If the _effective_sources list gets too long, it could indeed cause problems.

We can easily update VisualGDB to support the response file syntax (target name @<source list file>), but we are not aware of a straight-forward way to force CMake to use it for custom commands. You can try patching the CMake file on your end using temporary workarounds (e.g. manually generate a temporary list). If it works, we can easily upload an updated VisualGDB build with support for the @-syntax in that command.

[Waleed]

I’ve looked for the coverage.cmake file but it’s not present at the specified path i.e. AppData\Local\VisualGDB or even its subdirectories. However, acting upon your suggestion, I manually created a text file containing all the object files, put them into a CMake variable EFFECTIVE_SOURCES and then referenced them with the @-syntax in CMake with command below. The rest of the settings for code coverage i.e. instrumentation, were removed temporarily.

add_custom_command(TARGET ${target_name} PRE_LINK COMMAND VisualGDB.exe ARGS /decover ${target_path} "@${EFFECTIVE_SOURCES}")

This time, the build completed successfully. However, it appears that VisualGDB did not execute the pre-link command, which suggests that the object files may not have been patched (screenshot-1).

For debugging, I ran the CMake command directly on cmd.exe, passing it the variable referencing limited number of object files but received no coverage information there (screenshot-2). Additionally, I’ve noticed that there is no /decover option to VisualGDB I’ve got installed (version 6.1R2 – build 5524), yet it’s not reporting any type of error when run with it apparently.

• This reply was modified 1 week, 4 days ago by Waleed.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

The % syntax may not work correctly. We have updated VisualGDB to support the @-syntax in this build: VisualGDB-6.2.0.5619.msi.

E.g. /decover output.elf @inputs.rsp

If you are not sure what syntax Cmake is using, you can temporarily replace the custom command line with “cmd.exe ARGS /c echo VisualGDB.exe /decover …”. This will echo the actual VisualGDB command line instead of running it, so you can use it for troublehsooting.

[Waleed]

I have updated VisualGDB version using your provided link. However, this version still doesn’t seem to work with the @-syntax command when I run it in cmd. The results are captured in screenshot-1.
Furthermore, according to your last statement, I really don’t understand how replacing the CMake custom command would help me in troubleshooting the issue. I went ahead and changed it in CMake like this:

add_custom_command(TARGET ${target_name} PRE_LINK COMMAND cmd.exe ARGS /c echo VisualGDB.exe /decover "${CMAKE_CURRENT_SOURCE_DIR}/build/VisualGDB/Debug/${target_name}" @${target_name}.rsp WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/build/VisualGDB/Debug/CMakeFiles
)

It ran the build successfully and just echoed the command back in console as expected (screenshot-2). The response file here is being generated with the build and doesn’t get deleted because I have passed -d keeprsp arguments to the ninja.

[Waleed]

Sorry I forgot to add the screenshots with my previous reply. They’re attached with this one.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

Strange. Based on the screenshots you provided, it should work.

You can try a workaround like this:

1. Create a decover.bat file that runs VisualGDB with the response file.
2. Make sure it works when running manually. Note that the relative paths in the response files will be resolved against the working directory of VisualGDB.exe. So, you may want to hardcode full paths there for testing.
3. If it works, try updating CMakeLists.txt to just run the batch file. If it still works differently, you can try adding echo commands to the batch file to recheck that it actually gets invoked.

[Waleed]

Since the VisualGDB build is failing when code coverage is enabled with bsp_configure_code_coverage(ENABLED 1) and thus no ELF binary being generated, I had to divide it into steps:
1. Build the default application with code coverage disabled and save the resultant binary in separate folder temporarily.
2. Then, rebuild the application with code coverage enabled which will give me the response file with relative paths of object files. I replaced the relative paths with their absolute ones.
Finally, I ran the VisualGDB command manually with absolute path to the ELF and specifying response file with @-syntax, yet received no coverage information (screenshot-1). The same output is observed when the command is run inside decover.bat script.

As mentioned in your third point, what echo commands should I include in the batch? Could you please mention them here?

In addition, there’s one other workaround that I tried with this. The coverage.cmake file was located in the installation directory, and I edited its functionality to use response file rather than _effective_sources list. The new CMake command now looks as follows:

add_custom_command(TARGET ${target_name} PRE_LINK COMMAND "$ENV{VISUALGDB_DIR}/VisualGDB.exe" ARGS /decover $<TARGET_FILE:${target_name}> ${rspfile})

The ${rspfile} is the absolute path to the response file and the @-syntax is deliberately absent here because VisualGDB was giving error about not supporting this format. With this, and code coverage enabled in CMakeLists.txt, the build ran and one of the object files got detected at linking step after which it failed when there were issues e.g. memory overflow, undefined reference etc. found with it. I’m sure that this is not a problem with my existing application code. The complete logs for this build are captured in GDB-build.log. Could you please review them and let me know if anything stands out?

I’ve followed docs and made sure all the configuration is set accordingly. It would also be very helpful if you could share a working example of code coverage with a CMake-based project, including the generation of a coverage report. That would help me verify whether I’m not missing something.

• This reply was modified 4 days, 17 hours ago by Waleed.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

The easiest way to get it working would be to create a brand new CMake project containing just a couple of source files, enable the code coverage there, and make sure it works as intended. You can then use it as a reference, comparing to the big project that doesn’t work.

[Waleed]

Using response file syntax hasn’t worked, so I split the object files in ${target_name}.rsp file and processed them with VisualGDB /decover command one file at a time. This was quite a lengthy process but worth it since I was able to detach coverage information blocks from all of them. Then, I manually tried to link them into the final ELF with:

c:\sysgcc\arm-eabi\bin\arm-none-eabi-g++.exe -Og -g -Wl,--wrap=malloc -Wl,--wrap=calloc -Wl,--wrap=free -Wl,--wrap=realloc @CMakeFiles\${target_name}.rsp -o ${target_name}

Some of the memory regions inside binary overflowed which is unexpected since the memory overhead should have been reduced when I patched the object files with the VisualGDB command so they should be fitting inside their sections now. I’m sure I’ve been able to replicate the exact process VisualGDB follows when building project with code coverage, yet it’s not working.

Screenshot of linker error is attached for reference.

Attachments:

You must be logged in to view attached files.

[support]

Hi,

It could be that the project is just too large. Code coverage means reserving a small portion of device RAM for every branch in the code, so that the instrumented program can track whether that branch was ever taken.

If the program was approaching the limits of the RAM/FLASH already, it could very easily push it over the limit. Typically, it would be used with unit tests, so you could just try splitting the test executable into multiple smaller ones.

You can also try temporarily patching the linker script to increase the FLASH/RAM size by 10x to see how much space would the current setup need.

[Author]

Posts