aggregated coverage over different test projects

[dabramson]

Running the tests through the test explorer seems to always delete the old .gcda files on the deployment machine. This is fine until you have multiple test projects that contribute to the overall coverage of a multi-project system. If you don’t delete gcda files, the coverage data is aggregated. So running several test programs without cleaning up gcda file between them will let the gcda file reflect the total coverage of all test programs.

I’m not sure how that would fit into running tests from the test explorer, vgdb would need to wait until ALL the tests were deployed and run before pulling data from the deployment machine and would need to create scovreport at the solution level rather than the project level.

An option to “not delete existing coverage on deployment machine” for the command line would solve the issue for tfs builds. But I guess we’d also need something to trigger finalizing the coverage after running all of the test suites.

Is there already support for this that I’m missing?

Just some thoughts on how the coverage support may be improved.

[support]

Hi,

Thanks for the suggestion. VisualGDB handles the coverage reports on a per-project level (not per-solution), hence merging coverage information from multiple projects behind the scenes could cause weird results (e.g. coverage for subsequent reports belonging to the same project would differ based on the other tests executed during the same session an also the function invocation numbers would be off).

You can somewhat work around it by configuring VisualGDB to keep the raw coverage reports and then running the VisualGDB coverage report generator to build the aggregated report.

Please check the VisualGDB Diagnostics Console for command lines running valgrind2sprof64.exe. Each command line will reference one job file that has the following format:

>[base source directory #1]
[full paths to gcov files]
>[base source directory #2]
[full paths to gcov files]

Please try concatenating multiple job files into one and running valgrind2sprof64.exe on the combined job file. This should produce an aggregated coverage report that can be placed into your project’s coverage report directory in order for VisualGDB to show it.

[dabramson]

Hello,

I tried your suggestion. It sort of worked. The invocation numbers seem to have aggregated correctly, but the covered blocks and lines did not. It seems that whichever project’s .gcov is first in the concatenated job file is the one whose block and line coverage show in the output.

For me this is pretty limiting in how useful the coverage tool is. It works fine for single-project-single-test solutions, but in general that is not how projects at my company tend to go (especially on embedded projects running on linux).

I’m able to get the aggregated coverage that I’m looking for by running the tests directly (not through vgdb), then using gcov and gcovr together on the raw coverage files to get a readable report. But it it would be nice if I didn’t have to go down that road in the future.

[support]

No problem. We have done more experiments with this and it looks like we should be able to generate solution-level test reports with aggregated information from multiple test projects. However, this will involve non-trivial changes to the VisualGDB test engine that might break existing projects, so we will only be able to add this in the next major release (the very rough estimate for a preview build with this feature would be 3-4 months).

[Author]

Posts