[dabramson]

I turned that setting off because I wanted to NOT get coverage on the unit test project. I think that was the result of the suggestion in this post…

code coverage highlighting broken, and general question

The only coverage I’m interested in is the tested code. Thank you for trying to make this better.

[dabramson]

Hello,

The project I’m working with is just a toy anyway to work out all of this coverage stuff before trying to apply it to a real project. I attached that project in whole. If it is important that the project I send you be from scratch please let me know and I’ll try to find time to do it.

Thanks

Attachments:

You must be logged in to view attached files.

[dabramson]

Hello,

I tried your suggestion. It sort of worked. The invocation numbers seem to have aggregated correctly, but the covered blocks and lines did not. It seems that whichever project’s .gcov is first in the concatenated job file is the one whose block and line coverage show in the output.

For me this is pretty limiting in how useful the coverage tool is. It works fine for single-project-single-test solutions, but in general that is not how projects at my company tend to go (especially on embedded projects running on linux).

I’m able to get the aggregated coverage that I’m looking for by running the tests directly (not through vgdb), then using gcov and gcovr together on the raw coverage files to get a readable report. But it it would be nice if I didn’t have to go down that road in the future.

[dabramson]

OK. Tfs can now run through gtests via the visualgdb command.

Setting code coverage for debug and release did not help the empty coverage issue. The scovreport generated for the tests running from the tfs agent is empty, as is the manual run from the command line.

I have a project under test (PUT) and the test project (TEST). I’m attaching screen shots of the settings for both PUT and TEST.

The test container is also attached as requested.

• This reply was modified 5 years, 10 months ago by dabramson.

Attachments:

You must be logged in to view attached files.

[dabramson]

Ok. I ran with the update build.

I ran manually from the command line on the machine with the tfs build agent. That output is in manualrunfromcommandling.log. Again, the scovreport file was created but contains basically nothing. This is the command i used:

D:\03_agent_21342\_work\9\s>"%VISUALGDB_DIR%\visualgdb" /runtests VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:testresult.trx /diag /v > run.log

I used the same command in the tfs build. The only difference is that it uses absolute paths. You can see the command and the output from the build agent in outputfromtfsbuild.txt. You are correct that the tests never run. They have not been running from the build agent from the start. There is something with a modal dialog that I don’t know how to turn off. But when I run from the command line manually I do see for a split second a message box pop up on the screen. If you are testing with a build agent, are you running it as a service? We are.

Please let me know if you don’t get the outputs and I’ll do another ticket to get them to you.

• This reply was modified 5 years, 10 months ago by dabramson.

Attachments:

You must be logged in to view attached files.

[dabramson]

ok. sent the zip via a support ticket (#990398).

I’ll get back to you on the output from the 2789 build you linked above

[dabramson]

I didn’t attach it a second time.

I asking if, in general, you can use zip files as attachments on this site?

[dabramson]

I attached a zip. Does that not work?

I’ll get back to you on your request.

[dabramson]

This sounds very similar to one of the issues i’m experiencing in this post.

run gtest in tfs build with coverage

@dennyk – have you started having this issue again where the raw files are not saved if you run the tests through the command line?

[dabramson]

Hello,

The .msi I downloaded from the download site has “VisualGDB 5.4.100.2753” as the subject property in the file properties. I’m assuming this is the latest version. The version in the file properties of VisualGDB.exe on the TFS build server is 5.4.100.2753, so appears to be the latest version.

I updated the gtest framework on the build server to 20190124-1.8.1.

When I open the solution on the build server (in the build agent’s work directory) I can run a test suite and get the scovreport file containing coverage data, and the raw files are saved. This works correctly. This also indicates to me that the coverage files are created on the deployment machine. I double checked via ssh and they are indeed there.

When I use the command line – “%VISUALGDB_DIR%\visualgdb” /runtests VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:testresult.trx – from the agent’s sources directory (that is the directory containing the .sln file) I do get a .trx file as expected. But, the scovreport file contains no coverage information and there are no raw files stored. Visually comparing the scovreports from the first and second case shows the obvious lack of information in the second.

I did the same command but inserted the /v option. I didn’t see much difference in the output, maybe you will. I also verified via ssh that .gcda files were generated on the deployment machine (see the attached deployment_coveragedata_confirmation.txt).

The console output from both commands is attached in the zip file.

I tried the above on my local machine and got the same behavior I was getting from the build server (and I made sure gtest and vgdb were updated to latest on my local machine before trying it).

 

When I run the same command (just without the /v) from a Command Line TFS Build task I get the following output in the build log (see attached tfsbuildoutput.txt). I know you mentioned to try disabling the “custom console command” (which is linked to the “Launching custom program console” line in the output), but I don’t know how to do that. If you can tell me how to do that I will try it. I certainly did not turn it on knowingly.

[dabramson]

The environment variables mentioned are set as follows…

VISUALGDB_VERBOSE_OUTPUT=1

VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT=1

I’m not sure what those values ‘should’ be or if you think they are related to my issue. But I don’t know how to control them anyway. I tried turning the verbose setting off in the Options pane for VGDB in visual studio (set “Enable verbose output” to False), but the value did not change even after closing VS and opening a new command prompt. I’m getting those values from the help output by typing “VisualGDB” at the command line.

 

These are my versions:

• Visual Studio – 15.9.5 (I also updated to 15.9.6 and the same problems persisted)
• VGDB – 5.4R2 build 2753
• GoogleTest Framework – 20181002-1.8.1

 

I don’t know how to temporarily disable the custom console command. I did not consciously enable it. How do you enable/disable it? What is it? I’ll try it if you tell me how – that might get the command to run in the tfs build at least.

 

I want to make sure that we don’t lose sight of the goal here. In the end, I’m trying to run gtests on a remote target as part of an automated build from tfs. When I have binaries that were instrumented for coverage, I’d like to attach the scovreport file to the build. I’d also like to use the .gcno and .gcda files to create xml in cobertura format so that the tfs build can include a summarized report of coverage. I suppose I could run the gtest without using visualgdb, but then I’d assume I won’t get the scovreport file.

The first problem to overcome is that the gtests will not run via tfs with the command I’m using…so I need help finding the command that will let it run.

The second problem is not getting the raw gcov files or anything in the scovreport when running the tests from the command line (with the visualgdb /runtests). Is it possible that running it via the command line uses the gcov that is in the VisualGDB intall directory instead of the gcov provided with the toolchain used to build the project? That sounds like it might result in no coverage data…maybe?

 

Thanks,

Dave

 

 

[dabramson]

Hi,

Sorry for continually updating this post. I just saw that the scovreport file created when running unit tests at the command line via vgdb does not give coverage results. It is selectable in the VGDB code coverage window, but it does not show results. When running through test explorer the report shows coverage as I would expect.

 

[dabramson]

I’d like to add that when running the command manually I do not get the raw coverage files as I would if I run the tests from the test explorer. I do have the setting turned on in VGDB options to save raw coverage files. What I get is a folder with just an empty “job.txt” file in it. The folder is named the same as it would be when running through test explorer.

I would like to use those raw files to create an xml report in the cobertura format, which can be published to the tfs build report and acted on to fail the build if coverage is low.

[dabramson]

Gtests are now running on my target and I see the results in Test Explorer. Thanks for the fix!

[dabramson]

I added mkfifo to my system and things get further. I run all tests from test explorer, but all tests are skipped. It appears to be an incorrect formatting of the test names supplied to the –gtest_filter argument.

Here is what was passed by (I’m assuming) VGDB:

BasicCounterTests_ctor_WhenCalled.ResetsCount:BasicCounterTests_GetNextCount_WhenCalledRepeatedly.IncrementsCountForEachCall:BasicCounterTests_A_Failing.Test

The class name is BasicCounterTests and all test names have 2 underscores, but in the filter above there is a ‘.’ where the last underscore should be. If I run the tests manually on the target and put the ‘.’ after the classname and correct the test name (. becomes _) then all of the tests run as expected.

• This reply was modified 5 years, 11 months ago by dabramson.

[Author]

Posts