run gtest in tfs build with coverage

[dabramson]

Hello,

I’m not able to get our tfs build server to run gtests on the remote linux target under visual gdb. If all I wanted to do was to get the tests to run, I could just run gtest outside of visualgdb and pull the result xml from the target for publishing in TFS. But…I would also like to have the coverage report generated by visual gdb (scovreport) as a build artifact.

Here are the pertinent lines from the build server build log when trying to run the test...2019-01-25T20:31:56.3059929Z ==============================================================================
2019-01-25T20:31:56.3060265Z Task : Command Line
2019-01-25T20:31:56.3060398Z Description : Run a command line with arguments
2019-01-25T20:31:56.3060513Z Version : 1.1.3
2019-01-25T20:31:56.3060657Z Author : Microsoft Corporation
2019-01-25T20:31:56.3060786Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=613735)
2019-01-25T20:31:56.3060940Z ==============================================================================
...
2019-01-25T20:31:56.3145771Z ##[debug]C:\WINDOWS\system32\cmd.exe /c ""C:\Program Files (x86)\Sysprogs\VisualGDB\\visualgdb.exe" /runtests D:\03_agent_21342\_work\9\s\VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:D:\03_agent_21342\_work\9\s\TestSharedObj_HelloVgdbCrossWinMsBuild\Debug-testresult.trx"
2019-01-25T20:31:56.3146173Z ##[command]"C:\Program Files (x86)\Sysprogs\VisualGDB\\visualgdb.exe" /runtests D:\03_agent_21342\_work\9\s\VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:D:\03_agent_21342\_work\9\s\TestSharedObj_HelloVgdbCrossWinMsBuild\Debug-testresult.trx
2019-01-25T20:31:59.2691269Z Test container loaded (lookup: 1 msec, fetch: 0 msec, symbol list: 157 msec)
2019-01-25T20:31:59.2910575Z SSH: Connecting...
2019-01-25T20:31:59.2942428Z SSH: Handshaking...
2019-01-25T20:32:00.2507910Z SSH: Checking key...
2019-01-25T20:32:00.2630518Z SSH: Authenticating...
2019-01-25T20:32:01.5303370Z Discovered 4 tests. Starting a test session...
2019-01-25T20:32:01.6734372Z Deploying project output...
2019-01-25T20:32:01.6770470Z VisualGDB: Deploying D:\03_agent_21342\_work\9\s\VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild to /opt/psi/BKMainApp/TestSharedObj_HelloVgdbCrossWinMsBuild on 192.168.16.225...
2019-01-25T20:32:02.2159116Z
2019-01-25T20:32:02.4976411Z ... 48% ....
2019-01-25T20:32:02.5143138Z ... 96% ....
2019-01-25T20:32:02.5171533Z ... running ....
2019-01-25T20:32:02.6526868Z ... running ....VisualGDB: Launching gdbserver
2019-01-25T20:32:02.6532223Z VisualGDB: Run "gdbserver :2000 "$(DeployPath)" --gtest_filter=$(1).$(2):$(1).$(2):$(1).$(2):$(1).$(2)" in directory "$(DeployDir)" on root@192.168.16.225 (SSH)
2019-01-25T20:32:02.6549950Z VisualGDB: Launching custom program console
2019-01-25T20:32:02.8841416Z Build/launch failed: Showing a modal dialog box or form when the application is not running in UserInteractive mode is not a valid operation. Specify the ServiceNotification or DefaultDesktopOnly style to display a notification from a service application.
2019-01-25T20:32:02.8842193Z Press any key...
2019-01-25T20:32:02.8842721Z
2019-01-25T20:32:02.8844286Z Build/launch failed: Cannot read keys when either application does not have a console or when console input has been redirected from a file. Try Console.Read.
2019-01-25T20:32:03.0156050Z ##[error]Process completed with exit code 1.
2019-01-25T20:32:03.0167380Z ##[debug]System.Exception: Process completed with exit code 1.
at Microsoft.VisualStudio.Services.Agent.Worker.Handlers.ProcessHandler.d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.TaskRunner.d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.d__1.MoveNext()
2019-01-25T20:32:03.0172192Z ##[section]Finishing: Run Unit Tests

The –gtest_filter values show the proper substitution in the console view of the build, but don’t make it to the log…that’s not great, but obviously the bigger problem is that it won’t run the tests.

When I run the same command (see 2019-01-25T20:31:56.314577) on the build server from a command prompt the tests run, the trx file is generated, and the coverage reports are created. I also don’t see a modal dialog box or form pop up when I run from the command line as some of the error information in the build log suggests.

I didn’t see anything in the visualgdb command line help that would obviously fix this…I did notice that boxe VERBOSE_OUTPUT and SHOW_DIAGNOSTIC_OUTPUT are set to 1, but again I don’t see a dialog box when it runs.

Any thoughts here?

 

Thanks,

Dave Abramson

• This topic was modified 5 years, 11 months ago by support. Reason: formatting
• This topic was modified 5 years, 11 months ago by support. Reason: formatting

[dabramson]

I’d like to add that when running the command manually I do not get the raw coverage files as I would if I run the tests from the test explorer. I do have the setting turned on in VGDB options to save raw coverage files. What I get is a folder with just an empty “job.txt” file in it. The folder is named the same as it would be when running through test explorer.

I would like to use those raw files to create an xml report in the cobertura format, which can be published to the tfs build report and acted on to fail the build if coverage is low.

[dabramson]

Hi,

Sorry for continually updating this post. I just saw that the scovreport file created when running unit tests at the command line via vgdb does not give coverage results. It is selectable in the VGDB code coverage window, but it does not show results. When running through test explorer the report shows coverage as I would expect.

 

[support]

Hi,

Sorry for the delay. We have retested coverage report building with the latest VisualGDB 5.4R2 for cross-toolchains (using /runtests) and unfortunately could not reproduce this. Please double-check the environment variables you set (should be VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT, not just SHOW_DIAGNOSTIC_OUTPUT).

The problem might be also related to the custom console command you are using:

Launching custom program console

Please try temporarily disabling it.

Finally, it looks like you are using the latest gtest framework with an older VisualGDB build (pre-v5.4R2) that might not be able to handle the test selection rules properly (we had to change them to fix incorrect handling of test names with multiple underscores). Please try updating to the latest VisualGDB.

[dabramson]

The environment variables mentioned are set as follows…

VISUALGDB_VERBOSE_OUTPUT=1

VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT=1

I’m not sure what those values ‘should’ be or if you think they are related to my issue. But I don’t know how to control them anyway. I tried turning the verbose setting off in the Options pane for VGDB in visual studio (set “Enable verbose output” to False), but the value did not change even after closing VS and opening a new command prompt. I’m getting those values from the help output by typing “VisualGDB” at the command line.

 

These are my versions:

• Visual Studio – 15.9.5 (I also updated to 15.9.6 and the same problems persisted)
• VGDB – 5.4R2 build 2753
• GoogleTest Framework – 20181002-1.8.1

 

I don’t know how to temporarily disable the custom console command. I did not consciously enable it. How do you enable/disable it? What is it? I’ll try it if you tell me how – that might get the command to run in the tfs build at least.

 

I want to make sure that we don’t lose sight of the goal here. In the end, I’m trying to run gtests on a remote target as part of an automated build from tfs. When I have binaries that were instrumented for coverage, I’d like to attach the scovreport file to the build. I’d also like to use the .gcno and .gcda files to create xml in cobertura format so that the tfs build can include a summarized report of coverage. I suppose I could run the gtest without using visualgdb, but then I’d assume I won’t get the scovreport file.

The first problem to overcome is that the gtests will not run via tfs with the command I’m using…so I need help finding the command that will let it run.

The second problem is not getting the raw gcov files or anything in the scovreport when running the tests from the command line (with the visualgdb /runtests). Is it possible that running it via the command line uses the gcov that is in the VisualGDB intall directory instead of the gcov provided with the toolchain used to build the project? That sounds like it might result in no coverage data…maybe?

 

Thanks,

Dave

 

 

[support]

Hi,

Thanks for the detailed description. Just FYI: the latest version of the GoogleTest package is 20190124-1.8.1, not 20181002-1.8.1. It might resolve some of the issues.

Please also check that the TFS machine has the latest VisualGDB version installed (you might have deployed an earlier version of VisualGDB there manually and have not updated it). You can check the build number via File Properties on the VisualGDB.exe file.

If updating doesn’t help, please confirm whether you are getting the coverage files when running VisualGDB.exe /runtests from your main machine (not from the TFS machine). If not, please try adding “/v” to the VisualGDB’s command line and share the updated output. Please also check if the coverage files are created on the deployment machine in the directory specified via VS Project Properties -> Instrumentation -> Coverage Report Directory. If not, please double-check that the directory is writable.

VisualGDB indeed uses the gcov from its own directory, however it should not cause any problems as the processing it performs is platform-independent.

[dabramson]

Hello,

The .msi I downloaded from the download site has “VisualGDB 5.4.100.2753” as the subject property in the file properties. I’m assuming this is the latest version. The version in the file properties of VisualGDB.exe on the TFS build server is 5.4.100.2753, so appears to be the latest version.

I updated the gtest framework on the build server to 20190124-1.8.1.

When I open the solution on the build server (in the build agent’s work directory) I can run a test suite and get the scovreport file containing coverage data, and the raw files are saved. This works correctly. This also indicates to me that the coverage files are created on the deployment machine. I double checked via ssh and they are indeed there.

When I use the command line – “%VISUALGDB_DIR%\visualgdb” /runtests VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:testresult.trx – from the agent’s sources directory (that is the directory containing the .sln file) I do get a .trx file as expected. But, the scovreport file contains no coverage information and there are no raw files stored. Visually comparing the scovreports from the first and second case shows the obvious lack of information in the second.

I did the same command but inserted the /v option. I didn’t see much difference in the output, maybe you will. I also verified via ssh that .gcda files were generated on the deployment machine (see the attached deployment_coveragedata_confirmation.txt).

The console output from both commands is attached in the zip file.

I tried the above on my local machine and got the same behavior I was getting from the build server (and I made sure gtest and vgdb were updated to latest on my local machine before trying it).

 

When I run the same command (just without the /v) from a Command Line TFS Build task I get the following output in the build log (see attached tfsbuildoutput.txt). I know you mentioned to try disabling the “custom console command” (which is linked to the “Launching custom program console” line in the output), but I don’t know how to do that. If you can tell me how to do that I will try it. I certainly did not turn it on knowingly.

[support]

Sorry, unfortunately the output you mentioned did not get attached. We also rechecked everything on a clean machine and unfortunately could not reproduce the problem.

In case the VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT variable is not being handled correctly in your setup, we have added a new /diag switch to this build.

Please run it as shown below:

"%VISUALGDB_DIR%\visualgdb" /runtests VisualGDB\Debug\CrossCoverageTest.vgdbtestcontainer /vsoutput:testresult.trx /diag > run.log

The output should contain lines like these:

Downloading coverage files...
Received X11 channel request. Trying localhost:6000...
Directory set #1
        Build-time directory: c:\projects\temp\CrossCoverageTest\VisualGDB\Debug
        Runtime directory: /tmp/cov
Starting coverage report converter: C:\Program Files (x86)\Sysprogs\VisualGDB\valgrind2sprof64.exe --gcov "c:\projects\temp\CrossCoverageTest\CoverageReports\2_6_2019_11_19_00_AM\job.txt" "c:\projects\temp\CrossCoverageTest\CoverageReports\2_6_2019_11_19_00_AM.scovreport"

Please attach the updated output. It should explain what is going on inside the coverage report logic.

[dabramson]

I attached a zip. Does that not work?

I’ll get back to you on your request.

[support]

Sorry, it still did not work. We have just rechecked attachments on our side and they appear to be working, so the problem might be caused by your browser.

As a workaround, please try using a file hosting service like Dropbox, or simply send the attachments via our support form.

[dabramson]

I didn’t attach it a second time.

I asking if, in general, you can use zip files as attachments on this site?

[support]

Yes, we have just double-checked the attachment logic on our side and it appears to be working.

[dabramson]

ok. sent the zip via a support ticket (#990398).

I’ll get back to you on the output from the 2789 build you linked above

[support]

We have received the logs via the support system.

It looks like VisualGDB running from the test agent still completely fails to launch the tests (it also doesn’t get the VISUALGDB_SHOW_DIAGNOSTIC_OUTPUT variable).

Please try adding the “/v” and “/diag” flags to the VisualGDB command line in TFS (ensure you use build 2789) and send us the updated logs.

[dabramson]

Ok. I ran with the update build.

I ran manually from the command line on the machine with the tfs build agent. That output is in manualrunfromcommandling.log. Again, the scovreport file was created but contains basically nothing. This is the command i used:

D:\03_agent_21342\_work\9\s>"%VISUALGDB_DIR%\visualgdb" /runtests VisualGDB\Debug\TestSharedObj_HelloVgdbCrossWinMsBuild.vgdbtestcontainer /vsoutput:testresult.trx /diag /v > run.log

I used the same command in the tfs build. The only difference is that it uses absolute paths. You can see the command and the output from the build agent in outputfromtfsbuild.txt. You are correct that the tests never run. They have not been running from the build agent from the start. There is something with a modal dialog that I don’t know how to turn off. But when I run from the command line manually I do see for a split second a message box pop up on the screen. If you are testing with a build agent, are you running it as a service? We are.

Please let me know if you don’t get the outputs and I’ll do another ticket to get them to you.

• This reply was modified 5 years, 10 months ago by dabramson.

Attachments:

You must be logged in to view attached files.

[Author]

Posts