Forum Replies Created
Viewing 11 posts - 1 through 11 (of 11 total)
-
Posts
-
Thanks for the update and quick fix! Just tested it and it works fine now. I will keep you up to date when we find any issues when we start using it for our production code.
I have applied the fixes manually to the visual gdb created file and now it works! So if you do a diff on the two files posted below you should be able to generate files which can be used in jenkins.
Original file:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <TestRun id="590287a4-ce44-4bd5-883a-c510e6700676" name="CICD_Test_UnitTest-Debug" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"> <ResultSummary> <Counters total="3" executed="3" passed="2" failed="1" /> </ResultSummary> <TestDefinitions> <UnitTest name="CalcTestGroup::Multiply" id="f02fe075-6b68-4463-8b8d-cdbc0e1cb81e"> <Execution id="726f3d8f-8373-41c6-b6a2-6ddfc56f7039" /> </UnitTest> <UnitTest name="CalcTestGroup::Subtract" id="f50a734f-ffb2-4689-a82c-eb3043299a56"> <Execution id="ca9f4741-fac8-4190-b7fd-87244cffd7aa" /> </UnitTest> <UnitTest name="CalcTestGroup::Add" id="af68fb44-0096-4bab-966b-dedac12510d6"> <Execution id="cd3e14be-b33a-4203-818c-1d11944c47d0" /> </UnitTest> </TestDefinitions> <TestLists> <TestList name="All Tests" id="18bddc40-d190-4007-8170-ac27c2ba55d6" /> </TestLists> <Results> <UnitTestResult executionId="726f3d8f-8373-41c6-b6a2-6ddfc56f7039" testId="f02fe075-6b68-4463-8b8d-cdbc0e1cb81e" testName="CalcTestGroup::Multiply" duration="00:00:00" startTime="10/12/2019 16:02:22" endTime="10/12/2019 16:02:22" outcome="Passed" testListId="18bddc40-d190-4007-8170-ac27c2ba55d6"> <Output> <TextMessages> <Message>.</Message> </TextMessages> </Output> </UnitTestResult> <UnitTestResult executionId="ca9f4741-fac8-4190-b7fd-87244cffd7aa" testId="f50a734f-ffb2-4689-a82c-eb3043299a56" testName="CalcTestGroup::Subtract" duration="00:00:00.0020000" startTime="10/12/2019 16:02:22" endTime="10/12/2019 16:02:22" outcome="Passed" testListId="18bddc40-d190-4007-8170-ac27c2ba55d6"> <Output> <TextMessages> <Message>.</Message> </TextMessages> </Output> </UnitTestResult> <UnitTestResult executionId="cd3e14be-b33a-4203-818c-1d11944c47d0" testId="af68fb44-0096-4bab-966b-dedac12510d6" testName="CalcTestGroup::Add" duration="00:00:00.1150000" startTime="10/12/2019 16:02:22" endTime="10/12/2019 16:02:22" outcome="Failed" testListId="18bddc40-d190-4007-8170-ac27c2ba55d6"> <Output> <TextMessages> <Message> Calc_Tests.cpp:19: error: Failure in TEST(CalcTestGroup, Add) expected <1 0x1> but was <2 0x2> .</Message> </TextMessages> <ErrorInfo> <Message>expected <1 0x1> but was <2 0x2></Message> <StackTrace>at TEST_CalcTestGroup_Add_Test::testBody in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Calc_Tests.cpp:line 19 at helperDoTestBody in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 93 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at Utest::run in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 599 at UtestShell::runOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 211 at helperDoRunOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 118 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at UtestShell::runOneTest in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 185 at TestRegistry::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\TestRegistry.cpp:line 87 at CommandLineTestRunner::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 124 at CommandLineTestRunner::runAllTestsMain in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 83 at CommandLineTestRunner::RunAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 52 at main in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Main.cpp:line 33 </StackTrace> </ErrorInfo> </Output> </UnitTestResult> </Results> </TestRun>
Fixed file:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <TestRun id="191623c6-8a94-4bb3-b617-00a32d143bd2" name="CICD_Test_UnitTest-Debug" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"> <ResultSummary> <Counters total="3" executed="3" passed="2" failed="1" /> </ResultSummary> <TestDefinitions> <UnitTest name="CalcTestGroup::Multiply" id="fe6a2f48-7c29-4aac-a54f-c3c5b3ca876d"> <Execution id="ea9f9ae3-fc9a-4f80-a616-aec208705a3a" /> <TestMethod codeBase="c:/dev/pickles-results-harness/Pickles.TestHarness/Pickles.TestHarness.MSTest/bin/debug/Pickles.TestHarness.MSTest.DLL" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="Pickles.TestHarness.MSTest.AdditionFeature, Pickles.TestHarness.MSTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" name="CalcTestGroup::Multiply" /> </UnitTest> <UnitTest name="CalcTestGroup::Subtract" id="0e043fed-0f1a-4a9e-9b83-d2a70ff0376e"> <Execution id="a2bd2fc2-724f-48df-86c0-d2eef5571a9c" /> <TestMethod codeBase="c:/dev/pickles-results-harness/Pickles.TestHarness/Pickles.TestHarness.MSTest/bin/debug/Pickles.TestHarness.MSTest.DLL" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="Pickles.TestHarness.MSTest.AdditionFeature, Pickles.TestHarness.MSTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" name="CalcTestGroup::Subtract" /> </UnitTest> <UnitTest name="CalcTestGroup::Add" id="651a2fb3-6737-4c32-942d-333ffa6bc527"> <Execution id="e209f15e-6bab-4349-b180-76fb1e6b031a" /> <TestMethod codeBase="c:/dev/pickles-results-harness/Pickles.TestHarness/Pickles.TestHarness.MSTest/bin/debug/Pickles.TestHarness.MSTest.DLL" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="Pickles.TestHarness.MSTest.AdditionFeature, Pickles.TestHarness.MSTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" name="CalcTestGroup::Add" /> </UnitTest> </TestDefinitions> <TestLists> <TestList name="All Tests" id="ee45e271-96a6-4c28-9387-382e7b07f965" /> </TestLists> <TestEntries> <TestEntry testId="fe6a2f48-7c29-4aac-a54f-c3c5b3ca876d" executionId="ea9f9ae3-fc9a-4f80-a616-aec208705a3a" testListId="ee45e271-96a6-4c28-9387-382e7b07f965" /> <TestEntry testId="0e043fed-0f1a-4a9e-9b83-d2a70ff0376e" executionId="a2bd2fc2-724f-48df-86c0-d2eef5571a9c" testListId="ee45e271-96a6-4c28-9387-382e7b07f965" /> <TestEntry testId="651a2fb3-6737-4c32-942d-333ffa6bc527" executionId="e209f15e-6bab-4349-b180-76fb1e6b031a" testListId="ee45e271-96a6-4c28-9387-382e7b07f965" /> </TestEntries> <Results> <UnitTestResult executionId="ea9f9ae3-fc9a-4f80-a616-aec208705a3a" testId="fe6a2f48-7c29-4aac-a54f-c3c5b3ca876d" testName="CalcTestGroup::Multiply" duration="00:00:00" startTime="10/12/2019 15:39:06" endTime="10/12/2019 15:39:06" outcome="Passed" testListId="ee45e271-96a6-4c28-9387-382e7b07f965"> <Output> <StdOut> <Message>.</Message> </StdOut> </Output> </UnitTestResult> <UnitTestResult executionId="a2bd2fc2-724f-48df-86c0-d2eef5571a9c" testId="0e043fed-0f1a-4a9e-9b83-d2a70ff0376e" testName="CalcTestGroup::Subtract" duration="00:00:00" startTime="10/12/2019 15:39:06" endTime="10/12/2019 15:39:06" outcome="Passed" testListId="ee45e271-96a6-4c28-9387-382e7b07f965"> <Output> <StdOut> <Message>.</Message> </StdOut> </Output> </UnitTestResult> <UnitTestResult executionId="e209f15e-6bab-4349-b180-76fb1e6b031a" testId="651a2fb3-6737-4c32-942d-333ffa6bc527" testName="CalcTestGroup::Add" duration="00:00:00.1430000" startTime="10/12/2019 15:39:06" endTime="10/12/2019 15:39:06" outcome="Failed" testListId="ee45e271-96a6-4c28-9387-382e7b07f965"> <Output> <StdOut> <Message> Calc_Tests.cpp:19: error: Failure in TEST(CalcTestGroup, Add) expected <1 0x1> but was <2 0x2> </Message> </StdOut> <ErrorInfo> <Message> expected <1 0x1> but was <2 0x2> </Message> <StackTrace> at TEST_CalcTestGroup_Add_Test::testBody in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Calc_Tests.cpp:line 19 at helperDoTestBody in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 93 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at Utest::run in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 599 at UtestShell::runOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 211 at helperDoRunOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 118 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at UtestShell::runOneTest in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 185 at TestRegistry::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\TestRegistry.cpp:line 87 at CommandLineTestRunner::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 124 at CommandLineTestRunner::runAllTestsMain in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 83 at CommandLineTestRunner::RunAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 52 at main in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Main.cpp:line 33 </StackTrace> </ErrorInfo> </Output> </UnitTestResult> </Results> </TestRun>
I’ve looked up a random trx file generated by mstest.
That one works fine with the mstest plugin (on jenkins)
if i remove the following line the file stops working. So guess it has something to do with that: (it is missing in the visual gdb generated file)
<TestMethod codeBase=”c:/dev/pickles-results-harne…
https://github.com/x97mdr/pickles/blob/master/src/Pickles/Pickles.Test/results-example-mstest.trx
I’ve made it a bit smaller so it is easier to compare it to the one generated by visual gdb (the one posted above) :
Fail to add two numbers Given I have entered 50 into the calculator -> done: Steps.GivenIHaveEnteredSomethingIntoTheCalculator(50) (0.0s) And I have entered -1 into the calculator -> done: Steps.GivenIHaveEnteredSomethingIntoTheCalculator(-1) (0.0s) When I press add -> done: Steps.WhenIPressAdd() (0.0s) Then the result should be -5000 on the screen -> error: Assert.NotEqual() Failure Test method Pickles.TestHarness.MSTest.AdditionFeature.FailToAddTwoNumbers threw exception: Should.Core.Exceptions.NotEqualException: Assert.NotEqual() Failure at Pickles.TestHarness.MSTest.Steps.ThenTheResultShouldBePass(Int32 result) in C:\dev\pickles-results-harness\Pickles.TestHarness\Pickles.TestHarness.MSTest\Steps.cs:line 28 at lambda_method(Closure , IContextManager , Int32 ) at TechTalk.SpecFlow.Bindings.MethodBinding.InvokeAction(IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) at TechTalk.SpecFlow.Bindings.StepDefinitionBinding.Invoke(IContextManager contextManager, ITestTracer testTracer, Object[] arguments, TimeSpan& duration) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepArgs stepArgs) at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep() at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors() at Pickles.TestHarness.MSTest.AdditionFeature.ScenarioCleanup() in C:\dev\pickles-results-harness\Pickles.TestHarness\Pickles.TestHarness.MSTest\Addition.feature.cs:line 0 at Pickles.TestHarness.MSTest.AdditionFeature.FailToAddTwoNumbers() in c:\dev\pickles-results-harness\Pickles.TestHarness\Pickles.TestHarness.MSTest\Addition.feature:line 18
I’ve ran the unit test in jenkins, I made it fail intentionally. You can find the results file below. This results file Is given to MS test plugin and it does not detect test results.
Jenkins message: [MSTEST-PLUGIN] INFO processing test results in file(s) unit_test_results.trx </span></div> [MSTEST-PLUGIN] INFO processing report file: C:\Users\Mark\.jenkins\workspace\Example_Pipeline\unit_test_results.trx </span></div> [MSTEST-PLUGIN] WARNING Unable to delete the file C:\Users\Mark\.jenkins\workspace\Example_Pipeline\temporary-junit-reports </span></div> [MSTEST-PLUGIN] INFO This file is a reserved temporary file. You can delete it safely. </span></div> [MSTEST-PLUGIN] INFO XML coverage report file not found: C:\Users\Mark\.jenkins\workspace\Example_Pipeline\vstest.coveragexml </span></div> [MSTEST-PLUGIN] INFO XML coverage report file not found: C:\Users\Mark\.jenkins\workspace\Example_Pipeline\unit_test_results.coveragexml </span></div> [MSTEST-PLUGIN] SEVERE None of the test reports contained any result.
.trx file content:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <TestRun id="e04afaec-3698-4b27-86ff-9144e00e4724" name="CICD_Test_UnitTest-Debug" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"> <ResultSummary> <Counters total="1" executed="1" passed="0" failed="1" /> </ResultSummary> <TestDefinitions> <UnitTest name="CalcTestGroup::Add" id="733861c8-0c1f-4070-9503-8682f091c389"> <Execution id="54b2b785-2774-4785-a349-bed7e4357b43" /> </UnitTest> </TestDefinitions> <TestLists> <TestList name="All Tests" id="89b12475-5d59-4897-8a17-b74cd442d0de" /> </TestLists> <Results> <UnitTestResult executionId="54b2b785-2774-4785-a349-bed7e4357b43" testId="733861c8-0c1f-4070-9503-8682f091c389" testName="CalcTestGroup::Add" duration="00:00:00.1340000" startTime="09/12/2019 11:12:10" endTime="09/12/2019 11:12:10" outcome="Failed" testListId="89b12475-5d59-4897-8a17-b74cd442d0de"> <Output> <TextMessages> <Message> Calc_Tests.cpp:19: error: Failure in TEST(CalcTestGroup, Add) expected <1 0x1> but was <2 0x2> .</Message> </TextMessages> <ErrorInfo> <Message>expected <1 0x1> but was <2 0x2></Message> <StackTrace>at TEST_CalcTestGroup_Add_Test::testBody in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Calc_Tests.cpp:line 19 at helperDoTestBody in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 93 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at Utest::run in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 599 at UtestShell::runOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 211 at helperDoRunOneTestInCurrentProcess in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 118 at PlatformSpecificSetJmpImplementation in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\Platforms\Embedded\UtestPlatformEmbedded.cpp:line 144 at UtestShell::runOneTest in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\Utest.cpp:line 185 at TestRegistry::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\TestRegistry.cpp:line 87 at CommandLineTestRunner::runAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 124 at CommandLineTestRunner::runAllTestsMain in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 83 at CommandLineTestRunner::RunAllTests in C:\Users\Mark\AppData\Local\VisualGDB\TestFrameworks\com.sysprogs.unittest.CppUTest\src\CppUTest\CommandLineTestRunner.cpp:line 52 at main in C:\Users\Mark\.jenkins\workspace\Example_Pipeline\CICD_Test_UnitTest\Main.cpp:line 33 </StackTrace> </ErrorInfo> </Output> </UnitTestResult> </Results> </TestRun>
Do you have a suggestion on which tool to use? It does not seem to work with the general plugins of jenkins (xunit / MStest
Hi All,
I did some testing, but as soon as I enable the code coverage I run in to a issue. The compiler complains that the sram section is to small. This is due to the -profile-arcs which is automatically set because of the –coverage flag. Is this a known issue? (chip used: EFR32MG1P232F256GM48) Using the -fno-zero-initialized-in-bss does not solve the problem.
Kind regards,
Mark
Hi All,
I did some testing, but as soon as I enable the code coverage io run in to a issue. The compiler complains that the sram section is to small. This is due to the “-<strong class=”hu iv”>fprofile-arcs” which is automatically set because of the “–coverage flag”. Is this a known issue? (chip uses: EFR32MG1P232F256GM48)
Kind regards,
Mark
Viewing 11 posts - 1 through 11 (of 11 total)