MsTest DataRow support

[gasparnagy]

(This is a redo of PR #52 by @livioc, to be able to fix the generation issues.)

Types of changes

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).
• Performance improvement
• Refactoring (so no functional change)
• Other (docs, build config, etc)

Checklist:

• I've added tests for my code. (most of the time mandatory)
• I have added an entry to the changelog. (mandatory)
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[livioc]

@gasparnagy DataRowAttribute doesn't support any additional argument other than the test parameters array and the DisplayName (https://learn.microsoft.com/en-us/dotnet/api/microsoft.visualstudio.testtools.unittesting.datarowattribute.-ctor?view=visualstudiosdk-2022).
At runtime a System.Reflection.TargetParameterCountException will be thown when the test methods has a number of parameters different than the DataRowAttribute

Anyhow I'm still trying to understand what's the issue with this pull request, I have a lot of difficulties to investigate Reqnroll.Specs's local execution

[gasparnagy]

@livioc Reqnroll adds an additional string[] parameter to the method to know about the tags (regardless of whether the test framework supports it) and that had to be provided.

The rest of the issues seem to be related to how we do the testing: we let the tests run and then parse the TRX file to get the results. But MsTest generates the TRX file in a slightly different format in case of DataRow, so that causes the problems. I have a working env now, I can try to debug the rest.

[gasparnagy]

Oh God. This was a tricky issue. Somehow the code we generated was not working in case there was only one column in the examples table with MsTest v2.*...

Locally it is passing now. Let's hope it works.

But I had to learn that the local testability is really a pain, especially for generator changes. Need to improve this (https://github.com/orgs/reqnroll/discussions/38).

[livioc]

Hi @gasparnagy , today I have done some tests on a new .net48 test project but I still have problem.

I created the nuget packages with .\build.ps1 -Configuration Release on e999ecf.

Then I generated the test code using <PackageReference Include="Reqnroll.MsTest" Version="1.0.2-local" /> with this gherkin

 Scenario Outline: Simple scenario
		Given there is <count> <color> items

		Examples:
			| count | color |
			|     0 | blue  |
			|     0 | black |

When I run the generated test code

        [Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Simple scenario")]
        [Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "blue")]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "black")]
        public async System.Threading.Tasks.Task SimpleScenario(string count, string color, string[] exampleTags)
        {
            string[] tagsOfScenario = exampleTags;
            System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
            argumentsOfScenario.Add("count", count);
            argumentsOfScenario.Add("color", color);
            Reqnroll.ScenarioInfo scenarioInfo = new Reqnroll.ScenarioInfo("Simple scenario", null, tagsOfScenario, argumentsOfScenario, featureTags);
#line 15
this.ScenarioInitialize(scenarioInfo);
#line hidden
            if ((TagHelper.ContainsIgnoreTag(tagsOfScenario) || TagHelper.ContainsIgnoreTag(featureTags)))
            {
                testRunner.SkipScenario();
            }
            else
            {
                await this.ScenarioStartAsync();
#line 16
  await testRunner.GivenAsync(string.Format("there is {0} {1} items", count, color), ((string)(null)), ((Reqnroll.Table)(null)), "Given ");
#line hidden
            }
            await this.ScenarioCleanupAsync();
        }

I got the exception

 SimpleScenario (0,black)
   Source: Calculator.feature line 15
   Duration: 71 ms

  Message: 
Test method ReqnrollProject1.Features.CalculatorFeature.SimpleScenario threw exception: 
System.Reflection.TargetParameterCountException: Parameter count mismatch.
TestCleanup method ReqnrollProject1.Features.CalculatorFeature.TestTearDownAsync threw exception. System.NullReferenceException: Object reference not set to an instance of an object..

  Stack Trace: 
RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
TestCleanup Stack Trace
<OnScenarioEndAsync>d__34.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
TaskAwaiter.GetResult()
<OnScenarioEndAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
TaskAwaiter.GetResult()
<TestTearDownAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

  Standard Output: 
-> Loading plugin C:\dev\ExtGitHub\Reqnroll-PersonalTest\ReqnrollProject1\bin\Debug\net48\Reqnroll.MSTest.ReqnrollPlugin.dll
-> Loading plugin C:\dev\ExtGitHub\Reqnroll-PersonalTest\ReqnrollProject1\bin\Debug\net48\ReqnrollProject1.dll
-> Using default config
-> warning: The previous ScenarioContext was already disposed.


TestContext Messages:
-> warning: The previous ScenarioContext was already disposed.

Are you sure that we must pass the addition tags array parameter? Can't we just ignore them?

[gasparnagy]

@livioc The current test system is not working properly. Actually you are not testing with the version you have built (you can see it because the [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("0", "black")] has only two parameters, but it should have 3.
The problem is that the package has been cached to your C:\Users\<your user>\.nuget\packages folder and therefore it does not matter if you compile it locally, you will still test with the previous one (I work on a solution in #62).
What you can do is to manually delete all reqnroll related packages from C:\Users\<your user>\.nuget\packages and then try again.

[livioc]

@gasparnagy thanks, I have done same tests, this scenaraio doens't work with mstest

The generated code has a different number of parameters

        [Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another scenario")]
        [Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("1", "", new string[] {
                "mytag"})]
        [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("2", null)]
        public async System.Threading.Tasks.Task AnotherScenario(string count, string notUsed6248, string[] exampleTags)
        {
            string[] tagsOfScenario = exampleTags;
            System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
            argumentsOfScenario.Add("count", count);
            Reqnroll.ScenarioInfo scenarioInfo = new Reqnroll.ScenarioInfo("Another scenario", null, tagsOfScenario, argumentsOfScenario, featureTags);
#line 25
this.ScenarioInitialize(scenarioInfo);
#line hidden
            if ((TagHelper.ContainsIgnoreTag(tagsOfScenario) || TagHelper.ContainsIgnoreTag(featureTags)))
            {
                testRunner.SkipScenario();
            }
            else
            {
                await this.ScenarioStartAsync();
#line 26
 await testRunner.GivenAsync(string.Format("there is {0} blue items", count), ((string)(null)), ((Reqnroll.Table)(null)), "Given ");
#line hidden
            }
            await this.ScenarioCleanupAsync();
        }

but I don't understand this, this code works with a .net48 test project

      [Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
      [Microsoft.VisualStudio.TestTools.UnitTesting.DescriptionAttribute("Another scenario")]
      [Microsoft.VisualStudio.TestTools.UnitTesting.TestPropertyAttribute("FeatureTitle", "Calculator")]
      [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("1", new string[] {
                "mytag"})]
      [Microsoft.VisualStudio.TestTools.UnitTesting.DataRowAttribute("2", null)]
      public async System.Threading.Tasks.Task AnotherScenario(string count, string[] exampleTags)
      {
         string[] tagsOfScenario = exampleTags;
         System.Collections.Specialized.OrderedDictionary argumentsOfScenario = new System.Collections.Specialized.OrderedDictionary();
         argumentsOfScenario.Add("count", count);
         // ....
      }

[gasparnagy]

@livioc ah yes, thx. this is a bug and I know why. I'll fix. (I have no idea why this was working on .net48)

[gasparnagy]

@livioc I fixed it now. I could only do the related test in my new "System Test" concept in #62 (a214d47), but that will be merged at some point.

If you have a little more time, please retest it. You can also have a look at my system test and give feedback if such testing would be useful/efficient for generator changes.