Add tests to CI

* Add Runtime Test to CI * Fix: Add Runtime Test to CI * Limit Runtime test to net462 * Limit Runtime test to net6.0 * Add Generator and Plugin tests * Add ExternalData plugin tests * fix: Add ExternalData plugin tests * Add Specs job * Restore dependencies & build for specs * set .NET SDK version * run all xUnit Specs test * add test reporter * add permission for reporter * Add tests to quarantaine * exclude tests require MsBuild * duplicated scenario to be able to filter it out on CI build * add nunit and xunit specs * run all specs * make mstest deployment item work on linux? * make mstest deployment item work on linux, fix 2 * fix NUnit filter * fix path in test * try to use env for specs filter * run all specs
reqnroll · Jan 17, 2024 · 6d1d080 · 6d1d080
1 parent a881e72
commit 6d1d080
Show file tree

Hide file tree

Showing 10 changed files with 198 additions and 23 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -5,7 +5,13 @@ on:
     branches: [ "main" ]
   pull_request:
     branches: [ "main" ]
-
+
+permissions:
+  checks: write
+
+env:
+  SPECS_FILTER: "" # use for testing CI: "&Category=quarantaine"
+
 jobs:
   build:
 
@@ -23,5 +29,118 @@ jobs:
       run: dotnet restore
     - name: Build
       run: dotnet build --no-restore
-    # - name: Test
-    #   run: dotnet test --no-build --verbosity normal
+    - name: Runtime Tests
+      run: dotnet test ./Tests/Reqnroll.RuntimeTests/Reqnroll.RuntimeTests.csproj --no-build --verbosity normal -f net6.0
+    - name: Plugin Tests
+      run: dotnet test ./Tests/Reqnroll.PluginTests/Reqnroll.PluginTests.csproj --no-build --verbosity normal -f net6.0
+    - name: Generator Tests
+      run: dotnet test ./Tests/Reqnroll.GeneratorTests/Reqnroll.GeneratorTests.csproj --no-build --verbosity normal -f net6.0
+    - name: ExternalData Plugin Tests
+      run: dotnet test ./Plugins/Reqnroll.ExternalData/Reqnroll.ExternalData.ReqnrollPlugin.UnitTests/Reqnroll.ExternalData.ReqnrollPlugin.UnitTests.csproj --no-build --verbosity normal -f net6.0
+
+  specs-xunit:
+    runs-on: ubuntu-latest
+    needs: build
+
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0 # avoid shallow clone so nbgv can do its work.
+    - name: Setup .NET
+      uses: actions/setup-dotnet@v3
+      with:
+        dotnet-version: | 
+          6.0.x
+    - name: Set .NET 6 SDK
+      run: dotnet new globaljson --sdk-version 6.0.418
+    - name: Restore dependencies
+      run: dotnet restore
+    - name: Build
+      run: dotnet build --no-restore
+    - name: xUnit Specs
+      shell: pwsh
+      run: dotnet test ./Tests/Reqnroll.Specs/Reqnroll.Specs.csproj --no-build --verbosity normal -f net6.0 --filter "Category=xUnit&Category=Net60&Category!=requiresMsBuild$env:SPECS_FILTER" --logger "trx;LogFileName=specs-xunit-results.trx"
+    - uses: actions/upload-artifact@v3
+      if: success() || failure()      
+      with:
+        name: specs-results
+        path: "**/specs-xunit-results.trx"
+    - name: Test Report
+      uses: dorny/test-reporter@v1
+      if: success() || failure() 
+      with:
+        name: xUnit Specs
+        path: "**/specs-xunit-results.trx"
+        reporter: dotnet-trx   
+
+  specs-nunit:
+    runs-on: ubuntu-latest
+    needs: build
+
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0 # avoid shallow clone so nbgv can do its work.
+    - name: Setup .NET
+      uses: actions/setup-dotnet@v3
+      with:
+        dotnet-version: | 
+          6.0.x
+    - name: Set .NET 6 SDK
+      run: dotnet new globaljson --sdk-version 6.0.418
+    - name: Restore dependencies
+      run: dotnet restore
+    - name: Build
+      run: dotnet build --no-restore
+    - name: NUnit Specs
+      shell: pwsh
+      run: dotnet test ./Tests/Reqnroll.Specs/Reqnroll.Specs.csproj --no-build --verbosity normal -f net6.0 --filter "Category=NUnit3&Category=Net60&Category!=requiresMsBuild$env:SPECS_FILTER" --logger "trx;LogFileName=specs-nunit-results.trx"
+    - uses: actions/upload-artifact@v3
+      if: success() || failure()      
+      with:
+        name: specs-results
+        path: "**/specs-nunit-results.trx"
+    - name: Test Report
+      uses: dorny/test-reporter@v1
+      if: success() || failure() 
+      with:
+        name: NUnit Specs
+        path: "**/specs-nunit-results.trx"
+        reporter: dotnet-trx   
+
+
+  specs-mstest:
+    runs-on: ubuntu-latest
+    needs: build
+
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0 # avoid shallow clone so nbgv can do its work.
+    - name: Setup .NET
+      uses: actions/setup-dotnet@v3
+      with:
+        dotnet-version: | 
+          6.0.x
+    - name: Set .NET 6 SDK
+      run: dotnet new globaljson --sdk-version 6.0.418
+    - name: Restore dependencies
+      run: dotnet restore
+    - name: Build
+      run: dotnet build --no-restore
+    - name: MsTest Specs
+      shell: pwsh
+      run: dotnet test ./Tests/Reqnroll.Specs/Reqnroll.Specs.csproj --no-build --verbosity normal -f net6.0 --filter "Category=MsTest&Category=Net60&Category!=requiresMsBuild$env:SPECS_FILTER" --logger "trx;LogFileName=specs-mstest-results.trx"
+    - uses: actions/upload-artifact@v3
+      if: success() || failure()      
+      with:
+        name: specs-results
+        path: "**/specs-mstest-results.trx"
+    - name: Test Report
+      uses: dorny/test-reporter@v1
+      if: success() || failure() 
+      with:
+        name: MsTest Specs
+        path: "**/specs-mstest-results.trx"
+        reporter: dotnet-trx   
+
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -6,5 +6,8 @@
         "MD025": {
             "front_matter_title": ""
         }
-    }
+    },
+    "cSpell.words": [
+        "Reqnroll"
+    ]
 }
diff --git a/...ternalData.ReqnrollPlugin.UnitTests/Reqnroll.ExternalData.ReqnrollPlugin.UnitTests.csproj b/...ternalData.ReqnrollPlugin.UnitTests/Reqnroll.ExternalData.ReqnrollPlugin.UnitTests.csproj
@@ -1,8 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
 
   <PropertyGroup>
-    <TargetFramework>netcoreapp3.1</TargetFramework>
-
+    <TargetFrameworks>$(Reqnroll_Test_TFM)</TargetFrameworks>
     <AssemblyOriginatorKeyFile>$(Reqnroll_KeyFile)</AssemblyOriginatorKeyFile>
     <SignAssembly>$(Reqnroll_EnableStrongNameSigning)</SignAssembly>
     <PublicSign>$(Reqnroll_PublicSign)</PublicSign>

diff --git a/Reqnroll.Generator/UnitTestProvider/MsTestV2GeneratorProvider.cs b/Reqnroll.Generator/UnitTestProvider/MsTestV2GeneratorProvider.cs
@@ -2,6 +2,7 @@
 using System.CodeDom;
 using System.Collections.Generic;
 using System.Linq;
+using Reqnroll.Utils;
 using Reqnroll.Generator.CodeDom;
 
 namespace Reqnroll.Generator.UnitTestProvider
@@ -112,14 +113,16 @@ public override void SetTestMethod(TestClassGenerationContext generationContext,
                 var deploymentItems = generationContext.CustomData[DEPLOYMENTITEM_TAG] as IEnumerable<string>;
                 foreach (string deploymentItem in deploymentItems)
                 {
-                    var outputDirProvided = deploymentItem.Split(':').Any();
-                    if (outputDirProvided)
+                    var deploymentItemParts = deploymentItem.Split(':');
+                    var itemPath = FileSystemHelper.NormalizeDirectorySeparators(deploymentItemParts[0]);
+                    var outputDir = deploymentItemParts.Length > 1 ? deploymentItemParts[1] : null;
+                    if (outputDir != null)
                     {
-                        CodeDomHelper.AddAttribute(testMethod, DEPLOYMENTITEM_ATTR, deploymentItem.Split(':'));
+                        CodeDomHelper.AddAttribute(testMethod, DEPLOYMENTITEM_ATTR, itemPath, outputDir);
                     }
                     else
                     {
-                        CodeDomHelper.AddAttribute(testMethod, DEPLOYMENTITEM_ATTR, deploymentItem);
+                        CodeDomHelper.AddAttribute(testMethod, DEPLOYMENTITEM_ATTR, itemPath);
                     }
                 }
             }

diff --git a/Reqnroll.Utils/FileSystemHelper.cs b/Reqnroll.Utils/FileSystemHelper.cs
@@ -108,5 +108,20 @@ public static bool FileCompareContent(string filePath1, string fileContent)
             return string.CompareOrdinal(currentFileContent, fileContent) == 0;
 
         }
+
+        public static string NormalizeDirectorySeparators(string path)
+        {
+            if (path == null)
+                return null;
+
+            switch (Path.DirectorySeparatorChar)
+            {
+                case '\\':
+                    return path.Replace('/', '\\');
+                case '/':
+                    return path.Replace('\\', '/');
+            }
+            return path;
+        }
     }
 }
diff --git a/Tests/Reqnroll.Specs/Features/Build systems.feature b/Tests/Reqnroll.Specs/Features/Build systems.feature
@@ -1,6 +1,8 @@
 @dotnetcore
 Feature: Build systems
 
+@quarantaine
+@requiresMsBuild
 @globalusingdirective #MSBuild for VS2019 and Mono throws error CS8652: The feature 'global using directive' is currently in Preview and unsupported.
 Scenario: Use MSBuild for compiling
 	Given there is a scenario in a feature file

diff --git a/Tests/Reqnroll.Specs/Features/LanguageSupport.feature b/Tests/Reqnroll.Specs/Features/LanguageSupport.feature
@@ -1,6 +1,6 @@
 Feature: .NET Language Code-Behind Generation Support
 
-
+@quarantaine
 Scenario Outline: A project with a single scenario should compile successfully in all supported languages and build systems
     Given I have a '<Language>' test project
     And there is a feature file in the project as
@@ -18,12 +18,6 @@ Scenario Outline: A project with a single scenario should compile successfully i
         | Total | Succeeded |
         | 1     | 1         |
 
-@globalusingdirective #MSBuild for VS2019 and Mono throws error CS8652: The feature 'global using directive' is currently in Preview and unsupported.
-Examples:
-    | Description         | Language | Build Command |
-    | C# with MSBuild     | C#       | MSBuild       |
-    | VB.NET with MSBuild | VB.NET   | MSBuild       |
-
 Examples:
     | Description              | Language | Build Command |
     | C# with dotnet build     | C#       | dotnet build  |
@@ -33,3 +27,29 @@ Examples:
     | Description               | Language | Build Command  |
     | C# with dotnet msbuild    | C#       | dotnet msbuild |
     | VB.NET with dotnet msuild | VB.NET   | dotnet msbuild |
+
+# duplicated scenario to be able to filter it out on CI build
+@quarantaine
+@globalusingdirective #MSBuild for VS2019 and Mono throws error CS8652: The feature 'global using directive' is currently in Preview and unsupported.
+@requiresMsBuild
+Scenario Outline: A project with a single scenario should compile successfully in all supported languages and build systems (MsBuild)
+    Given I have a '<Language>' test project
+    And there is a feature file in the project as
+        """
+        Feature: Simple Feature
+        Scenario: Simple Scenario
+            When I do something
+        """
+    And all steps are bound and pass
+
+    When I compile the solution using '<Build Command>'
+    And I execute the tests
+
+    Then the execution summary should contain
+        | Total | Succeeded |
+        | 1     | 1         |
+
+Examples:
+    | Description         | Language | Build Command |
+    | C# with MSBuild     | C#       | MSBuild       |
+    | VB.NET with MSBuild | VB.NET   | MSBuild       |
diff --git a/Tests/Reqnroll.Specs/Features/MultipleSpecsProjects.feature b/Tests/Reqnroll.Specs/Features/MultipleSpecsProjects.feature
@@ -1,17 +1,27 @@
 Feature: Multiple Specs Projects
 
+@quarantaine
 Scenario Outline: Two projects with the same unit test provider
 	Given I have Specs.Project.A and Specs.Project.B using the same unit test provider
 	And Specs.Project.B references Specs.Project.A
 	When I build the solution using '<Build Tool>'
 	Then the build should succeed
 
-	@globalusingdirective #MSBuild for VS2019 and Mono throws error CS8652: The feature 'global using directive' is currently in Preview and unsupported.
-	Examples: 
-	| Build Tool    |
-	| MSBuild       |
-
 	Examples:
 	| Build Tool     |
 	| dotnet build   |
 	| dotnet msbuild |
+
+# duplicated scenario to be able to filter it out on CI build
+@quarantaine
+@requiresMsBuild
+@globalusingdirective #MSBuild for VS2019 and Mono throws error CS8652: The feature 'global using directive' is currently in Preview and unsupported.
+Scenario Outline: Two projects with the same unit test provider (MsBuild)
+	Given I have Specs.Project.A and Specs.Project.B using the same unit test provider
+	And Specs.Project.B references Specs.Project.A
+	When I build the solution using '<Build Tool>'
+	Then the build should succeed
+
+	Examples: 
+	| Build Tool    |
+	| MSBuild       |
diff --git a/Tests/Reqnroll.Specs/Features/UnitTestProviderSpecific/MsTest/DeploymentItem.feature b/Tests/Reqnroll.Specs/Features/UnitTestProviderSpecific/MsTest/DeploymentItem.feature
@@ -39,6 +39,7 @@ Scenario: Should be able to deploy files
          | Succeeded |
          | 1         |
 
+@quarantaine
 @config
 Scenario: Should be able to deploy files to specific folder
     Given there is a Reqnroll project
@@ -56,7 +57,7 @@ Scenario: Should be able to deploy files to specific folder
             [Then(@"the file '(.*)' exists")]
             public void ThenTheFileExists(string fileName)
             {
-                Assert.IsTrue(File.Exists(fileName));
+                Assert.IsTrue(File.Exists(fileName.Replace('\\', Path.DirectorySeparatorChar)));
             }
         }
         """

diff --git a/Tests/Reqnroll.Specs/StepDefinitions/ContentFileSteps.cs b/Tests/Reqnroll.Specs/StepDefinitions/ContentFileSteps.cs
@@ -1,3 +1,5 @@
+using System.IO;
+using Reqnroll.Utils;
 using Reqnroll.TestProjectGenerator.Driver;
 
 namespace Reqnroll.Specs.StepDefinitions
@@ -15,6 +17,7 @@ public ContentFileSteps(ProjectsDriver projectsDriver)
         [Given("there is a content file '(.*)' in the project as")]
         public void GivenThereIsAContentFileInTheProjectAs(string fileName, string fileContent)
         {
+            fileName = FileSystemHelper.NormalizeDirectorySeparators(fileName);
             _projectsDriver.AddFile(fileName, fileContent, "Content");
         }
     }