-
Notifications
You must be signed in to change notification settings - Fork 72
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Changes in preparation for Release 1.8.0 (#235)
* changes from functional testing * fixing bugs * fixing issues with json reporting and grouping * fixing unit tests * final README updates * adding globbing environment variable * adding globbing environment variable
- Loading branch information
Showing
12 changed files
with
219 additions
and
76 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Functional Tests for the config-file-validator | ||
|
|
||
| Manual procedures for functionally testing the config-file-validator. These will eventually be used as requirements for creating automated function tests | ||
|
|
||
| ## Setup | ||
|
|
||
| 1. Build the latest changes in a container `docker build . -t cfv:<feature>` tagging the local container with a tag that indicates what feature is being tested | ||
| 2. Run the container and mount the test directory `docker run -it --rm -v $(pwd)/test:/test --entrypoint sh cfv:<feature>` | ||
|
|
||
| ## Basic Functionality | ||
|
|
||
| This section tests basic validator functionality | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator` | 37 passed and 12 failed | | | ||
| | `/validator /test` | 37 passed and 12 failed | | | ||
| | `/validator /test/fixtures/subdir /test/fixtures/subdir2` | 3 passed and 12 failed | | | ||
| | `/validator --help` | Help is displayed | | | ||
| | `/validator --version` | Output should read "validator version unknown" | | | ||
| | `/validator /badpath` | Error output should read: "An error occurred during CLI execution: Unable to find files: stat /badpath: no such file or directory" | | | ||
| | `/validator -v` | Error "flag provided but not defined: -v" is displayed on the terminal in addition to the help output | | | ||
|
|
||
|
|
||
| ## Reports | ||
|
|
||
| This section validates report output | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --reporter=json:-` | JSON output is produced on the terminal and summary is `"summary": {"passed": 37,"failed": 12}` | | | ||
| | `cd /test && /validator --reporter=standard:-` | Text output is produced on the screen | | | ||
| | `cd /test && /validator --reporter=junit:-` | JUnit XML is produced on the terminal | | | ||
| | `cd /test && /validator --reporter=sarif:-` | Sarif output is produced on the terminal | | | ||
| | `cd /test && /validator --reporter=json:/json_report.json` | JSON is written to `/json_report.json` and summary in the contents of the file reads `"summary": {"passed": 37,"failed": 12}` | | | ||
| | `cd /test && /validator --reporter=standard:/standard_report.txt` | Text is written to `/standard_report.txt` | | | ||
| | `cd /test && /validator --reporter=junit:/junit_report.xml` | JUnit XML is written to `/junit_report.xml` | | | ||
| | `cd /test && /validator --reporter=sarif:/sarif_report.sarif` | Sarif JSON is written to `/sarif_report.sarif` | This is currently failing as the sarif report is written to stdout in addition to the file | | ||
| | `cd /test && /validator --reporter=json:/json_report_2.json --reporter=standard:-` | JSON is written to `/json_report_2.json` and standard text is output to the terminal | | | ||
| | `cd /test && /validator --reporter=bad` | Error message "Wrong parameter value for reporter, only supports standard, json, junit, or sarif" should be displayed in addition to the help output | | | ||
| | `cd /test && /validator --reporter=json --quiet` | Nothing is displayed to the terminal since the `--quiet` flag supresses the output | | | ||
| | `cd /test && /validator --reporter=json:/json_report_3.json --quiet` | Nothing is displayed to the terminal since the `--quiet` flag supresses the output but the `/json_report_3.json` file is populated | | | ||
|
|
||
| ## Grouping | ||
|
|
||
| This section validates organization of the output | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --groupby=filetype` | Results are grouped by file type | | | ||
| | `cd /test && /validator --groupby=pass-fail` | Results are grouped by pass/fail | Bug filed for duplicate summaries | | ||
| | `cd /test && /validator --reporter=standard --groupby=pass-fail` | Results are grouped by pass/fail | Bug filed for duplicate summaries | | ||
| | `cd /test && /validator --groupby=directory` | Results are grouped by directory | | | ||
| | `cd /test && /validator --groupby=filetype,directory` | Results are grouped by file type, then directory | | | ||
| | `cd /test && /validator --groupby=pass-fail,filetype` | Results are grouped by pass-fail, then filetype | | | ||
| | `cd /test && /validator --groupby=pass-fail,directory` | Results are grouped by pass-fail, then directory | | | ||
| | `cd /test && /validator --groupby=pass-fail,directory,filetype` | Results are grouped by pass-fail, then directory, then file type | | | ||
| | `cd /test && /validator --groupby=pass-fail,pass-fail` | Error "Wrong parameter value for groupby, duplicate values are not allowed" is displayed with help output | | | ||
| | `cd /test && /validator --reporter=json --groupby=filetype,directory` | JSON Results are grouped by file type, then directory | This does not work, bug filed | | ||
| | `cd /test && /validator --reporter=junit --groupby=pass-fail` | Error "Wrong parameter value for reporter, groupby is only supported for standard and JSON reports" is displayed with help output | | | ||
| | `cd /test && /validator --reporter=sarif --groupby=directory` | Error "Wrong parameter value for reporter, groupby is only supported for standard and JSON reports" is displayed with help output | | | ||
|
|
||
| ## Depth | ||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd / && /validator --depth=0 /test` | Nothing is displayed since there are no config files at the root of the `/test` directory and recursion is disabled with depth set to 0 | | | ||
| | `cd / && /validator --depth=1 /test` | Files in `/test/fixtures/` are validated | | | ||
| | `cd / && /validator --depth=2 /test` | Files in `/test/fixtures/*` directories are validated | | | ||
|
|
||
| ## Globbing | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --globbing "fixtures/**/*.json"` | All json files in subdirectories are displayed | | | ||
| | `cd /test && /validator --globbing "fixtures/**/*.json" /test/fixtures/subdir2` | All json files in subdirectories are displayed and other files in `/test/fixtures/subdir2` are displayed | | | ||
| | `cd /test && /validator --groupby=pass-fail --globbing "fixtures/**/*.json"` | All json files in subdirectories are displayed and grouped by pass/fail | | | ||
| | `cd /test && /validator --reporter=json --globbing "fixtures/**/*.json"` | All json files in subdirectories are displayed as a JSON report | | | ||
| | `cd /test && /validator "fixtures/**/*.json"` | Error "An error occurred during CLI execution: Unable to find files: stat fixtures/**/*.json: no such file or directory" should be displayed when using glob patterns without flag enabling it | | | ||
| | `cd /test && /validator --exclude-file-types=json --globbing "fixtures/**/*.json"` | Error "the -globbing flag cannot be used with --exclude-dirs or --exclude-file-types" is displayed | | | ||
| | `cd /test && /validator --exclude-dirs=subdir2 --globbing "fixtures/**/*.json"` | Error "the -globbing flag cannot be used with --exclude-dirs or --exclude-file-types" is displayed | | | ||
|
|
||
|
|
||
| ## Environment Variables | ||
|
|
||
| Run `unset <var>` to unset the previous var before testing | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `export CFV_REPORTER=json:- && /validator /test` | JSON output should display on the terminal | | | ||
| | `export CFV_REPORTER=json:/test_env_var.json && /validator /test` | JSON output should be written to `/test_env_var.json` and NOT displayed on the terminal | | | ||
| | `export CFV_GLOBBING=true && cd /test && /validator "fixtures/**/*.json"` | Results should include json files in all subdirectories for fixtures | This does not work and a bug has been filed | | ||
| | `export CFV_QUIET=true && cd /test && /validator` | No output should be displayed on the terminal | | | ||
| | `export CFV_QUIET=false && cd /test && /validator` | Output should be displayed on the terminal | | | ||
| | `export CFV_GROUPBY=pass-fail && cd /test && /validator` | Output should be displayed on the terminal and grouped by pass-fail | | | ||
| | `export CFV_DEPTH=0 && cd /test && /validator` | Output should only display config files at the root of `/test/fixtures` | | | ||
| | `export CFV_EXCLUDE_DIRS=subdir2,subdir && cd /test && /validator` | `subdir` and `subdir2` directories should be excluded | | | ||
| | `export CFV_EXCLUDE_FILE_TYPES=yml,yaml,json,toml,properties,hocon,csv,hcl,ini,env,plist,editorconfig,xml && cd /test && /validator` | No output since all types are excluded | | | ||
| | `export CFV_EXCLUDE_FILE_TYPES=yml,yaml,json,toml,properties,hocon,csv,hcl,ini,env,plist,editorconfig && cd /test && /validator --exclude-file-types=""` | All config files should be displayed since the argument overrides the environment variable | | | ||
|
|
||
| ## Exclude Dirs | ||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --exclude-dirs=baddir` | Non-existent subdirectory is ignored | | | ||
| | `cd /test && /validator --exclude-dirs=subdir,subdir2` | `subdir` and `subdir2` directories are ignored | | | ||
| | ` cd /test && /validator --exclude-dirs=test /` | `test` subdirectory is excluded from root directory search path `/` | | | ||
|
|
||
|
|
||
| ## Exclude File Types | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --exclude-file-types=xml` | XML validation should be skipped | | | ||
| | `cd /test && /validator --exclude-file-types=yml` | `.yaml` and `.yml` should be excluded from validation | | | ||
| | `cd /test && /validator --exclude-file-types=YaML` | `.yaml` and `.yml` should be excluded from validation since argument values are not case sensitive | | | ||
|
|
||
|
|
||
| ## Other Flags | ||
|
|
||
| | Test | Expected Result | Notes | | ||
| | ---- | --------------- | ----- | | ||
| | `cd /test && /validator --quiet` | Nothing is displayed to the terminal since the `--quiet` flag supresses the output | | | ||
| | `cd /test && /validator /badpath --quiet` | Error "An error occurred during CLI execution: Unable to find files: stat /badpath: no such file or directory" is output to the terminal even through `--quiet` was enabled | | | ||
|
|
Oops, something went wrong.