DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation.
ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace
string
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate
policy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullName
bool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex
string
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex
string
IncludeTestRegex is used to include tests based on a regular expression.
repeatCount
int
RepeatCount indicates how many times the tests should be executed.
testFile
string
TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed.
Template determines whether resources should be considered for templating.
file
string
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
OperationBase defines common elements to all operations.
Field
Type
Required
Inline
Description
description
string
Description contains a description of the operation.
continueOnError
bool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Container in pod to get logs from else --all-containers is used.
tail
int
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
cluster
string
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation.
ReportName defines the name of report to create. It defaults to "chainsaw-report".
namespace
string
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplate
policy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullName
bool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegex
string
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegex
string
IncludeTestRegex is used to include tests based on a regular expression.
repeatCount
int
RepeatCount indicates how many times the tests should be executed.
testFile
string
TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed.
Template determines whether resources should be considered for templating.
file
string
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as "manifest/*.yaml" for all YAML files within the "manifest" directory.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
OperationBase defines common elements to all operations.
Field
Type
Required
Inline
Description
description
string
Description contains a description of the operation.
continueOnError
bool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Container in pod to get logs from else --all-containers is used.
tail
int
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
cluster
string
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
WaitForJsonPath represents parameters for waiting on a json path of a resource.
Field
Type
Required
Inline
Description
path
string
Path defines the json path to wait for, e.g. '{.status.phase}'.
value
string
Value defines the expected value to wait for, e.g., "Running".
\ No newline at end of file
diff --git a/main/search/search_index.json b/main/search/search_index.json
index 7dd8b7984..10d50ee7f 100644
--- a/main/search/search_index.json
+++ b/main/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"cicd/gh-action/","title":"GitHub action","text":"
A GitHub action is available to easily install Chainsaw in your workflows.
The GitHub action is available at kyverno/action-install-chainsaw or in the marketplace.
If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.
Input Description releasechainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"community/","title":"Community","text":"
Chainsaw has a growing community and we would definitely love to see you join and contribute.
Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.
Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters
To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.
This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.
If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces
Chainsaw, developed by Kyverno, is an advanced end-to-end testing tool for Kubernetes. Our community plays a crucial role in shaping the project by reporting bugs, suggesting features, and improving documentation.
We aim to make our issue tracker, discussion board, and documentation well-structured and easy to navigate. By following our guidelines, you can help us address your requests efficiently.
"},{"location":"community/contribute/#how-you-can-contribute","title":"How you can contribute","text":"
We appreciate your efforts in reporting bugs, requesting features, and engaging in discussions. Here's how you can contribute:
"},{"location":"community/contribute/#creating-an-issue","title":"Creating an issue","text":"
Something is not working?
Report a bug in Chainsaw by creating an issue with a reproduction
Report a bug
Missing information in our docs?
Report missing information or potential inconsistencies in our documentation
Report a docs issue
Want to submit an idea?
Propose a change, feature request, or suggest an improvement
Request a change
Have a question or need help?
Ask a question on our discussion board and get in touch with our community
Before interacting within the project, please consider the following questions to ensure you're using the correct issue template and providing all necessary information.
Issues, discussions, and comments are forever
Please note that everything you write is permanent and will remain for everyone to read \u2013 forever. Therefore, please always be nice and constructive, follow our contribution guidelines, and comply with our Code of Conduct.
"},{"location":"community/contribute/#before-creating-an-issue","title":"Before creating an issue","text":"
Are you using the appropriate issue template, or is there another issue template that better fits the context of your request?
Have you checked if a similar bug report or change request has already been created, or have you stumbled upon something that might be related?
Did you fill out every field as requested and provide all additional information needed to comprehend your request?
"},{"location":"community/contribute/#before-asking-a-question","title":"Before asking a question","text":"
Is the topic a question for our discussion board, or is it a bug report or change request that should be raised on our [issue tracker]?
Is there an open discussion on the topic of your request? If the answer is yes, does your question match the direction of the discussion, or should you open a new discussion?
Did you provide our community with all the necessary information to understand your question and help you quickly, or can you make it easier to help you?
Is your comment relevant to the topic of the current page, post, issue, or discussion, or is it better to create a new issue or discussion?
Does your comment add value to the conversation? Is it constructive and respectful to our community and maintainers? Could you just use a reaction instead?
"},{"location":"community/contribute/#rights-and-responsibilities","title":"Rights and responsibilities","text":"
As maintainers, we are entrusted with the responsibility to moderate communication within our community, including the authority to close, remove, reject, or edit issues, discussions, comments, commits, and to block users who do not align with our contribution guidelines and our Code of Conduct. This role requires us to be actively involved in maintaining the integrity and positive atmosphere of our community. Upholding these standards decisively ensures a respectful and inclusive environment for all members.
"},{"location":"community/contribute/#code-of-conduct","title":"Code of Conduct","text":"
Our Code of Conduct outlines the expectation for all community members to treat one another with respect, employing inclusive and welcoming language. Our commitment is to foster a positive and supportive environment, free of inappropriate, offensive, or harmful behavior.
We take any violations seriously and will take appropriate action in response to uphold these values.1
"},{"location":"community/contribute/#incomplete-issues-and-duplicates","title":"Incomplete issues and duplicates","text":"
We have invested significant time and effort in the setup of our contribution process, ensuring that we assess the essential requirements for reviewing and responding to issues effectively. Each field in our issue templates is thoughtfully designed to help us fully understand your concerns and the nature of your matter. We encourage all members to utilize the search function before submitting new issues or starting discussions to help avoid duplicates. Your cooperation is crucial in keeping our community's discussions constructive and organized.
Mandatory completion of issue templates: We need all of the information required in our issue templates because it ensures that every user and maintainer, regardless of their experience, can understand the content and severity of your bug report or change request.
Closing incomplete issues: We reserve the right to close issues lacking essential information, such as but not limited to [minimal reproductions] or those not adhering to the quality standards and requirements specified in our issue templates. Such issues can be reopened once the missing information has been provided.
Handling duplicates: To maintain organized and efficient communication within our [issue tracker] and discussion board, we reserve the right to close any duplicated issues or lock duplicated discussions. Opening multiple channels to ask the same question or report the same issue across different forums hinders our ability to manage and address community concerns effectively. This approach is vital for efficient time management, as duplicated questions can consume the time of multiple team members simultaneously. Ensuring that each issue or discussion is unique and progresses with new information helps us to maintain focus and support our community.
We further reserve the right to immediately close discussions or issues that are reopened without providing new information or simply because users have not yet received a response to their issue/question, as the issue is marked as incomplete.
Limitations of automated tools: While we believe in the value and efficiency that automated tools bring to identifying potential issues (such as those identified by Lighthouse, Accessibility tools, and others), simply submitting an issue generated by these tools does not constitute a complete bug report. These tools sometimes produce verbose outputs and may include false positives, which necessitate a critical evaluation. You are of course welcome to attach generated reports to your issue. However, this does not substitute the requirement for a minimal reproduction or a thorough discussion of the findings. We reserve the right to mark these issues as incomplete and close them. This practice ensures that we are addressing genuine concerns with precision and clarity, rather than navigating through extensive automated outputs.
Warning and blocking policy: Given the increasing popularity of our project and our commitment to a healthy community, we've defined clear guidelines on how we proceed with violations:
1.1. First warning: Users displaying repeated inappropriate, offensive, or harmful behavior will receive a first warning. This warning serves as a formal notice that their behavior is not in alignment with our community standards and Code of Conduct. The first warning is permanent.
1.2. Second warning and opportunity for resolution: If the behavior persists, a second warning will be issued. Upon receiving the second warning, the user will be given a 5-day period for reflection, during which they are encouraged to publicly explain or apologize for their actions. This period is designed to offer an opportunity for openly clearing out any misunderstanding.
1.3. Blocking: Should there be no response or improvement in behavior following the second warning, we reserve the right to block the user from the community and repository. Blocking is considered a last resort, used only when absolutely necessary to protect the community's integrity and positive atmosphere.
Blocking has been an exceptionally rare necessity in our overwhelmingly positive community, highlighting our preference for constructive dialogue and mutual respect. It aims to protect our community members and team.\u00a0\u21a9
You can contribute to Chainsaw by making a pull request that will be reviewed by maintainers and integrated into the main repository when the changes made are approved. You can contribute bug fixes, documentation changes, or new functionalities.
Considering a pull request
Before deciding to spend effort on making changes and creating a pull request, please discuss what you intend to do. If you are responding to what you think might be a bug, please issue a bug report first. If you intend to work on documentation, create a documentation issue. If you want to work on a new feature, please create a change request.
Keep in mind the guidance given and let people advise you. It might be that there are easier solutions to the problem you perceive and want to address. It might be that what you want to achieve can already be done by configuration or [customization].
"},{"location":"community/making-a-pull-request/#learning-about-pull-requests","title":"Learning about pull requests","text":"
Pull requests are a concept layered on top of Git by services that provide Git hosting. Before you consider making a pull request, you should familiarize yourself with the documentation on GitHub, the service we are using. The following articles are of particular importance:
Forking a repository
Creating a pull request from a fork
Creating a pull request
Note that they provide tailored documentation for different operating systems and different ways of interacting with GitHub. We do our best in the documentation here to describe the process as it applies to Chainsaw but cannot cover all possible combinations of tools and ways of doing things. It is also important that you understand the concept of a pull-request in general before continuing.
In the following, we describe the general process for making pull requests. The aim here is to provide the 30k ft overview before describing details later on.
"},{"location":"community/making-a-pull-request/#preparing-changes-and-draft-pr","title":"Preparing changes and draft PR","text":"
The diagram below describes what typically happens to repositories in the process or preparing a pull request. We will be discussing the review-revise process below. It is important that you understand the overall process first before you worry about specific commands. This is why we cover this first before providing instructions below.
sequenceDiagram\n autonumber\n\n participant chainsaw\n participant PR\n participant fork\n participant local\n\n chainsaw ->> fork: fork on GitHub\n fork ->> local: clone to local\n local ->> local: branch\n loop prepare\n loop push\n loop edit\n local ->> local: commit\n end\n local ->> fork: push\n end\n chainsaw ->> fork: merge in any changes\n fork ->>+ PR: create draft PR\n PR ->> PR: review your changes\n end
Fork the Repository: Fork the Chainsaw repository on GitHub to create your own copy.
Clone to Local: Clone your fork to your local machine.
Create a Branch: Create a topic branch for your changes.
Set Up Development Environment: Follow the instructions to set up a development environment.
Iterate and Commit: Make incremental changes and commit them with meaningful messages.
Push Regularly: Push your commits to your fork regularly.
Merge Changes from Upstream: Regularly merge changes from the original Chainsaw repository to avoid conflicts.
Create a Draft Pull Request: Once satisfied with your changes, create a draft pull request for early feedback.
Review and Revise: Review your work critically, address feedback, and refine your changes.
Once you are happy with your changes, you can move to the next step, finalizing your pull request and asking for a more formal and detailed review. The diagram below shows the process:
Chainsaw, developed by Kyverno, is an actively maintained project that we constantly strive to improve. With a project of this size and complexity, bugs may occur. If you think you have discovered a bug, you can help us by submitting an issue in our public issue tracker, following this guide.
"},{"location":"community/reporting-a-bug/#before-creating-an-issue","title":"Before Creating an Issue","text":"
With numerous users, issues are created regularly. The maintainers of this project strive to address bugs promptly. By following this guide, you will know exactly what information we need to help you quickly.
Please do the following before creating an issue:
"},{"location":"community/reporting-a-bug/#upgrade-to-latest-version","title":"Upgrade to Latest Version","text":"
Chances are that the bug you discovered was already fixed in a subsequent version. Before reporting an issue, ensure that you're running the [latest version] of Chainsaw. Consult our [upgrade guide] to learn how to upgrade to the latest version.
Bug fixes are not backported
Please understand that only bugs that occur in the latest version of Chainsaw will be addressed. Also, to reduce duplicate efforts, fixes cannot be backported to earlier versions.
If you're using customizations like additional configurations, remove them before reporting a bug. We can't offer official support for bugs that might hide in your overrides, so make sure to omit custom settings from your configuration files.
Don't be shy to ask on our discussion board for help if you run into problems.
"},{"location":"community/reporting-a-bug/#search-for-solutions","title":"Search for Solutions","text":"
At this stage, we know that the problem persists in the latest version and is not caused by any of your customizations. However, the problem might result from a small typo or a syntactical error in a configuration file.
Before creating a bug report, save time for us and yourself by doing some research:
Search our documentation for relevant sections related to your problem. Ensure everything is configured correctly.
[Search our issue tracker] as another user might have already reported the same problem.
[Search our discussion board] to see if other users are facing similar issues and find possible solutions.
Keep track of all search terms and relevant links; you'll need them in the bug report.
If you still haven't found a solution to your problem, create an issue. It's now likely that you've encountered something new. Read the following section to learn how to create a complete and helpful bug report.
We have created a new issue template to make the bug reporting process as simple as possible and more efficient for our community and us. It consists of the following parts:
A good title is short and descriptive. It should be a one-sentence executive summary of the issue, so the impact and severity of the bug can be inferred from the title.
Example Clear Chainsaw apply command fails with specific CRD Wordy The apply command in Chainsaw fails when used with a certain Custom Resource Definition Unclear Command does not work Useless Help"},{"location":"community/reporting-a-bug/#context","title":"Context optional","text":"
Before describing the bug, you can provide additional context to help us understand what you were trying to achieve. Explain the circumstances under which you're using Chainsaw, and what you think might be relevant. Don't describe the bug here.
Provide a clear, focused, specific, and concise summary of the bug you encountered. Explain why you think this is a bug that should be reported to Chainsaw, and not to one of its dependencies. Follow these principles:
Explain the what, not the how \u2013 don't explain how to reproduce the bug here, we're getting there. Focus on articulating the problem and its impact.
Keep it short and concise \u2013 if the bug can be precisely explained in one or two sentences, perfect. Don't inflate it.
One bug at a time \u2013 if you encounter several unrelated bugs, create separate issues for them.
Share links to relevant sections of our documentation and any related issues or discussions. This helps us improve our documentation and understand the problem better.
A minimal reproduction is essential for a well-written bug report, as it allows us to recreate the conditions necessary to inspect the bug. Follow the guide to create a reproduction:
After creating the reproduction, you should have a .zip file, ideally not larger than 1 MB. Drag and drop the .zip file into the issue field, which will automatically upload it to GitHub.
Don't share links to repositories
While linking to a repository is a common practice, we currently don't support this. The reproduction, created using the built-in info plugin, contains all necessary environment information.
"},{"location":"community/reporting-a-bug/#steps-to-reproduce","title":"Steps to Reproduce","text":"
List specific steps to follow when running your reproduction to observe the bug. Keep the steps concise and ensure nothing is left out. Use simple language and focus on continuity.
If the bug only occurs in specific browsers, let us know which ones are affected. This field is optional, as it is only relevant for bugs that do not involve a crash when previewing or building your site.
Incognito Mode
Verify that the bug is not caused by a browser extension by switching to incognito mode. If the bug disappears, it is likely caused by an extension.
The Chainsaw documentation includes extensive information on features, configurations, customizations, and more. If you have found an inconsistency or see room for improvement, please follow this guide to submit an issue on our issue tracker.
Reporting a documentation issue is usually less involved than reporting a bug, as we don't need a [reproduction]. Please thoroughly read this guide before creating a new documentation issue, and provide the following information as part of the issue:
A good title should be a short, one-sentence description of the issue, containing all relevant information and keywords to simplify the search in our issue tracker.
Example Clear Clarify resource templating setup in Chainsaw Unclear Missing information in the docs Useless Help"},{"location":"community/reporting-a-docs-issue/#description","title":"Description","text":"
Provide a clear and concise summary of the inconsistency or issue you encountered in the documentation or the documentation section that needs improvement. Explain why you think the documentation should be adjusted and describe the severity of the issue:
Keep it short and concise \u2013 if the inconsistency or issue can be precisely explained in one or two sentences, perfect. Maintainers and future users will be grateful for having to read less.
One issue at a time \u2013 if you encounter several unrelated inconsistencies, please create separate issues for them.
Why we need this: describing the problem clearly and concisely is a prerequisite for improving our documentation \u2013 we need to understand what's wrong so we can fix it.
After you describe the documentation section that needs to be adjusted, share the link to this specific documentation section and other possibly related sections. Use anchor links (permanent links) where possible, as it simplifies discovery.
Why we need this: providing the links to the documentation helps us understand which sections of our documentation need to be adjusted, extended, or overhauled.
Now that you have provided us with the description and links to the documentation sections, you can help us, maintainers, and the community by proposing an improvement. You can sketch out rough ideas or write a concrete proposal. This field is optional but very helpful.
Why we need this: an improvement proposal can be beneficial for other users who encounter the same issue, as they offer solutions before we maintainers can update the documentation.
Thanks for following the guide and providing valuable feedback for our documentation \u2013 you are almost done. The checklist ensures that you have read this guide and have worked to your best knowledge to provide us with every piece of information we need to improve it.
Chainsaw by Kyverno is a powerful tool for end-to-end testing. Serving a wide range of use cases, we value every idea or contribution from our community. Please follow this guide before submitting your change request in our public issue tracker. This helps us better understand the proposed change and how it will benefit our community.
"},{"location":"community/requesting-a-change/#before-creating-an-issue","title":"Before Creating an Issue","text":"
Before you invest time in submitting a change request, answer these questions to determine if your idea is a good fit for Chainsaw and matches the project's philosophy and tone.
"},{"location":"community/requesting-a-change/#its-not-a-bug-its-a-feature","title":"It's Not a Bug, It's a Feature","text":"
Change requests suggest minor adjustments, new features, or influence the project's direction. They are not intended for reporting bugs. Refer to our bug reporting guide for that.
"},{"location":"community/requesting-a-change/#look-for-sources-of-inspiration","title":"Look for Sources of Inspiration","text":"
If your idea is implemented in another tool or framework, collect information on its implementation. This helps us evaluate its fit more quickly.
"},{"location":"community/requesting-a-change/#connect-with-our-community","title":"Connect with Our Community","text":"
Our discussion board is the best place to connect with our community. Seeking input from other users helps implement features that benefit a larger number of users.
A good title is short and descriptive, summarizing the idea so the potential impact and benefit can be inferred.
Example Clear Support for resource templating in Chainsaw Wordy Add support for templating resources in Chainsaw for easier testing Unclear Improve templating Useless Help"},{"location":"community/requesting-a-change/#context","title":"Context optional","text":"
Provide additional context to help us understand what you are trying to achieve. Explain the circumstances and relevant settings without describing the change request itself.
Provide a detailed and clear description of your idea. Explain why your idea is relevant to Chainsaw and should be implemented here, not in one of its dependencies.
Explain the what, not the why \u2013 focus on describing the change request precisely.
Keep it short and concise \u2013 be brief and to the point.
One idea at a time \u2013 if you have multiple ideas, open separate change requests for each.
Explain how your change request would work from an author's and user's perspective. What is the expected impact, and why does it benefit other users? Would it potentially break existing functionality?
If you have any visuals, such as sketches, screenshots, mockups, or external assets, present them in this section. If you have seen this change used in other tools, showcase and describe its implementation.
Thanks for following the guide and creating a high-quality change request. The checklist ensures that you have read this guide and provided all necessary information for us to review your idea.
Your change request got rejected? We're sorry for that. We understand it can be frustrating, but we always need to consider the needs of our entire community. If you're unsure why your change request was rejected, please ask for clarification.
We consider the following principles when evaluating change requests:
Alignment with the project's vision and tone
Compatibility with existing features and plugins
Compatibility with all screen sizes and browsers
Effort of implementation and maintenance
Usefulness to the majority of users
Simplicity and ease of use
Accessibility
If your idea was rejected, you can always implement it via [customization]. If you're unsure how or want to know if someone has already done it, get in touch with our community on the discussion board.
In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.
Cleanup options contain the configuration used by Chainsaw for cleaning up resources.
"},{"location":"configuration/options/cleanup/#supported-elements","title":"Supported elements","text":"Element Default Description skipDeletefalse If set, do not delete the resources after running a test. delayBeforeCleanup DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts."},{"location":"configuration/options/cleanup/#delay-before-cleanup","title":"Delay before cleanup","text":"
At the end of each test, Chainsaw will delete the resources it created during the test.
When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update its internal state.
Every cluster is registered by name and supports the following elements:
Element Default Description kubeconfigstring Kubeconfig is the path to the referenced file. contextstring Context is the name of the context to use."},{"location":"configuration/options/clusters/#configuration","title":"Configuration","text":""},{"location":"configuration/options/clusters/#with-file","title":"With file","text":"
apiVersion: chainsaw.kyverno.io/v1alpha2\nkind: Configuration\nmetadata:\n name: custom-config\nspec:\n clusters:\n # this cluster will use the default (current) context\n # configured in the kubeconfig file\n cluster-1:\n kubeconfig: /path/to/kubeconfig-1\n # this cluster will use the context named `context-2`\n # in the kubeconfig file\n cluster-2:\n kubeconfig: /path/to/kubeconfig-2\n context: context-2\n
Deletion options determine the configuration used by Chainsaw for deleting resources.
"},{"location":"configuration/options/deletion/#supported-elements","title":"Supported elements","text":"Element Default Description propagationBackground Propagation decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation."},{"location":"configuration/options/deletion/#propagation","title":"Propagation","text":"
This element will affect Kubernetes cascading deletion. Supported values are Orphan, Background and Foreground.
Tip
Setting Orphan is probably never a good idea because it would leak resources in the test cluster. Chainsaw uses Background as its default value which is a reasonable choice.
Note that Foreground can be useful to fail when the dependent resources fail to delete.
Discovery options contain the discovery configuration used by Chainsaw when discovering tests in specified folders.
"},{"location":"configuration/options/discovery/#supported-elements","title":"Supported elements","text":"Element Default Description testFilechainsaw-test TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed. fullNamefalse FullName makes use of the full test case folder path instead of the folder name. includeTestRegex IncludeTestRegex is used to include tests based on a regular expression. excludeTestRegex ExcludeTestRegex is used to exclude tests based on a regular expression."},{"location":"configuration/options/discovery/#configuration","title":"Configuration","text":""},{"location":"configuration/options/discovery/#with-file","title":"With file","text":"
Error options contain the global error configuration used by Chainsaw.
"},{"location":"configuration/options/error/#supported-elements","title":"Supported elements","text":"Field Default Description catch Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels."},{"location":"configuration/options/error/#configuration","title":"Configuration","text":""},{"location":"configuration/options/error/#with-file","title":"With file","text":"
Execution options determine how tests are run by Chainsaw.
"},{"location":"configuration/options/execution/#supported-elements","title":"Supported elements","text":"Element Default Description failFastfalse FailFast determines whether the test should stop upon encountering the first failure. parallelauto The maximum number of tests to run at once. repeatCount1 RepeatCount indicates how many times the tests should be executed. forceTerminationGracePeriod ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments."},{"location":"configuration/options/execution/#termination-grace-period","title":"Termination grace period","text":"
Some Kubernetes resources can take time before being terminated. For example, deleting a pod can take time if the underlying container doesn't quit quickly enough.
Chainsaw can override the grace period for the following resource kinds:
Namespace options contain the configuration used by Chainsaw to allocate a namespace for each test.
"},{"location":"configuration/options/namespace/#supported-elements","title":"Supported elements","text":"Element Default Description name Name defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec. template Template defines a template to create the test namespace."},{"location":"configuration/options/namespace/#configuration","title":"Configuration","text":""},{"location":"configuration/options/namespace/#with-file","title":"With file","text":"
Chainsaw can be configured to pause and wait for user input when a failure happens. This is useful when Chainsaw is run locally to allow debugging and troubleshooting failures.
Reporting options contain the configuration used by Chainsaw for reporting.
"},{"location":"configuration/options/report/#supported-elements","title":"Supported elements","text":"Element Default Description formatJSON ReportFormat determines test report format (JSON path ReportPath defines the path. namechainsaw-report ReportName defines the name of report to create. It defaults to \"chainsaw-report\"."},{"location":"configuration/options/report/#configuration","title":"Configuration","text":""},{"location":"configuration/options/report/#with-file","title":"With file","text":"
Templating options contain the templating configuration.
"},{"location":"configuration/options/templating/#supported-elements","title":"Supported elements","text":"Element Default Description enabledtrue Enabled determines whether resources should be considered for templating.
Tip
Templating was disabled by default in v0.1.* but is now enabled by default since v0.2.1.
Timeouts in Chainsaw are specified per type of operation. This is required because the timeout varies greatly depending on the nature of an operation.
For example, applying a manifest in a cluster is expected to execute reasonably fast, while validating a resource can be a much longer operation.
"},{"location":"configuration/options/timeouts/#supported-timeouts","title":"Supported timeouts","text":"Element Default Description apply 5s Used when Chainsaw applies manifests in a cluster assert 30s Used when Chainsaw validates resources in a cluster cleanup 30s Used when Chainsaw removes resources created for a test delete 15s Used when Chainsaw deletes resources from a cluster error 30s Used when Chainsaw validates resources in a cluster exec 5s Used when Chainsaw executes arbitrary commands or scripts"},{"location":"configuration/options/timeouts/#configuration","title":"Configuration","text":""},{"location":"configuration/options/timeouts/#with-file","title":"With file","text":"
The number of concurrent tests can be configured globally using a configuration file or with the --parallel flag.
Alternatively, the concurrent nature of a test can specified at the test level:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # concurrency can be specified per test (`true` or `false`)\n # default value is `true`\n concurrent: false\n # ...\n
All non-concurrent tests are executed first, followed by the concurrent tests running in parallel.
"},{"location":"examples/crds/","title":"Work with CRDs","text":"
New CRDs are not immediately available for use in the Kubernetes API until the Kubernetes API has acknowledged them.
If a CRD is being defined inside of a test step, be sure to wait for it to appear.
The test below applies a CRD and waits for it to become available:
The test below fetches the Kubernetes cluster version using the x_k8s_server_version function. It then uses the minor version retrieved to adapt an assertion based on the value in the $minorversion binding.
Tip
You can implement a ternary operator in JMESPath using an expression like this:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n bindings:\n - name: version\n value: (x_k8s_server_version($config))\n - name: minorversion\n value: (to_number($version.minor))\n steps:\n - try:\n - apply:\n resource:\n apiVersion: v1\n kind: Pod\n metadata:\n name: pod01\n spec:\n containers:\n - name: busybox\n image: busybox:1.35\n # ...\n - assert:\n resource:\n apiVersion: v1\n kind: Pod\n metadata:\n annotations:\n # If the minor version of the Kubernetes cluster against\n # which this is tested is less than 29, the annotation is\n # expected to have the group 'system:masters' in it.\n # Otherwise, due to a change in kubeadm, the group should\n # be 'kubeadm:cluster-admins'.\n kyverno.io/created-by: (($minorversion < `29` && '{\"groups\":[\"system:masters\",\"system:authenticated\"],\"username\":\"kubernetes-admin\"}') || '{\"groups\":[\"kubeadm:cluster-admins\",\"system:authenticated\"],\"username\":\"kubernetes-admin\"}')\n name: pod01\n
"},{"location":"examples/label-selectors/","title":"Work with label selectors","text":"
Chainsaw can filter the tests to run using label selectors.
You can pass label selectors using the --selector flag when invoking the chainsaw test command.
Chainsaw supports registering and using multiple clusters in tests.
We can also register clusters dynamically and combine this with cluster selection to achieve scenarios where clusters are dynamically allocated in a test step, used in the following steps, and cleaned up at the end.
The following test demonstrates such a scenario by creating a local kind cluster in the first, using it in the second step, and configuring a cleanup script to delete the cluster when the test terminates:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n # create a local cluster\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n # register `cleanup` operations to delete the cluster\n # at the end of the test\n cleanup:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n # register the `dynamic` cluster in this step\n - clusters:\n dynamic:\n kubeconfig: ./dynamic\n # and use the `dynamic` cluster for all operations in the step\n cluster: dynamic\n try:\n - apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n namespace: default\n data:\n foo: bar\n - assert:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n namespace: default\n data:\n foo: bar\n
Running the test above will produce the following output:
| 10:44:53 | example | @setup | CREATE | OK | v1/Namespace @ chainsaw-useful-seahorse\n | 10:44:53 | example | step-1 | TRY | RUN |\n | 10:44:53 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c kind create cluster --name dynamic --kubeconfig ./dynamic\n | 10:45:10 | example | step-1 | SCRIPT | LOG |\n === STDERR\n Creating cluster \"dynamic\" ...\n \u2022 Ensuring node image (kindest/node:v1.27.3) \ud83d\uddbc ...\n \u2713 Ensuring node image (kindest/node:v1.27.3) \ud83d\uddbc\n \u2022 Preparing nodes \ud83d\udce6 ...\n \u2713 Preparing nodes \ud83d\udce6 \n \u2022 Writing configuration \ud83d\udcdc ...\n \u2713 Writing configuration \ud83d\udcdc\n \u2022 Starting control-plane \ud83d\udd79\ufe0f ...\n \u2713 Starting control-plane \ud83d\udd79\ufe0f\n \u2022 Installing CNI \ud83d\udd0c ...\n \u2713 Installing CNI \ud83d\udd0c\n \u2022 Installing StorageClass \ud83d\udcbe ...\n \u2713 Installing StorageClass \ud83d\udcbe\n Set kubectl context to \"kind-dynamic\"\n You can now use your cluster with:\n\n kubectl cluster-info --context kind-dynamic --kubeconfig ./dynamic\n\n Thanks for using kind! \ud83d\ude0a\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | TRY | DONE |\n | 10:45:10 | example | step-2 | TRY | RUN |\n | 10:45:10 | example | step-2 | APPLY | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | CREATE | OK | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | APPLY | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | ASSERT | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | ASSERT | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | TRY | DONE |\n | 10:45:10 | example | step-2 | CLEANUP | RUN |\n | 10:45:10 | example | step-2 | DELETE | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | DELETE | OK | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | DELETE | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | CLEANUP | DONE |\n | 10:45:10 | example | step-1 | CLEANUP | RUN |\n | 10:45:10 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c kind delete cluster --name dynamic\n | 10:45:10 | example | step-1 | SCRIPT | LOG |\n === STDERR\n Deleting cluster \"dynamic\" ...\n Deleted nodes: [\"dynamic-control-plane\"]\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c rm -f ./dynamic\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | CLEANUP | DONE |\n | 10:45:10 | example | @cleanup | DELETE | RUN | v1/Namespace @ chainsaw-useful-seahorse\n | 10:45:11 | example | @cleanup | DELETE | OK | v1/Namespace @ chainsaw-useful-seahorse\n | 10:45:16 | example | @cleanup | DELETE | DONE | v1/Namespace @ chainsaw-useful-seahorse\n
Negative testing is the process of testing cases that are supposed to fail. That is, a test expects errors to happen and if the expected errors don't occur the test must fail.
Chainsaw supports negative testing by letting you decide what should be considered an error or not.
Tip
By default, Chainsaw will consider an operation failed if there was an error executing it (non-zero exit code in scripts and commands, error returned by the API server when calling into Kubernetes, etc...).
The test below expects an error and validates the returned error message:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n content: kubectl get foo\n check:\n ($error != null): true\n ($stderr): |-\n error: the server doesn't have a resource type \"foo\"\n
If for whatever reason, the kubectl get foo doesn't return an error, or the message received in standard error output is not error: the server doesn't have a resource type \"foo\", Chainsaw will consider the operation failed.
If it returns an error and the expected error message, Chainsaw will consider the operation successful.
"},{"location":"examples/negative-testing/#working-with-resources","title":"Working with resources","text":"
The test below tries to apply resources in a cluster but expects the operation to fail:
Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.
The test below uses the x_k8s_list function to query the list of nodes in the cluster. It uses the results to compare the number of nodes found with a known number (1 in this case).
Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.
One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.
Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures that the string '1.28' is found\n # in stdout or else fails\n (contains($stdout, '1.28')): true\n
Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.
- script:\n content: kubectl version\n check:\n # This check ensures that the string '1.25' is NOT found\n # in stdout or else fails\n (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"
In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures the contents of stdout are exactly as shown.\n # Any deviations will cause a failure.\n ($stdout): |-\n Client Version: v1.28.2\n Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"
In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.
kubectl foo\n\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n top\n
Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check bad kubectl command\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This check below ensures that the string 'top' is found in stderr or else fails\n (contains($stderr, 'top')): true\n
Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This checks that the output is exactly as intended.\n ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"examples/values/","title":"Pass data to tests","text":"
Chainsaw can pass arbitrary values when running tests using the --values flag. Values will be available to tests under the $values binding.
This is useful when a test needs to be configured externally.
"},{"location":"examples/values/#invoking-chainsaw","title":"Invoking Chainsaw","text":""},{"location":"examples/values/#read-values-from-a-file","title":"Read values from a file","text":"
chainsaw test --values ./values.yaml\n
"},{"location":"examples/values/#read-from-stdin","title":"Read from stdin","text":"
You can think of bindings as a side context where you can store and retrieve data by name.
This is particularly useful when some data is only known at runtime. For example, to pass data from one operation to another, to implement resource templating, to fetch data from an external system, or anything that needs to be computed at runtime.
The test below illustrates bindings declaration at different levels:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # bindings can be declared at the test level\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n # bindings can also be declared at the step level\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n # bindings can also be declared at the operation level\n bindings:\n - name: awesome\n value: awesome\n env:\n # combined bindings together using the `join` functions and\n # assign the result to the GREETINGS environment variable\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n content: echo $GREETINGS\n
Different operations have a different model passed through the assertion tree.
Please consult the Built-in bindings reference documentation to learn what is available depending on the operation.
"},{"location":"general/checks/#expect-vs-check","title":"Expect vs Check","text":"
While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply, create, delete, patch and update operations. Those operations can operate on files containing multiple resources and every resource can lead to a different result and expectation.
To support more granular checks we use the expect field that contains an array of Expectations.
Every expectation is made of an optional match combined with a check statement.
This way it is possible to control the scope of a check:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - create:\n file: resources.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
In the test above, only config maps are expected to fail. If the resources.yaml file contains other type of resources they are supposed to be created without error (if an error happens for a non config map resource, the operation will be considered a failure).
Chainsaw has a concept of levels and most of the configuration elements and dynamic elements are inherited from one layer to the next in one way or another.
flowchart TD\n Configuration -. Configuration elements are inherited in tests .-> Test\n Test -. Test elements are inherited in test steps .-> Step\n Step -. Step elements are inherited in step operations .-> Operation
The first layer comes from the Chainsaw configuration. You can think about this layer as the global scope and a way to configure how Chainsaw will behave globally.
Under certain circumstances, lower layers will be allowed to consume and/or override elements from upper layers.
By default, Chainsaw will create an ephemeral namespace with a random name for each test, unless a specific namespace name is provided at the global or test level.
One way to control the namespace used to run tests is to specify the name in the Chainsaw configuration Namespace options.
If a namespace name is specified at the configuration level Chainsaw will use it to run the tests (unless an individual test overrides the namespace name).
If the test name is specified in a test spec, Chainsaw will use it to run the test regardless of whether a namespace name was configured at the global level.
Because the name of the namespace is only known at runtime, depending on the resource being manipulated, Chainsaw will eventually inject the namespace name, except if:
The resource below is a namespaced one and has no namespace specified. Chainsaw will automatically inject the namespace name in it:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: chainsaw-quick-start\n # there is no namespace configured and the resource\n # is a namespaced one.\n # Chainsaw will automatically inject the test namespace\ndata:\n foo: bar\n
Operation outputs can be useful for communicating and reusing computation results across operations.
Chainsaw evaluates outputs after an operation has finished executing. The results of output evaluations are registered in the bindings and are made available for the following operations.
An output supports an optional match field. The match statement is used to conditionally assign the output binding.
The test below illustrates output with matching:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n bindings:\n - name: awesome\n value: awesome\n env:\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n # output is used to register a new `$OUTPUT` binding\n outputs:\n # by default, the `$OUTPUT` binding is assigned\n # the content of the standard output\n - name: OUTPUT\n value: ($stdout)\n # if the match statement evaluates to true,\n # the `$OUTPUT` binding will be set to\n # 'YES! chainsaw is awesome'\n - match:\n ($OUTPUT): hello chainsaw is awesome\n name: OUTPUT\n value: YES! chainsaw is awesome\n content: echo $GREETINGS\n - script:\n # output from the previous operation is used\n # to configure an evironment variable\n env:\n - name: INPUT\n value: ($OUTPUT)\n content: echo $INPUT\n
This doesn't encourage file reuse but can be handy, especially when the resource definition is short or when the execution environment doesn't support file system access.
A third option is to use a URL. Chainsaw uses https://github.com/hashicorp/go-getter, it will download the content from the remote service and load it in the operation resources:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n # use an URL\n file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/step/configmap.yaml\n
When using file-based references, it is important to note that the referenced file(s) can declare multiple resources. Internally, Chainsaw will duplicate the operation once per resource.
This is important to keep this in mind, especially when working with bindings and outputs. Bindings and outputs will be evaluated for every operation instance.
Chainsaw simplifies dynamic configuration with native templating support.
In the past, users have created all sorts of hacks using tools like envsubst for dynamic substitution of env-variables. Those workarounds usually lack flexibility and introduce new problems like hiding the real resources from Chainsaw, preventing it from cleaning resources properly.
Templating in Chainsaw solves exactly this kind of problem.
In the template below, we are using the $namespace binding at two different places, effectively injecting the ephemeral namespace name in the name and the data.foo fields:
"},{"location":"guides/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"guides/kuttl-migration/#overview","title":"Overview","text":"
The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.
chainsaw migrate kuttl config
migrates a KUTTL TestSuite to the corresponding Chainsaw Configuration
chainsaw migrate kuttl tests
migrates KUTTL tests to the corresponding Chainsaw Tests
See below for an example test and the corresponding built docs.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: basic\nspec:\n description: This is a very simple test that creates a configmap and checks the content is as expected.\n steps:\n - description: This steps applies the configmap in the cluster and checks the configmap content.\n try:\n - description: Create the configmap.\n apply:\n file: configmap.yaml\n - description: Check the configmap content.\n assert:\n file: configmap-assert.yaml\n
While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.
The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed when an operation consists of multiple actions. For this reason, operations consisting of multiple actions are not allowed.
"},{"location":"operations/#common-fields","title":"Common fields","text":""},{"location":"operations/#continue-on-error","title":"Continue on error","text":"
The continueOnError field determines whether a test step should continue executing or not if the operation fails (in any case the test will be marked as failed).
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n # in case of error the test will be marked as failed\n # but the step will not stop execution and will\n # continue executing the following operations\n - continueOnError: true\n apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n data:\n foo: bar\n
The apply operation defines resources that should be applied to a Kubernetes cluster. If the resource does not exist yet it will be created, otherwise, it will be configured to match the provided configuration.
The full structure of the Apply is documented here.
"},{"location":"operations/apply/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/apply/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Assert is documented here.
"},{"location":"operations/assert/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support | Operation checks support"},{"location":"operations/assert/#templating","title":"Templating","text":"
When working with assert and error operations, the content is already an assertion tree and therefore mostly represents a logical operation. An exception to this rule is for fields participating in the resource selection process.
For this reason, only elements used for looking up the resources from the cluster will be considered for templating. That is, apiVersion, kind, name, namespace and labels.
The full structure of the Command is documented here.
"},{"location":"operations/command/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/command/#kubeconfig","title":"KUBECONFIG","text":"
Unless --no-cluster is specified, Chainsaw always executes commands in the context of a temporary KUBECONFIG, built from the configured target cluster.
This specific KUBECONFIG has a single cluster, auth info and context configured (all named chainsaw).
The full structure of the Create is documented here.
"},{"location":"operations/create/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/create/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - create:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The delete operation defines resources that should be deleted from a Kubernetes cluster.
Warning
The propagation policy is forced to Background because some types default to Orphan (this is the case for unmanaged jobs for example) and we don't want to let dangling pods run in the cluster after cleanup.
The full structure of the Delete is documented here.
"},{"location":"operations/delete/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/delete/#examples","title":"Examples","text":"
The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.
The full structure of the Error is documented here.
"},{"location":"operations/error/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support | Operation checks support"},{"location":"operations/error/#templating","title":"Templating","text":"
When working with assert and error operations, the content is already an assertion tree and therefore mostly represents a logical operation. An exception to this rule is for fields participating in the resource selection process.
For this reason, only elements used for looking up the resources from the cluster will be considered for templating. That is, apiVersion, kind, name, namespace and labels.
The full structure of the Patch is documented here.
"},{"location":"operations/patch/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/patch/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - patch:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Script is documented here.
"},{"location":"operations/script/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/script/#kubeconfig","title":"KUBECONFIG","text":"
Unless --no-cluster is specified, Chainsaw always executes commands in the context of a temporary KUBECONFIG, built from the configured target cluster.
This specific KUBECONFIG has a single cluster, auth info and context configured (all named chainsaw).
The full structure of the Sleep is documented here.
"},{"location":"operations/sleep/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/sleep/#examples","title":"Examples","text":"
The full structure of the Update is documented here.
"},{"location":"operations/update/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/update/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - update:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Describe resource is documented here.
"},{"location":"operations/helpers/describe/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/describe/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
The full structure of the Events resource is documented here.
"},{"location":"operations/helpers/events/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/events/#test-namespace","title":"Test namespace","text":"
When used with a namespaced resource, Chainsaw will default the scope to the ephemeral test namespace.
The full structure of the Get resource is documented here.
"},{"location":"operations/helpers/get/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/get/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
The full structure of the PodLogs resource is documented here.
"},{"location":"operations/helpers/logs/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/logs/#test-namespace","title":"Test namespace","text":"
Chainsaw will default the scope to the ephemeral test namespace.
The full structure of the Wait resource is documented here.
"},{"location":"operations/helpers/wait/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/wait/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
Chainsaw allows declaring complex assertions with a simple and no-code approach, allowing assertions based on comparisons beyond simple equality, working with arrays, and other scenarios that could not be achieved before.
Tip
Under the hood, Chainsaw uses kyverno-json assertion trees. Refer to the assertion trees documentation for more details on the supported syntax.
When asking Chainsaw to execute the assertion above, it will look for a deployment named coredns in the kube-system namespace and will compare the existing resource with the (partial) resource definition contained in the assertion.
In this specific case, if the field spec.replicas is set to 2 in the existing resource, the assertion will be considered valid. If it is not equal to 2 the assertion will be considered failed.
This is the most basic assertion Chainsaw can evaluate.
"},{"location":"quick-start/assertion-trees/#slightly-less-basic-assertion","title":"Slightly less basic assertion","text":"
Chainsaw will look up all deployments with the k8s-app: kube-dns label in the kube-system namespace. The assertion will be considered valid if at least one deployment matches the (partial) resource definition contained in the assertion. If none match, the assertion will be considered failed.
Apart from the resource lookup process being a little bit more interesting, this kind of assertion is essentially the same as the previous one. Chainsaw is basically making a decision by comparing an actual and expected resource.
The assertion below will check that the number of replicas for a deployment is greater than 1 AND less than 4.
Chainsaw doesn't need to know the exact expected number of replicas. The (replicas > 1 && replicas < 4) expression will be evaluated until the result is true or the operation timeout expires (making the assertion fail).
Chainsaw offers detailed resource diffs upon assertion failures.
In the example below, the assertion failure message (metadata.annotations.foo: Invalid value: \"null\": Expected value: \"bar\") is augmented with a resource diff.
It provides a clear view of discrepancies between expected and actual resources and gives more context around the specific failure (we can easily identify the owner of the offending pod for example).
You can think of bindings as a side context where you can store and retrieve data based on keys.
This is particularly useful when some data is only known at runtime. For example, to pass data from one operation to another, to implement resource templating, to fetch data from an external system, etc.
Chainsaw offers some built-in bindings you can directly use in your tests but you can also create your own bindings if needed.
The $namespace binding is a good example of a built-in binding provided by Chainsaw. It contains the name of the ephemeral namespace used to execute a test (by default Chainsaw will create an ephemeral namespace for each test).
In the operation below, we are assigning the value of the $namespace binding to an environment variable, and echo it in a script:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n env:\n # assign the value of the `$namespace` binding\n # to the environment variable `FOO`\n - name: FOO\n value: ($namespace)\n content: echo $FOO\n
On top of built-in bindings, you can also create your own ones, combine bindings together, call JMESPath functions using bindings as arguments, etc.
In the test below we create custom bindings at different levels in the test, combine them by calling the join function, assign the result to an environment variable, and echo it in a script:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # bindings can be declared at the test level\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n # bindings can also be declared at the step level\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n # bindings can also be declared at the operation level\n bindings:\n - name: awesome\n value: awesome\n env:\n # combined bindings together using the `join` functions and\n # assign the result to the GREETINGS environment variable\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n content: echo $GREETINGS\n
Let's see how bindings can be useful with resource templating.
"},{"location":"quick-start/cleanup/","title":"Control your cleanup","text":"
Unless configured differently, by default Chainsaw will automatically remove the resources it created after a test finishes.
Cleanup happens in reverse order of creation (created last, cleaned up first). This is important, especially when the controller being tested makes use of finalizers.
Overriding cleanup timeout
Note that Chainsaw performs a blocking deletion, that is, it will wait until the resource is not present anymore in the cluster before proceeding with the next resource cleanup.
Under certain circumstances, automatic cleanup is not enough and we want to execute custom operations.
Chainsaw allows registering cleanup operations that will be run after automatic cleanup. Custom cleanup operations live at the test step level:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # this step will create a local cluster\n - try:\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n # at cleanup time, we want to delete the local cluster we created\n # and remove the associated kubeconfig\n cleanup:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n
Once installed, use chainsaw completion command to generate and register the autocompletion script for the specified shell.
Supported shells are:
bash
fish
powershell
zsh
"},{"location":"quick-start/first-test/","title":"Create a test","text":"
To create a Chainsaw test all you need to do is to create one (or more) YAML file(s).
The recommended approach is to create one folder per test, with a chainsaw-test.yaml file containing one (or more) test definition(s). The test definition can reference other files in the same folder or anywhere else on the file system as needed.
Tip
While chainsaw supports other syntaxes, we strongly recommend the explicit approach.
"},{"location":"quick-start/first-test/#what-is-a-test","title":"What is a test?","text":"
To put it simply, a test can be represented as an ordered sequence of test steps.
In turn, a test step can be represented as an ordered sequence of operations.
When one of the operations fails the test is considered failed.
If all operations succeed the test is considered successful.
"},{"location":"quick-start/first-test/#lets-write-our-first-test","title":"Let's write our first test","text":"
For this quick start, we will create a (very simple) Test with one step and two operations:
Create a ConfigMap from a manifest
Verify the ConfigMap was created and contains the expected data
Follow the instructions below to create the folder and files defining our first test.
"},{"location":"quick-start/first-test/#create-a-test-folder","title":"Create a test folder","text":"
# create test folder\nmkdir chainsaw-quick-start\n\n# enter test folder\ncd chainsaw-quick-start\n
"},{"location":"quick-start/first-test/#create-a-configmap-manifest","title":"Create a ConfigMap manifest","text":"
"},{"location":"quick-start/first-test/#create-a-test-manifest","title":"Create a test manifest","text":"
By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.
# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n - try:\n # first operation: create the config map\n - apply:\n # file is relative to the test folder\n file: configmap.yaml\n # second operation: verify the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n file: configmap.yaml\nEOF\n
You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.
We also provide a GitHub action to easily install Chainsaw in your workflows.
"},{"location":"quick-start/install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"quick-start/install/#homebrew-tap","title":"Homebrew tap","text":"
add tap:
brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n
install chainsaw:
brew install kyverno/chainsaw/chainsaw\n
Don't forget to specify the tap name
Homebrew core already has a tool named chainsaw.
Be sure that you specify the tap name when installing to install the right tool.
Download the pre-compiled binaries for your system from the releases page and copy them to the desired location.
"},{"location":"quick-start/install/#install-using-go-install","title":"Install using go install","text":"
You can install with go install with:
go install github.com/kyverno/chainsaw@latest\n
"},{"location":"quick-start/install/#run-with-docker","title":"Run with Docker","text":"
Chainsaw is also available as a Docker image which you can pull and run:
docker pull ghcr.io/kyverno/chainsaw:<version>\n
Warning
Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.
Using nix-env permanently modifies a local profile of installed packages. This must be updated and maintained by the user in the same way as with a traditional package manager, foregoing many of the benefits that make Nix uniquely powerful. Using nix-shell or a NixOS configuration is recommended instead.
A nix-shell will temporarily modify your $PATH environment variable. This can be used to try a piece of software before deciding to permanently install it. Use the following command to install kyverno-chainsaw :
An output supports an optional match field. The match is used to conditionally create a binding.
In the case of applying a file, for example, the file may contain multiple resources. The match can be used to select the resource to use for creating the binding.
"},{"location":"quick-start/operation-outputs/#load-an-existing-resource","title":"Load an existing resource","text":"
The example below invokes a kubectl command to get a configmap from the cluster in json format.
The json output is then parsed and added to the $cm binding and the next operation performs an assertion on it by reading the binding instead of querying the cluster.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n content: kubectl get cm quick-start -n $NAMESPACE -o json\n outputs:\n # parse stdout json output and bind the result to `$cm`\n - name: cm\n value: (json_parse($stdout))\n - assert:\n resource:\n ($cm):\n metadata:\n (uid != null): true\n
"},{"location":"quick-start/operation-outputs/#match-a-resource","title":"Match a resource","text":"
The example below applies resources from a file.
When the resource being applied is a configmap, we bind the resource to an output to print its UID in the next operation.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n file: ./resources.yaml\n outputs:\n # match the configmap resource and bind it to `$cm`\n - match:\n apiVersion: v1\n kind: ConfigMap\n name: cm\n value: (@)\n - script:\n env:\n - name: UID\n value: ($cm.metadata.uid)\n content: echo $UID\n
Chainsaw simplifies dynamic resource configuration with native resource templating support.
Sometimes things we need to create resources or assertions are only known at runtime.
In the past, users have created all sorts of hacks using tools like envsubst for dynamic substitution of env-variables. Those workarounds usually lack flexibility and introduce new problems like hiding the real resources from Chainsaw, preventing it from cleaning resources properly.
Tip
Resource templating is heavily based on bindings and uses JMESPath language.
In the template below, we are using the $namespace binding at two different places, effectively injecting the ephemeral namespace name in the name and the data.foo fields:
After installing chainsaw and writing tests, the next natural step is to run Chainsaw to execute the tests.
"},{"location":"quick-start/run-tests/#create-a-local-cluster","title":"Create a local cluster","text":"
To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.
Not a cluster management tool
We consider this is not the responsibility of Chainsaw to manage clusters. There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.
The command below will create a local cluster using kind. Use the tool of your choice or directly jump to the next section if you already have a KUBECONFIG configured and pointing to a valid cluster.
Chainsaw expects a path to the test folder and will discover tests by analyzing files recursively. When no path is provided Chainsaw will use the current path by default (.).
The test above demonstrates the most basic usage of Chainsaw. In the next sections, we will look at the main features that make Chainsaw a very unique tool.
"},{"location":"quick-start/timeouts/","title":"Control your timeouts","text":"
Timeouts in Chainsaw are specified per type of operation. This is handy because the timeout varies greatly depending on the nature of an operation.
For example, applying a manifest in a cluster is expected to be reasonably fast, while validating a resource can be a long operation.
Timeouts can be configured globally and at the test, step or individual operation level.
All timeouts configured at a given level are automatically inherited in child levels. When looking up a timeout, the most specific one takes precedence over the others.
Info
To learn more about timeouts and how to configure global values, see the timeouts configuration page.
"},{"location":"quick-start/timeouts/#at-the-test-level","title":"At the test level","text":"
When a timeout is configured at the test level it will apply to all operations and steps in the test, unless overridden at a more specific level.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # timeouts configured at the test level will apply to all operations and steps\n # unless overriden at the step level and/or individual operation level\n timeouts:\n apply: 5s\n assert: 1m\n # ...\n steps:\n - try:\n - apply:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n - assert:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
"},{"location":"quick-start/timeouts/#at-the-step-level","title":"At the step level","text":"
When a timeout is configured at the step level it will apply to all operations in the step, unless overridden at a more specific level.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # timeouts configured at the step level will apply to all operations\n # in the step unless overriden at the individual operation level\n - timeouts:\n apply: 5s\n # ...\n try:\n - apply:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n # timeouts configured at the step level will apply to all operations\n # in the step unless overriden at the individual operation level\n - timeouts:\n assert: 1m\n # ...\n try:\n - assert:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
"},{"location":"quick-start/timeouts/#at-the-operation-level","title":"At the operation level","text":"
When a timeout is configured at the operation level, it takes precedence over all timeouts configured at upper levels.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n # timeout configured at the operation level takes precedence\n # over timeouts configured at upper levels\n timeout: 5s\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n - assert:\n # timeout configured at the operation level takes precedence\n # over timeouts configured at upper levels\n timeout: 1m\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
In the next section, we will see how Chainsaw manages cleanup.
"},{"location":"quick-start/try-catch/","title":"Use try, catch and finally","text":"
A test step is made of 3 main blocks used to determine the actions Chainsaw will perform when executing the step, depending on operations outcome.
The try block (required)
The catch block (optional)
The finally block (optional)
Operations defined in the try block are executed first, then:
If an operation fails to execute, Chainsaw won't execute the remaining operations and will execute all operations defined in the catch block instead (if any).
If all operations succeed, Chainsaw will NOT execute operations defined in the catch block (if any).
Regardless of the step outcome (success or failure), Chainsaw will execute all operations defined in the finally block (if any).
Note
Note that all operations coming from the catch or finally blocks are executed. If one operation fails, Chainsaw will mark the test as failed and continue executing with the next operation.
At the end of a test, Chainsaw automatically cleans up the resources created during the test (cleanup is done in the opposite order of creation).
All operations from the catch and finally blocks are executed before the cleanup process kicks in. This order allows analyzing the resources that potentially caused the step failure before they are deleted.
Operations in a finally block will always execute regardless of the success or failure of the test step.
This is particularly useful to perform manual cleanup.
In the example below we create a local cluster in a script operation. The cluster deletion script is added to the finally block, guaranteeing the cluster will be deleted regardless of the test outcome.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # create a local cluster\n - try:\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n - apply:\n # ...\n - assert:\n # ...\n # add cluster deletion script in the `finally` block\n # to guarantee the cluster will be deleted after the test\n finally:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n
"},{"location":"reference/builtins/#common","title":"Common","text":"Name Purpose Type $values Values provided when invoking chainsaw with --values flag any$namespace Name of the current test namespace string$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$config Kubernetes client config chainsaw is connected to (if not running with --no-cluster) object"},{"location":"reference/builtins/#in-tests","title":"In tests","text":"Name Purpose Type $test.id Current test id int$test.metadata Current test metadata metav1.ObjectMeta
Note
$test.id starts at 1 for the first test
"},{"location":"reference/builtins/#in-steps","title":"In steps","text":"Name Purpose Type $step.id Current step id int
Note
$step.id starts at 1 for the first step
"},{"location":"reference/builtins/#in-operations","title":"In operations","text":"Name Purpose Type $operation.id Current operation id int$operation.resourceId Current resource id int
Note
$operation.id starts at 1 for the first operation
$operation.resourceId maps to the resource id (starting at 1) in case the operation loads a file that contains multiple resources (the same operation is repeated once per resource)
"},{"location":"reference/builtins/#in-checks-and-outputs","title":"In checks and outputs","text":"Name Purpose Type @ The state of the resource (if any) at the end of the operation any$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string
Note
$stdout and $stderr are only available in script and command operations
ActionObject contains object selector options for an action.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ActionObjectSelectorActionObjectSelector No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ActionObjectSelector","title":"ActionObjectSelector","text":"
Appears in:
ActionObject
Events
PodLogs
ActionObjectSelector contains object selector options for an action.
Field Type Required Inline Description ObjectNameObjectName No description provided. selectorstring
Apply represents a set of configurations or resources that should be applied during testing.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Assert","title":"Assert","text":"
Appears in:
Operation
Assert represents a test condition that is expected to hold true during the testing process.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckRefActionCheckRef No description provided. ActionClustersActionClusters No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Binding","title":"Binding","text":"
Appears in:
ActionBindings
ActionEnv
Output
Scenario
TestSpec
TestStepSpec
Binding represents a key/value set as a binding in an executing test.
Command describes a command to run as a part of a test step.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckActionCheck No description provided. ActionClustersActionClusters No description provided. ActionEnvActionEnv No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. entrypointstring
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation.
reportFormatReportFormatType
ReportFormat determines test report format (JSON reportPathstring
ReportPath defines the path.
reportNamestring
ReportName defines the name of report to create. It defaults to \"chainsaw-report\".
namespacestring
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplatepolicy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullNamebool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegexstring
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegexstring
IncludeTestRegex is used to include tests based on a regular expression.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
testFilestring
TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
catch[]CatchFinally
Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels.
Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Delete","title":"Delete","text":"
Appears in:
CatchFinally
Operation
Delete is a reference to an object that should be deleted
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionExpectationsActionExpectations No description provided. ActionTimeoutActionTimeout No description provided. templatebool
Template determines whether resources should be considered for templating.
filestring
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided. showEventsbool
Show Events indicates whether to include related events.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckRefActionCheckRef No description provided. ActionClustersActionClusters No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Events","title":"Events","text":"
Appears in:
CatchFinally
Operation
Events defines how to collect events.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionFormatActionFormat No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Expectation","title":"Expectation","text":"
Appears in:
ActionExpectations
Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.
Field Type Required Inline Description matchpolicy/v1alpha1.Any
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectName","title":"ObjectName","text":"
Appears in:
ActionObjectSelector
ObjectReference
Proxy
ObjectName represents an object namespace and name.
Field Type Required Inline Description namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ObjectNameObjectName No description provided. labelsmap[string]string
Operation defines a single operation, only one action is permitted for a given operation.
Field Type Required Inline Description OperationBaseOperationBase
OperationBase defines common elements to all operations.
applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a deletion operation.
describeDescribe
Describe determines the resource describe collector to execute.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
eventsEvents
Events determines the events collector to execute.
getGet
Get determines the resource get collector to execute.
patchPatch
Patch represents a patch operation.
podLogsPodLogs
PodLogs determines the pod logs collector to execute.
proxyProxy
Proxy runs a proxy request.
scriptScript
Script defines a script to run.
sleepSleep
Sleep defines zzzz.
updateUpdate
Update represents an update operation.
waitWait
Wait determines the resource wait collector to execute.
OperationBase defines common elements to all operations.
Field Type Required Inline Description descriptionstring
Description contains a description of the operation.
continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-PodLogs","title":"PodLogs","text":"
Appears in:
CatchFinally
Operation
PodLogs defines how to collect pod logs.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided. containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. ObjectNameObjectName No description provided. ObjectTypeObjectType No description provided. portstring
TargetPort defines the target port to proxy the request.
pathstring
TargetPath defines the target path to proxy the request.
Script describes a script to run as a part of a test step.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckActionCheck No description provided. ActionClustersActionClusters No description provided. ActionEnvActionEnv No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. contentstring
Content defines a shell script (run with \"sh -c ...\").
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
clusterstring
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
skipDeletebool
SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
templatebool
Template determines whether resources should be considered for templating.
bindings[]Binding
Bindings defines additional binding key/values.
try[]Operation
Try defines what the step will try to execute.
catch[]CatchFinally
Catch defines what the step will execute when an error happens.
finally[]CatchFinally
Finally defines what the step will execute after the step is terminated.
cleanup[]CatchFinally
Cleanup defines what will be executed after the test is terminated.
Update represents a set of resources that should be updated. If a resource does not exist in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Wait","title":"Wait","text":"
Appears in:
CatchFinally
Operation
Wait specifies how to perform wait operations on resources.
Field Type Required Inline Description ActionTimeoutActionTimeout No description provided. ActionFormatActionFormat No description provided. ActionClustersActionClusters No description provided. ActionObjectActionObject No description provided. forWaitFor
ActionObject contains object selector options for an action.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ActionObjectSelectorActionObjectSelector No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ActionObjectSelector","title":"ActionObjectSelector","text":"
Appears in:
ActionObject
Events
PodLogs
ActionObjectSelector contains object selector options for an action.
Field Type Required Inline Description ObjectNameObjectName No description provided. selectorstring
Apply represents a set of configurations or resources that should be applied during testing.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Assert","title":"Assert","text":"
Appears in:
OperationAction
Assert represents a test condition that is expected to hold true during the testing process.
Field Type Required Inline Description ActionCheckRefActionCheckRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-CleanupOptions","title":"CleanupOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
CleanupOptions contains the configuration used for cleaning up resources.
Field Type Required Inline Description skipDeletebool
If set, do not delete the resources after running a test.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
Command describes a command to run as a part of a test step.
Field Type Required Inline Description ActionCheckActionCheck No description provided. ActionEnvActionEnv No description provided. ActionTimeoutActionTimeout No description provided. entrypointstring
Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Delete","title":"Delete","text":"
Appears in:
OperationAction
Delete is a reference to an object that should be deleted
Field Type Required Inline Description ActionExpectationsActionExpectations No description provided. ActionTimeoutActionTimeout No description provided. templatebool
Template determines whether resources should be considered for templating.
filestring
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Field Type Required Inline Description ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided. showEventsbool
Show Events indicates whether to include related events.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description ActionCheckRefActionCheckRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ErrorOptions","title":"ErrorOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
ErrorOptions contains the global error configuration.
Field Type Required Inline Description catch[]CatchFinally
Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels.
Field Type Required Inline Description ActionFormatActionFormat No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ExecutionOptions","title":"ExecutionOptions","text":"
Appears in:
ConfigurationSpec
ExecutionOptions determines how tests are run.
Field Type Required Inline Description failFastbool
FailFast determines whether the test should stop upon encountering the first failure.
parallelint
The maximum number of tests to run at once.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
Field Type Required Inline Description ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-NamespaceOptions","title":"NamespaceOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
NamespaceOptions contains the configuration used to allocate a namespace for each test.
Field Type Required Inline Description namestring
Name defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
templatepolicy/v1alpha1.Any
Template defines a template to create the test namespace.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ObjectNameObjectName No description provided. labelSelectormeta/v1.LabelSelector
Field Type Required Inline Description OperationActionOperationAction No description provided. OperationBindingsOperationBindings No description provided. OperationClustersOperationClusters No description provided. OperationOutputsOperationOutputs No description provided. descriptionstring
Description contains a description of the operation.
OperationAction defines an operation action, only one action should be specified per operation.
Field Type Required Inline Description applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a deletion operation.
describeDescribe
Describe determines the resource describe collector to execute.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
eventsEvents
Events determines the events collector to execute.
getGet
Get determines the resource get collector to execute.
patchPatch
Patch represents a patch operation.
podLogsPodLogs
PodLogs determines the pod logs collector to execute.
scriptScript
Script defines a script to run.
sleepSleep
Sleep defines zzzz.
updateUpdate
Update represents an update operation.
waitWait
Wait determines the resource wait collector to execute.
Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-PodLogs","title":"PodLogs","text":"
Appears in:
OperationAction
PodLogs defines how to collect pod logs.
Field Type Required Inline Description ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided. containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Script describes a script to run as a part of a test step.
Field Type Required Inline Description ActionCheckActionCheck No description provided. ActionEnvActionEnv No description provided. ActionTimeoutActionTimeout No description provided. contentstring
Content defines a shell script (run with \"sh -c ...\").
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
clusterstring
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
skipDeletebool
SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
templatebool
Template determines whether resources should be considered for templating.
bindings[]Binding
Bindings defines additional binding key/values.
try[]TryOperation
Try defines what the step will try to execute.
catch[]Operation
Catch defines what the step will execute when an error happens.
finally[]Operation
Finally defines what the step will execute after the step is terminated.
cleanup[]Operation
Cleanup defines what will be executed after the test is terminated.
Field Type Required Inline Description OperationOperation No description provided. continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Update represents a set of resources that should be updated. If a resource does not exist in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Wait","title":"Wait","text":"
Appears in:
OperationAction
Wait specifies how to perform wait operations on resources.
Field Type Required Inline Description ActionTimeoutActionTimeout No description provided. ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. forWaitFor
--clustered Defines if the resource is clustered (only applies when resource is loaded from a file)\n -f, --file string Path to the file to assert or '-' to read from stdin\n -h, --help help for assert\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use (default \"default\")\n --no-color Removes output colors\n -r, --resource string Path to the file containing the resource\n --timeout duration The assert timeout to use (default 30s)\n
--catalog string Path to the built test catalog file\n -h, --help help for docs\n --readme-file string Name of the built docs file (default \"README.md\")\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test\")\n
--description If set, adds description when applicable (default true)\n --force If set, existing test will be deleted if needed\n -h, --help help for test\n --save If set, created test will be saved\n
--autogenTag Determines if the generated docs should contain a timestamp (default true)\n -h, --help help for docs\n -o, --output string Output path (default \".\")\n --website Website version\n
--apply-timeout duration The apply timeout to use as default for configuration (default 5s)\n --assert-timeout duration The assert timeout to use as default for configuration (default 30s)\n --cleanup-delay duration Adds a delay between the time a test ends and the time cleanup starts\n --cleanup-timeout duration The cleanup timeout to use as default for configuration (default 30s)\n --cluster strings Register cluster (format <cluster name>=<kubeconfig path>:[context name])\n --config string Chainsaw configuration file\n --delete-timeout duration The delete timeout to use as default for configuration (default 15s)\n --deletion-propagation-policy string The deletion propagation policy (Foreground|Background|Orphan) (default \"Background\")\n --error-timeout duration The error timeout to use as default for configuration (default 30s)\n --exclude-test-regex string Regular expression to exclude tests\n --exec-timeout duration The exec timeout to use as default for configuration (default 5s)\n --fail-fast Stop the test upon encountering the first failure\n --force-termination-grace-period duration If specified, overrides termination grace periods in applicable resources\n --full-name Use full test case folder path instead of folder name\n -h, --help help for test\n --include-test-regex string Regular expression to include tests\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use for tests\n --no-cluster Runs without cluster\n --no-color Removes output colors\n --parallel int The maximum number of tests to run at once\n --pause-on-failure Pause test execution failure (implies no concurrency)\n --remarshal Remarshals tests yaml to apply anchors before parsing\n --repeat-count int Number of times to repeat each test (default 1)\n --report-format string Test report format (JSON|XML|nil)\n --report-name string The name of the report to create (default \"chainsaw-report\")\n --report-path string The path of the report to create\n --selector strings Selector (label query) to filter on\n --skip-delete If set, do not delete the resources after running the tests\n --template If set, resources will be considered for templating (default true)\n --test-dir strings Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test\")\n --values strings Values passed to the tests\n
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # `try` defines operations to execute in the step\n - try: [...]\n # `catch` defines operations to execute when the step fails\n catch: [...]\n # `finally` defines operations to execute at the end of the step\n finally: [...]\n # `cleanup` defines operations to execute at the end of the test\n cleanup: [...]\n
Operations defined in the try block are executed first, then:
If an operation fails to execute, Chainsaw won't execute the remaining operations and will execute all operations defined in the catch block instead (if any).
If all operations succeed, Chainsaw will NOT execute operations defined in the catch block (if any).
Regardless of the step outcome (success or failure), Chainsaw will execute all operations defined in the finally block (if any).
Tip
Note that all operations coming from the catch or finally blocks are executed. If one operation fails, Chainsaw will mark the test as failed and continue executing with the next operations.
sequenceDiagram\n autonumber\n\n participant S as Step N\n\n box Try block\n participant T1 as Op 1\n participant T2 as Op N\n end\n box Catch block\n end\n box Finally block\n participant F1 as Op 1\n participant F2 as Op N\n end\n participant S1 as Step N+1\n\n S -->> T1 : try\n T1 ->> T2 : success\n T2 -->> S : done\n S -->> F1 : finally\n F1 ->> F2 : done\n F2 -->> S : done\n S -->> S1 : next step
Step starts by executing operations in the try block
Operations in the try block execute sequentially
All operations in the try block terminate
Step starts executing operations in the finally block
Operations in the finally block execute sequentially
sequenceDiagram\n autonumber\n\n participant S as Step N\n\n box Try block\n participant T1 as Op 1\n participant T2 as Op N\n end\n box Catch block\n participant C1 as Op 1\n participant C2 as Op N\n end\n box Finally block\n participant F1 as Op 1\n participant F2 as Op N\n end\n\n S -->> T1 : try\n T1 ->> T2 : success\n T2 -->> S : error\n S -->> C1 : catch\n C1 ->> C2 : done\n C2 -->> S : done\n S -->> F1 : finally\n F1 ->> F2 : done\n F2 -->> S : done
Step starts by executing operations in the try block
Operations in the try block execute sequentially until an error happens
Operations in the try block stop when an error occurs
Step starts executing operations in the catch block
Operations in the catch block execute sequentially
All operations in the catch block terminate
Step starts executing operations in the finally block
Operations in the finally block execute sequentially
Under certain circumstances, it can be useful to configure catch blocks at a higher level than the step grain. At the test or configuration level.
This allows for declaring common catch statements we want to execute when an error occurs. Those catch blocks are combined to produce the final catch block in the following order:
catch statements from the configuration level are executed first (if any)
catch statements from the test level are executed next (if any)
catch statements from the step level are executed last (if any)
A cleanup statement is similar to a finally statement but will execute after the test finishes executing, while finally executes after the step finishes executing.
Tip
All operations of a cleanup statement will be executed regardless of the success or failure of each of them.
A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.
Tip
All operations of a finally statement will be executed regardless of the success or failure of each of them.
While this syntax is simple, it suffers lots of limitations. It doesn't support deletion operations, commands, scripts, and all Chainsaw helpers.
It is also impossible to specify additional configuration per test, step or individual operation (timeouts, additional verifications, etc...), making this approach highly limited.
It also relies a lot on file naming conventions which can be error prone.
Finally, this approach doesn't encourage reusing files across tests and leads to duplication, making maintenance harder.
The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.
The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.
This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the config map does not contain the lorem: ipsum data.
For such a simple test, the conventional approach works reasonably well but will quickly become limited when the test scenarios get more complex.
Look at the explicit approach for a lot more flexible solution.
The explicit is a bit more verbose than the conventional one but offers far more flexibility and features:
It does not rely on file naming conventions for operations ordering
It encourages file reuse across tests, reducing duplication and maintenance
It offers the flexibility to provide additional configurations like timeouts, complex logic, etc...
It supports all operations without restrictions
"},{"location":"test/explicit/#the-test-resource","title":"The Test resource","text":"
A Test resource, like any other Kubernetes resource, has an apiVersion, kind and metadata section.
It also comes with a spec section used to declaratively represent the test logic, steps and operations, as well as other configuration elements belonging to the test being defined.
Reference documentation
The full structure of the Test resource is documented here.
The Test below illustrates a simple test. Chainsaw will load the Test and steps defined in its spec section.
It's worth noting that:
The test defines its own timeouts
It also states that this test should not be executed in parallel with other tests
It has multiple steps, most of them reference files that can be used in other tests if needed
It uses an arbitrary shell script
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # state that this test should not be executed in parallel with other tests\n concurrent: false\n # timeouts for this specific test\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # step 1\n # apply a configmap to the cluster\n # the path to the configmap is relative to the folder\n # containing the test, hence allow reusing manifests\n # across multiple tests\n - try:\n - apply:\n file: ../resources/configmap.yaml\n # step 2\n # execute assert statements against existing resources\n # in the cluster\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # step 3\n # execute error statements against existing resources\n # in the cluster\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n # step 4\n # execute an arbitrary shell script\n - try:\n - script:\n content: echo \"goodbye\"\n
At the end of the test, Chainsaw cleans up resources it created during the test, in the opposite order of creation.
By default, when a step fails, Chainsaw stops the execution and the remaining steps are not executed. The cleanup process starts at the moment the test stops executing.
Tip
Note that when a failure happens during cleanup, the test is marked as failed and Chainsaw continues executing cleanup for the remaining steps.
sequenceDiagram\n autonumber\n participant T as Test\n participant S1 as Step 1\n participant S2 as Step 2\n participant S3 as Step 3\n\n T ->> S1: execute\n S1 ->> S2: execute (fail)\n\n Note left of S3: Step 3 is NOT executed\n\n S2 -->> S1: cleanup\n S1 -->> T: cleanup
Test starts by executing Step 1
Step 1 terminates -> Step 2 starts executing
Step 2 fails -> Cleanup for Step 2 starts
Cleanup for Step 2 terminates -> Cleanup for Step 1 is executed
Steps are what tests will execute when they are run, see Test step spec dedicated section.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"cicd/gh-action/","title":"GitHub action","text":"
A GitHub action is available to easily install Chainsaw in your workflows.
The GitHub action is available at kyverno/action-install-chainsaw or in the marketplace.
If you want to install Chainsaw from its main version by using go install under the hood, you can set release as main. Once you did that, Chainsaw will be installed via go install which means that please ensure that go is installed.
Input Description releasechainsaw version to use instead of the default. install-dir directory to place the chainsaw binary into instead of the default ($HOME/.chainsaw). use-sudo set to true if install-dir location requires sudo privs. Defaults to false. verify set to true to enable cosign verification of the downloaded archive."},{"location":"community/","title":"Community","text":"
Chainsaw has a growing community and we would definitely love to see you join and contribute.
Everyone is welcome to make suggestions, report bugs, open feature requests, contribute code or docs, participate in discussions, write blogs or anything that can benefit the project.
Chainsaw is built and maintained under the Kyverno umbrella but decisions are Community driven Everyone's voice matters
To attend our community meetings, join the Chainsaw group. You will then be sent a meeting invite and will have access to the agenda and meeting notes. Any member may suggest topics for discussion.
This is a public, weekly for Kyverno-Chainsaw maintainers to make announcements and provide project updates, and request input and feedback. This forum allows community members to raise agenda items of any sort, including but not limited to any PRs or issues on which they are working.
If you are using Chainsaw and want to share it publicly we always appreciate a bit of support. Pull requests to the ADOPTERS LIST will put a smile on our faces
Chainsaw, developed by Kyverno, is an advanced end-to-end testing tool for Kubernetes. Our community plays a crucial role in shaping the project by reporting bugs, suggesting features, and improving documentation.
We aim to make our issue tracker, discussion board, and documentation well-structured and easy to navigate. By following our guidelines, you can help us address your requests efficiently.
"},{"location":"community/contribute/#how-you-can-contribute","title":"How you can contribute","text":"
We appreciate your efforts in reporting bugs, requesting features, and engaging in discussions. Here's how you can contribute:
"},{"location":"community/contribute/#creating-an-issue","title":"Creating an issue","text":"
Something is not working?
Report a bug in Chainsaw by creating an issue with a reproduction
Report a bug
Missing information in our docs?
Report missing information or potential inconsistencies in our documentation
Report a docs issue
Want to submit an idea?
Propose a change, feature request, or suggest an improvement
Request a change
Have a question or need help?
Ask a question on our discussion board and get in touch with our community
Before interacting within the project, please consider the following questions to ensure you're using the correct issue template and providing all necessary information.
Issues, discussions, and comments are forever
Please note that everything you write is permanent and will remain for everyone to read \u2013 forever. Therefore, please always be nice and constructive, follow our contribution guidelines, and comply with our Code of Conduct.
"},{"location":"community/contribute/#before-creating-an-issue","title":"Before creating an issue","text":"
Are you using the appropriate issue template, or is there another issue template that better fits the context of your request?
Have you checked if a similar bug report or change request has already been created, or have you stumbled upon something that might be related?
Did you fill out every field as requested and provide all additional information needed to comprehend your request?
"},{"location":"community/contribute/#before-asking-a-question","title":"Before asking a question","text":"
Is the topic a question for our discussion board, or is it a bug report or change request that should be raised on our [issue tracker]?
Is there an open discussion on the topic of your request? If the answer is yes, does your question match the direction of the discussion, or should you open a new discussion?
Did you provide our community with all the necessary information to understand your question and help you quickly, or can you make it easier to help you?
Is your comment relevant to the topic of the current page, post, issue, or discussion, or is it better to create a new issue or discussion?
Does your comment add value to the conversation? Is it constructive and respectful to our community and maintainers? Could you just use a reaction instead?
"},{"location":"community/contribute/#rights-and-responsibilities","title":"Rights and responsibilities","text":"
As maintainers, we are entrusted with the responsibility to moderate communication within our community, including the authority to close, remove, reject, or edit issues, discussions, comments, commits, and to block users who do not align with our contribution guidelines and our Code of Conduct. This role requires us to be actively involved in maintaining the integrity and positive atmosphere of our community. Upholding these standards decisively ensures a respectful and inclusive environment for all members.
"},{"location":"community/contribute/#code-of-conduct","title":"Code of Conduct","text":"
Our Code of Conduct outlines the expectation for all community members to treat one another with respect, employing inclusive and welcoming language. Our commitment is to foster a positive and supportive environment, free of inappropriate, offensive, or harmful behavior.
We take any violations seriously and will take appropriate action in response to uphold these values.1
"},{"location":"community/contribute/#incomplete-issues-and-duplicates","title":"Incomplete issues and duplicates","text":"
We have invested significant time and effort in the setup of our contribution process, ensuring that we assess the essential requirements for reviewing and responding to issues effectively. Each field in our issue templates is thoughtfully designed to help us fully understand your concerns and the nature of your matter. We encourage all members to utilize the search function before submitting new issues or starting discussions to help avoid duplicates. Your cooperation is crucial in keeping our community's discussions constructive and organized.
Mandatory completion of issue templates: We need all of the information required in our issue templates because it ensures that every user and maintainer, regardless of their experience, can understand the content and severity of your bug report or change request.
Closing incomplete issues: We reserve the right to close issues lacking essential information, such as but not limited to [minimal reproductions] or those not adhering to the quality standards and requirements specified in our issue templates. Such issues can be reopened once the missing information has been provided.
Handling duplicates: To maintain organized and efficient communication within our [issue tracker] and discussion board, we reserve the right to close any duplicated issues or lock duplicated discussions. Opening multiple channels to ask the same question or report the same issue across different forums hinders our ability to manage and address community concerns effectively. This approach is vital for efficient time management, as duplicated questions can consume the time of multiple team members simultaneously. Ensuring that each issue or discussion is unique and progresses with new information helps us to maintain focus and support our community.
We further reserve the right to immediately close discussions or issues that are reopened without providing new information or simply because users have not yet received a response to their issue/question, as the issue is marked as incomplete.
Limitations of automated tools: While we believe in the value and efficiency that automated tools bring to identifying potential issues (such as those identified by Lighthouse, Accessibility tools, and others), simply submitting an issue generated by these tools does not constitute a complete bug report. These tools sometimes produce verbose outputs and may include false positives, which necessitate a critical evaluation. You are of course welcome to attach generated reports to your issue. However, this does not substitute the requirement for a minimal reproduction or a thorough discussion of the findings. We reserve the right to mark these issues as incomplete and close them. This practice ensures that we are addressing genuine concerns with precision and clarity, rather than navigating through extensive automated outputs.
Warning and blocking policy: Given the increasing popularity of our project and our commitment to a healthy community, we've defined clear guidelines on how we proceed with violations:
1.1. First warning: Users displaying repeated inappropriate, offensive, or harmful behavior will receive a first warning. This warning serves as a formal notice that their behavior is not in alignment with our community standards and Code of Conduct. The first warning is permanent.
1.2. Second warning and opportunity for resolution: If the behavior persists, a second warning will be issued. Upon receiving the second warning, the user will be given a 5-day period for reflection, during which they are encouraged to publicly explain or apologize for their actions. This period is designed to offer an opportunity for openly clearing out any misunderstanding.
1.3. Blocking: Should there be no response or improvement in behavior following the second warning, we reserve the right to block the user from the community and repository. Blocking is considered a last resort, used only when absolutely necessary to protect the community's integrity and positive atmosphere.
Blocking has been an exceptionally rare necessity in our overwhelmingly positive community, highlighting our preference for constructive dialogue and mutual respect. It aims to protect our community members and team.\u00a0\u21a9
You can contribute to Chainsaw by making a pull request that will be reviewed by maintainers and integrated into the main repository when the changes made are approved. You can contribute bug fixes, documentation changes, or new functionalities.
Considering a pull request
Before deciding to spend effort on making changes and creating a pull request, please discuss what you intend to do. If you are responding to what you think might be a bug, please issue a bug report first. If you intend to work on documentation, create a documentation issue. If you want to work on a new feature, please create a change request.
Keep in mind the guidance given and let people advise you. It might be that there are easier solutions to the problem you perceive and want to address. It might be that what you want to achieve can already be done by configuration or [customization].
"},{"location":"community/making-a-pull-request/#learning-about-pull-requests","title":"Learning about pull requests","text":"
Pull requests are a concept layered on top of Git by services that provide Git hosting. Before you consider making a pull request, you should familiarize yourself with the documentation on GitHub, the service we are using. The following articles are of particular importance:
Forking a repository
Creating a pull request from a fork
Creating a pull request
Note that they provide tailored documentation for different operating systems and different ways of interacting with GitHub. We do our best in the documentation here to describe the process as it applies to Chainsaw but cannot cover all possible combinations of tools and ways of doing things. It is also important that you understand the concept of a pull-request in general before continuing.
In the following, we describe the general process for making pull requests. The aim here is to provide the 30k ft overview before describing details later on.
"},{"location":"community/making-a-pull-request/#preparing-changes-and-draft-pr","title":"Preparing changes and draft PR","text":"
The diagram below describes what typically happens to repositories in the process or preparing a pull request. We will be discussing the review-revise process below. It is important that you understand the overall process first before you worry about specific commands. This is why we cover this first before providing instructions below.
sequenceDiagram\n autonumber\n\n participant chainsaw\n participant PR\n participant fork\n participant local\n\n chainsaw ->> fork: fork on GitHub\n fork ->> local: clone to local\n local ->> local: branch\n loop prepare\n loop push\n loop edit\n local ->> local: commit\n end\n local ->> fork: push\n end\n chainsaw ->> fork: merge in any changes\n fork ->>+ PR: create draft PR\n PR ->> PR: review your changes\n end
Fork the Repository: Fork the Chainsaw repository on GitHub to create your own copy.
Clone to Local: Clone your fork to your local machine.
Create a Branch: Create a topic branch for your changes.
Set Up Development Environment: Follow the instructions to set up a development environment.
Iterate and Commit: Make incremental changes and commit them with meaningful messages.
Push Regularly: Push your commits to your fork regularly.
Merge Changes from Upstream: Regularly merge changes from the original Chainsaw repository to avoid conflicts.
Create a Draft Pull Request: Once satisfied with your changes, create a draft pull request for early feedback.
Review and Revise: Review your work critically, address feedback, and refine your changes.
Once you are happy with your changes, you can move to the next step, finalizing your pull request and asking for a more formal and detailed review. The diagram below shows the process:
Chainsaw, developed by Kyverno, is an actively maintained project that we constantly strive to improve. With a project of this size and complexity, bugs may occur. If you think you have discovered a bug, you can help us by submitting an issue in our public issue tracker, following this guide.
"},{"location":"community/reporting-a-bug/#before-creating-an-issue","title":"Before Creating an Issue","text":"
With numerous users, issues are created regularly. The maintainers of this project strive to address bugs promptly. By following this guide, you will know exactly what information we need to help you quickly.
Please do the following before creating an issue:
"},{"location":"community/reporting-a-bug/#upgrade-to-latest-version","title":"Upgrade to Latest Version","text":"
Chances are that the bug you discovered was already fixed in a subsequent version. Before reporting an issue, ensure that you're running the [latest version] of Chainsaw. Consult our [upgrade guide] to learn how to upgrade to the latest version.
Bug fixes are not backported
Please understand that only bugs that occur in the latest version of Chainsaw will be addressed. Also, to reduce duplicate efforts, fixes cannot be backported to earlier versions.
If you're using customizations like additional configurations, remove them before reporting a bug. We can't offer official support for bugs that might hide in your overrides, so make sure to omit custom settings from your configuration files.
Don't be shy to ask on our discussion board for help if you run into problems.
"},{"location":"community/reporting-a-bug/#search-for-solutions","title":"Search for Solutions","text":"
At this stage, we know that the problem persists in the latest version and is not caused by any of your customizations. However, the problem might result from a small typo or a syntactical error in a configuration file.
Before creating a bug report, save time for us and yourself by doing some research:
Search our documentation for relevant sections related to your problem. Ensure everything is configured correctly.
[Search our issue tracker] as another user might have already reported the same problem.
[Search our discussion board] to see if other users are facing similar issues and find possible solutions.
Keep track of all search terms and relevant links; you'll need them in the bug report.
If you still haven't found a solution to your problem, create an issue. It's now likely that you've encountered something new. Read the following section to learn how to create a complete and helpful bug report.
We have created a new issue template to make the bug reporting process as simple as possible and more efficient for our community and us. It consists of the following parts:
A good title is short and descriptive. It should be a one-sentence executive summary of the issue, so the impact and severity of the bug can be inferred from the title.
Example Clear Chainsaw apply command fails with specific CRD Wordy The apply command in Chainsaw fails when used with a certain Custom Resource Definition Unclear Command does not work Useless Help"},{"location":"community/reporting-a-bug/#context","title":"Context optional","text":"
Before describing the bug, you can provide additional context to help us understand what you were trying to achieve. Explain the circumstances under which you're using Chainsaw, and what you think might be relevant. Don't describe the bug here.
Provide a clear, focused, specific, and concise summary of the bug you encountered. Explain why you think this is a bug that should be reported to Chainsaw, and not to one of its dependencies. Follow these principles:
Explain the what, not the how \u2013 don't explain how to reproduce the bug here, we're getting there. Focus on articulating the problem and its impact.
Keep it short and concise \u2013 if the bug can be precisely explained in one or two sentences, perfect. Don't inflate it.
One bug at a time \u2013 if you encounter several unrelated bugs, create separate issues for them.
Share links to relevant sections of our documentation and any related issues or discussions. This helps us improve our documentation and understand the problem better.
A minimal reproduction is essential for a well-written bug report, as it allows us to recreate the conditions necessary to inspect the bug. Follow the guide to create a reproduction:
After creating the reproduction, you should have a .zip file, ideally not larger than 1 MB. Drag and drop the .zip file into the issue field, which will automatically upload it to GitHub.
Don't share links to repositories
While linking to a repository is a common practice, we currently don't support this. The reproduction, created using the built-in info plugin, contains all necessary environment information.
"},{"location":"community/reporting-a-bug/#steps-to-reproduce","title":"Steps to Reproduce","text":"
List specific steps to follow when running your reproduction to observe the bug. Keep the steps concise and ensure nothing is left out. Use simple language and focus on continuity.
If the bug only occurs in specific browsers, let us know which ones are affected. This field is optional, as it is only relevant for bugs that do not involve a crash when previewing or building your site.
Incognito Mode
Verify that the bug is not caused by a browser extension by switching to incognito mode. If the bug disappears, it is likely caused by an extension.
The Chainsaw documentation includes extensive information on features, configurations, customizations, and more. If you have found an inconsistency or see room for improvement, please follow this guide to submit an issue on our issue tracker.
Reporting a documentation issue is usually less involved than reporting a bug, as we don't need a [reproduction]. Please thoroughly read this guide before creating a new documentation issue, and provide the following information as part of the issue:
A good title should be a short, one-sentence description of the issue, containing all relevant information and keywords to simplify the search in our issue tracker.
Example Clear Clarify resource templating setup in Chainsaw Unclear Missing information in the docs Useless Help"},{"location":"community/reporting-a-docs-issue/#description","title":"Description","text":"
Provide a clear and concise summary of the inconsistency or issue you encountered in the documentation or the documentation section that needs improvement. Explain why you think the documentation should be adjusted and describe the severity of the issue:
Keep it short and concise \u2013 if the inconsistency or issue can be precisely explained in one or two sentences, perfect. Maintainers and future users will be grateful for having to read less.
One issue at a time \u2013 if you encounter several unrelated inconsistencies, please create separate issues for them.
Why we need this: describing the problem clearly and concisely is a prerequisite for improving our documentation \u2013 we need to understand what's wrong so we can fix it.
After you describe the documentation section that needs to be adjusted, share the link to this specific documentation section and other possibly related sections. Use anchor links (permanent links) where possible, as it simplifies discovery.
Why we need this: providing the links to the documentation helps us understand which sections of our documentation need to be adjusted, extended, or overhauled.
Now that you have provided us with the description and links to the documentation sections, you can help us, maintainers, and the community by proposing an improvement. You can sketch out rough ideas or write a concrete proposal. This field is optional but very helpful.
Why we need this: an improvement proposal can be beneficial for other users who encounter the same issue, as they offer solutions before we maintainers can update the documentation.
Thanks for following the guide and providing valuable feedback for our documentation \u2013 you are almost done. The checklist ensures that you have read this guide and have worked to your best knowledge to provide us with every piece of information we need to improve it.
Chainsaw by Kyverno is a powerful tool for end-to-end testing. Serving a wide range of use cases, we value every idea or contribution from our community. Please follow this guide before submitting your change request in our public issue tracker. This helps us better understand the proposed change and how it will benefit our community.
"},{"location":"community/requesting-a-change/#before-creating-an-issue","title":"Before Creating an Issue","text":"
Before you invest time in submitting a change request, answer these questions to determine if your idea is a good fit for Chainsaw and matches the project's philosophy and tone.
"},{"location":"community/requesting-a-change/#its-not-a-bug-its-a-feature","title":"It's Not a Bug, It's a Feature","text":"
Change requests suggest minor adjustments, new features, or influence the project's direction. They are not intended for reporting bugs. Refer to our bug reporting guide for that.
"},{"location":"community/requesting-a-change/#look-for-sources-of-inspiration","title":"Look for Sources of Inspiration","text":"
If your idea is implemented in another tool or framework, collect information on its implementation. This helps us evaluate its fit more quickly.
"},{"location":"community/requesting-a-change/#connect-with-our-community","title":"Connect with Our Community","text":"
Our discussion board is the best place to connect with our community. Seeking input from other users helps implement features that benefit a larger number of users.
A good title is short and descriptive, summarizing the idea so the potential impact and benefit can be inferred.
Example Clear Support for resource templating in Chainsaw Wordy Add support for templating resources in Chainsaw for easier testing Unclear Improve templating Useless Help"},{"location":"community/requesting-a-change/#context","title":"Context optional","text":"
Provide additional context to help us understand what you are trying to achieve. Explain the circumstances and relevant settings without describing the change request itself.
Provide a detailed and clear description of your idea. Explain why your idea is relevant to Chainsaw and should be implemented here, not in one of its dependencies.
Explain the what, not the why \u2013 focus on describing the change request precisely.
Keep it short and concise \u2013 be brief and to the point.
One idea at a time \u2013 if you have multiple ideas, open separate change requests for each.
Explain how your change request would work from an author's and user's perspective. What is the expected impact, and why does it benefit other users? Would it potentially break existing functionality?
If you have any visuals, such as sketches, screenshots, mockups, or external assets, present them in this section. If you have seen this change used in other tools, showcase and describe its implementation.
Thanks for following the guide and creating a high-quality change request. The checklist ensures that you have read this guide and provided all necessary information for us to review your idea.
Your change request got rejected? We're sorry for that. We understand it can be frustrating, but we always need to consider the needs of our entire community. If you're unsure why your change request was rejected, please ask for clarification.
We consider the following principles when evaluating change requests:
Alignment with the project's vision and tone
Compatibility with existing features and plugins
Compatibility with all screen sizes and browsers
Effort of implementation and maintenance
Usefulness to the majority of users
Simplicity and ease of use
Accessibility
If your idea was rejected, you can always implement it via [customization]. If you're unsure how or want to know if someone has already done it, get in touch with our community on the discussion board.
In this example, Chainsaw will load a configuration file but the timeout configuration and other settings will be overridden by the values set in the flags, regardless of the value in the loaded configuration file.
Cleanup options contain the configuration used by Chainsaw for cleaning up resources.
"},{"location":"configuration/options/cleanup/#supported-elements","title":"Supported elements","text":"Element Default Description skipDeletefalse If set, do not delete the resources after running a test. delayBeforeCleanup DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts."},{"location":"configuration/options/cleanup/#delay-before-cleanup","title":"Delay before cleanup","text":"
At the end of each test, Chainsaw will delete the resources it created during the test.
When testing operators, it can be useful to wait a little bit before starting the cleanup process to make sure the operator/controller has the necessary time to update its internal state.
Every cluster is registered by name and supports the following elements:
Element Default Description kubeconfigstring Kubeconfig is the path to the referenced file. contextstring Context is the name of the context to use."},{"location":"configuration/options/clusters/#configuration","title":"Configuration","text":""},{"location":"configuration/options/clusters/#with-file","title":"With file","text":"
apiVersion: chainsaw.kyverno.io/v1alpha2\nkind: Configuration\nmetadata:\n name: custom-config\nspec:\n clusters:\n # this cluster will use the default (current) context\n # configured in the kubeconfig file\n cluster-1:\n kubeconfig: /path/to/kubeconfig-1\n # this cluster will use the context named `context-2`\n # in the kubeconfig file\n cluster-2:\n kubeconfig: /path/to/kubeconfig-2\n context: context-2\n
Deletion options determine the configuration used by Chainsaw for deleting resources.
"},{"location":"configuration/options/deletion/#supported-elements","title":"Supported elements","text":"Element Default Description propagationBackground Propagation decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation."},{"location":"configuration/options/deletion/#propagation","title":"Propagation","text":"
This element will affect Kubernetes cascading deletion. Supported values are Orphan, Background and Foreground.
Tip
Setting Orphan is probably never a good idea because it would leak resources in the test cluster. Chainsaw uses Background as its default value which is a reasonable choice.
Note that Foreground can be useful to fail when the dependent resources fail to delete.
Discovery options contain the discovery configuration used by Chainsaw when discovering tests in specified folders.
"},{"location":"configuration/options/discovery/#supported-elements","title":"Supported elements","text":"Element Default Description testFilechainsaw-test TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed. fullNamefalse FullName makes use of the full test case folder path instead of the folder name. includeTestRegex IncludeTestRegex is used to include tests based on a regular expression. excludeTestRegex ExcludeTestRegex is used to exclude tests based on a regular expression."},{"location":"configuration/options/discovery/#configuration","title":"Configuration","text":""},{"location":"configuration/options/discovery/#with-file","title":"With file","text":"
Error options contain the global error configuration used by Chainsaw.
"},{"location":"configuration/options/error/#supported-elements","title":"Supported elements","text":"Field Default Description catch Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels."},{"location":"configuration/options/error/#configuration","title":"Configuration","text":""},{"location":"configuration/options/error/#with-file","title":"With file","text":"
Execution options determine how tests are run by Chainsaw.
"},{"location":"configuration/options/execution/#supported-elements","title":"Supported elements","text":"Element Default Description failFastfalse FailFast determines whether the test should stop upon encountering the first failure. parallelauto The maximum number of tests to run at once. repeatCount1 RepeatCount indicates how many times the tests should be executed. forceTerminationGracePeriod ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments."},{"location":"configuration/options/execution/#termination-grace-period","title":"Termination grace period","text":"
Some Kubernetes resources can take time before being terminated. For example, deleting a pod can take time if the underlying container doesn't quit quickly enough.
Chainsaw can override the grace period for the following resource kinds:
Namespace options contain the configuration used by Chainsaw to allocate a namespace for each test.
"},{"location":"configuration/options/namespace/#supported-elements","title":"Supported elements","text":"Element Default Description name Name defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec. template Template defines a template to create the test namespace."},{"location":"configuration/options/namespace/#configuration","title":"Configuration","text":""},{"location":"configuration/options/namespace/#with-file","title":"With file","text":"
Chainsaw can be configured to pause and wait for user input when a failure happens. This is useful when Chainsaw is run locally to allow debugging and troubleshooting failures.
Reporting options contain the configuration used by Chainsaw for reporting.
"},{"location":"configuration/options/report/#supported-elements","title":"Supported elements","text":"Element Default Description formatJSON ReportFormat determines test report format (JSON path ReportPath defines the path. namechainsaw-report ReportName defines the name of report to create. It defaults to \"chainsaw-report\"."},{"location":"configuration/options/report/#configuration","title":"Configuration","text":""},{"location":"configuration/options/report/#with-file","title":"With file","text":"
Templating options contain the templating configuration.
"},{"location":"configuration/options/templating/#supported-elements","title":"Supported elements","text":"Element Default Description enabledtrue Enabled determines whether resources should be considered for templating.
Tip
Templating was disabled by default in v0.1.* but is now enabled by default since v0.2.1.
Timeouts in Chainsaw are specified per type of operation. This is required because the timeout varies greatly depending on the nature of an operation.
For example, applying a manifest in a cluster is expected to execute reasonably fast, while validating a resource can be a much longer operation.
"},{"location":"configuration/options/timeouts/#supported-timeouts","title":"Supported timeouts","text":"Element Default Description apply 5s Used when Chainsaw applies manifests in a cluster assert 30s Used when Chainsaw validates resources in a cluster cleanup 30s Used when Chainsaw removes resources created for a test delete 15s Used when Chainsaw deletes resources from a cluster error 30s Used when Chainsaw validates resources in a cluster exec 5s Used when Chainsaw executes arbitrary commands or scripts"},{"location":"configuration/options/timeouts/#configuration","title":"Configuration","text":""},{"location":"configuration/options/timeouts/#with-file","title":"With file","text":"
The number of concurrent tests can be configured globally using a configuration file or with the --parallel flag.
Alternatively, the concurrent nature of a test can specified at the test level:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # concurrency can be specified per test (`true` or `false`)\n # default value is `true`\n concurrent: false\n # ...\n
All non-concurrent tests are executed first, followed by the concurrent tests running in parallel.
"},{"location":"examples/crds/","title":"Work with CRDs","text":"
New CRDs are not immediately available for use in the Kubernetes API until the Kubernetes API has acknowledged them.
If a CRD is being defined inside of a test step, be sure to wait for it to appear.
The test below applies a CRD and waits for it to become available:
The test below fetches the Kubernetes cluster version using the x_k8s_server_version function. It then uses the minor version retrieved to adapt an assertion based on the value in the $minorversion binding.
Tip
You can implement a ternary operator in JMESPath using an expression like this:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n bindings:\n - name: version\n value: (x_k8s_server_version($config))\n - name: minorversion\n value: (to_number($version.minor))\n steps:\n - try:\n - apply:\n resource:\n apiVersion: v1\n kind: Pod\n metadata:\n name: pod01\n spec:\n containers:\n - name: busybox\n image: busybox:1.35\n # ...\n - assert:\n resource:\n apiVersion: v1\n kind: Pod\n metadata:\n annotations:\n # If the minor version of the Kubernetes cluster against\n # which this is tested is less than 29, the annotation is\n # expected to have the group 'system:masters' in it.\n # Otherwise, due to a change in kubeadm, the group should\n # be 'kubeadm:cluster-admins'.\n kyverno.io/created-by: (($minorversion < `29` && '{\"groups\":[\"system:masters\",\"system:authenticated\"],\"username\":\"kubernetes-admin\"}') || '{\"groups\":[\"kubeadm:cluster-admins\",\"system:authenticated\"],\"username\":\"kubernetes-admin\"}')\n name: pod01\n
"},{"location":"examples/label-selectors/","title":"Work with label selectors","text":"
Chainsaw can filter the tests to run using label selectors.
You can pass label selectors using the --selector flag when invoking the chainsaw test command.
Chainsaw supports registering and using multiple clusters in tests.
We can also register clusters dynamically and combine this with cluster selection to achieve scenarios where clusters are dynamically allocated in a test step, used in the following steps, and cleaned up at the end.
The following test demonstrates such a scenario by creating a local kind cluster in the first, using it in the second step, and configuring a cleanup script to delete the cluster when the test terminates:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n # create a local cluster\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n # register `cleanup` operations to delete the cluster\n # at the end of the test\n cleanup:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n # register the `dynamic` cluster in this step\n - clusters:\n dynamic:\n kubeconfig: ./dynamic\n # and use the `dynamic` cluster for all operations in the step\n cluster: dynamic\n try:\n - apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n namespace: default\n data:\n foo: bar\n - assert:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n namespace: default\n data:\n foo: bar\n
Running the test above will produce the following output:
| 10:44:53 | example | @setup | CREATE | OK | v1/Namespace @ chainsaw-useful-seahorse\n | 10:44:53 | example | step-1 | TRY | RUN |\n | 10:44:53 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c kind create cluster --name dynamic --kubeconfig ./dynamic\n | 10:45:10 | example | step-1 | SCRIPT | LOG |\n === STDERR\n Creating cluster \"dynamic\" ...\n \u2022 Ensuring node image (kindest/node:v1.27.3) \ud83d\uddbc ...\n \u2713 Ensuring node image (kindest/node:v1.27.3) \ud83d\uddbc\n \u2022 Preparing nodes \ud83d\udce6 ...\n \u2713 Preparing nodes \ud83d\udce6 \n \u2022 Writing configuration \ud83d\udcdc ...\n \u2713 Writing configuration \ud83d\udcdc\n \u2022 Starting control-plane \ud83d\udd79\ufe0f ...\n \u2713 Starting control-plane \ud83d\udd79\ufe0f\n \u2022 Installing CNI \ud83d\udd0c ...\n \u2713 Installing CNI \ud83d\udd0c\n \u2022 Installing StorageClass \ud83d\udcbe ...\n \u2713 Installing StorageClass \ud83d\udcbe\n Set kubectl context to \"kind-dynamic\"\n You can now use your cluster with:\n\n kubectl cluster-info --context kind-dynamic --kubeconfig ./dynamic\n\n Thanks for using kind! \ud83d\ude0a\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | TRY | DONE |\n | 10:45:10 | example | step-2 | TRY | RUN |\n | 10:45:10 | example | step-2 | APPLY | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | CREATE | OK | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | APPLY | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | ASSERT | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | ASSERT | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | TRY | DONE |\n | 10:45:10 | example | step-2 | CLEANUP | RUN |\n | 10:45:10 | example | step-2 | DELETE | RUN | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | DELETE | OK | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | DELETE | DONE | v1/ConfigMap @ default/quick-start\n | 10:45:10 | example | step-2 | CLEANUP | DONE |\n | 10:45:10 | example | step-1 | CLEANUP | RUN |\n | 10:45:10 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c kind delete cluster --name dynamic\n | 10:45:10 | example | step-1 | SCRIPT | LOG |\n === STDERR\n Deleting cluster \"dynamic\" ...\n Deleted nodes: [\"dynamic-control-plane\"]\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | SCRIPT | RUN |\n === COMMAND\n /bin/sh -c rm -f ./dynamic\n | 10:45:10 | example | step-1 | SCRIPT | DONE |\n | 10:45:10 | example | step-1 | CLEANUP | DONE |\n | 10:45:10 | example | @cleanup | DELETE | RUN | v1/Namespace @ chainsaw-useful-seahorse\n | 10:45:11 | example | @cleanup | DELETE | OK | v1/Namespace @ chainsaw-useful-seahorse\n | 10:45:16 | example | @cleanup | DELETE | DONE | v1/Namespace @ chainsaw-useful-seahorse\n
Negative testing is the process of testing cases that are supposed to fail. That is, a test expects errors to happen and if the expected errors don't occur the test must fail.
Chainsaw supports negative testing by letting you decide what should be considered an error or not.
Tip
By default, Chainsaw will consider an operation failed if there was an error executing it (non-zero exit code in scripts and commands, error returned by the API server when calling into Kubernetes, etc...).
The test below expects an error and validates the returned error message:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n content: kubectl get foo\n check:\n ($error != null): true\n ($stderr): |-\n error: the server doesn't have a resource type \"foo\"\n
If for whatever reason, the kubectl get foo doesn't return an error, or the message received in standard error output is not error: the server doesn't have a resource type \"foo\", Chainsaw will consider the operation failed.
If it returns an error and the expected error message, Chainsaw will consider the operation successful.
"},{"location":"examples/negative-testing/#working-with-resources","title":"Working with resources","text":"
The test below tries to apply resources in a cluster but expects the operation to fail:
Under certain circumstances, it makes sense to evaluate assertions that do not depend on resources. For example, when asserting the number of nodes in a cluster is equal to a known value.
The test below uses the x_k8s_list function to query the list of nodes in the cluster. It uses the results to compare the number of nodes found with a known number (1 in this case).
Chainsaw can be used to easily check terminal output from CLIs and other commands. This is useful in that convoluted bash scripts involving chaining together tools like grep can be avoided or at least minimized to only complex use cases. Output to both stdout and stderr can be checked for a given string or precise contents.
One basic use case for content checking is that the output simply contains a given string or piece of content. For example, you might want to run automated tests on a CLI binary you build to ensure that a given command produces output that contains some content you specify somewhere in the output. Let's use the following output from the kubectl version command to show these examples.
Below is an example that ensures the string '1.28' is found somewhere in that output. So long as the content is present anywhere, the test will succeed. To perform this check, the contains() JMESPath filter is used.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures that the string '1.28' is found\n # in stdout or else fails\n (contains($stdout, '1.28')): true\n
Checks for content containing a given value can be negated as well. For example, checking to ensure the output does NOT contain the string '1.25'.
- script:\n content: kubectl version\n check:\n # This check ensures that the string '1.25' is NOT found\n # in stdout or else fails\n (contains($stdout, '1.25')): false\n
"},{"location":"examples/test-output/#checking-output-is-exactly","title":"Checking Output Is Exactly","text":"
In addition to checking that CLI/command output contains some contents, you may need to ensure that the contents are exactly as intended. The Chainsaw test below accomplishes this by comparing the entire contents of stdout with those specified in the block scalar. If so much as one character, space, or line break is off, the test will fail. This is useful in that not only can content be checked but the formatting of that content can be ensured it matches a given declaration.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl version\n check:\n # This check ensures the contents of stdout are exactly as shown.\n # Any deviations will cause a failure.\n ($stdout): |-\n Client Version: v1.28.2\n Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\n Server Version: v1.27.4+k3s1\n
"},{"location":"examples/test-output/#checking-output-in-errors","title":"Checking Output In Errors","text":"
In addition to testing that commands succeed and with output in a given shape, it's equally valuable and necessary to perform negative tests; that tests fail and with contents that are as expected. Similarly, those checks can be for output which has some contents as well as output which appears exactly as desired. For example, you may wish to check that running the kubectl foo command not only fails as expected but that the output shown to users contains a certain word or sentence.
kubectl foo\n\nerror: unknown command \"foo\" for \"kubectl\"\n\nDid you mean this?\n top\n
Below you can see an example where the command kubectl foo is expected to fail but that the error message returned contains some output, in this case the string 'top'.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check bad kubectl command\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This check below ensures that the string 'top' is found in stderr or else fails\n (contains($stderr, 'top')): true\n
Likewise, this failure output can be checked that it is precise. Note that in the example below, due to the use of a tab character in the output of kubectl foo, the value of the ($stderr) field is given as a string to preserve these non-printing characters.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: test\nspec:\n steps:\n - name: Check kubectl\n try:\n - script:\n content: kubectl foo\n check:\n # This checks that the result of the content was an error.\n ($error != null): true\n # This checks that the output is exactly as intended.\n ($stderr): \"error: unknown command \\\"foo\\\" for \\\"kubectl\\\"\\n\\nDid you mean this?\\n\\ttop\"\n
"},{"location":"examples/values/","title":"Pass data to tests","text":"
Chainsaw can pass arbitrary values when running tests using the --values flag. Values will be available to tests under the $values binding.
This is useful when a test needs to be configured externally.
"},{"location":"examples/values/#invoking-chainsaw","title":"Invoking Chainsaw","text":""},{"location":"examples/values/#read-values-from-a-file","title":"Read values from a file","text":"
chainsaw test --values ./values.yaml\n
"},{"location":"examples/values/#read-from-stdin","title":"Read from stdin","text":"
You can think of bindings as a side context where you can store and retrieve data by name.
This is particularly useful when some data is only known at runtime. For example, to pass data from one operation to another, to implement resource templating, to fetch data from an external system, or anything that needs to be computed at runtime.
The test below illustrates bindings declaration at different levels:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # bindings can be declared at the test level\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n # bindings can also be declared at the step level\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n # bindings can also be declared at the operation level\n bindings:\n - name: awesome\n value: awesome\n env:\n # combined bindings together using the `join` functions and\n # assign the result to the GREETINGS environment variable\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n content: echo $GREETINGS\n
Different operations have a different model passed through the assertion tree.
Please consult the Built-in bindings reference documentation to learn what is available depending on the operation.
"},{"location":"general/checks/#expect-vs-check","title":"Expect vs Check","text":"
While a simple check is enough to determine the result of a single operation, we needed a more advanced construct to cover apply, create, delete, patch and update operations. Those operations can operate on files containing multiple resources and every resource can lead to a different result and expectation.
To support more granular checks we use the expect field that contains an array of Expectations.
Every expectation is made of an optional match combined with a check statement.
This way it is possible to control the scope of a check:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - create:\n file: resources.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
In the test above, only config maps are expected to fail. If the resources.yaml file contains other type of resources they are supposed to be created without error (if an error happens for a non config map resource, the operation will be considered a failure).
Chainsaw has a concept of levels and most of the configuration elements and dynamic elements are inherited from one layer to the next in one way or another.
flowchart TD\n Configuration -. Configuration elements are inherited in tests .-> Test\n Test -. Test elements are inherited in test steps .-> Step\n Step -. Step elements are inherited in step operations .-> Operation
The first layer comes from the Chainsaw configuration. You can think about this layer as the global scope and a way to configure how Chainsaw will behave globally.
Under certain circumstances, lower layers will be allowed to consume and/or override elements from upper layers.
By default, Chainsaw will create an ephemeral namespace with a random name for each test, unless a specific namespace name is provided at the global or test level.
One way to control the namespace used to run tests is to specify the name in the Chainsaw configuration Namespace options.
If a namespace name is specified at the configuration level Chainsaw will use it to run the tests (unless an individual test overrides the namespace name).
If the test name is specified in a test spec, Chainsaw will use it to run the test regardless of whether a namespace name was configured at the global level.
Because the name of the namespace is only known at runtime, depending on the resource being manipulated, Chainsaw will eventually inject the namespace name, except if:
The resource below is a namespaced one and has no namespace specified. Chainsaw will automatically inject the namespace name in it:
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: chainsaw-quick-start\n # there is no namespace configured and the resource\n # is a namespaced one.\n # Chainsaw will automatically inject the test namespace\ndata:\n foo: bar\n
Operation outputs can be useful for communicating and reusing computation results across operations.
Chainsaw evaluates outputs after an operation has finished executing. The results of output evaluations are registered in the bindings and are made available for the following operations.
An output supports an optional match field. The match statement is used to conditionally assign the output binding.
The test below illustrates output with matching:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n bindings:\n - name: awesome\n value: awesome\n env:\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n # output is used to register a new `$OUTPUT` binding\n outputs:\n # by default, the `$OUTPUT` binding is assigned\n # the content of the standard output\n - name: OUTPUT\n value: ($stdout)\n # if the match statement evaluates to true,\n # the `$OUTPUT` binding will be set to\n # 'YES! chainsaw is awesome'\n - match:\n ($OUTPUT): hello chainsaw is awesome\n name: OUTPUT\n value: YES! chainsaw is awesome\n content: echo $GREETINGS\n - script:\n # output from the previous operation is used\n # to configure an evironment variable\n env:\n - name: INPUT\n value: ($OUTPUT)\n content: echo $INPUT\n
This doesn't encourage file reuse but can be handy, especially when the resource definition is short or when the execution environment doesn't support file system access.
A third option is to use a URL. Chainsaw uses https://github.com/hashicorp/go-getter, it will download the content from the remote service and load it in the operation resources:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n # use an URL\n file: https://raw.githubusercontent.com/kyverno/chainsaw/main/testdata/step/configmap.yaml\n
When using file-based references, it is important to note that the referenced file(s) can declare multiple resources. Internally, Chainsaw will duplicate the operation once per resource.
This is important to keep this in mind, especially when working with bindings and outputs. Bindings and outputs will be evaluated for every operation instance.
Chainsaw simplifies dynamic configuration with native templating support.
In the past, users have created all sorts of hacks using tools like envsubst for dynamic substitution of env-variables. Those workarounds usually lack flexibility and introduce new problems like hiding the real resources from Chainsaw, preventing it from cleaning resources properly.
Templating in Chainsaw solves exactly this kind of problem.
In the template below, we are using the $namespace binding at two different places, effectively injecting the ephemeral namespace name in the name and the data.foo fields:
"},{"location":"guides/kuttl-migration/","title":"Migration from KUTTL","text":""},{"location":"guides/kuttl-migration/#overview","title":"Overview","text":"
The chainsaw migrate kuttl tests and chainsaw migrate kuttl config commands are designed for the migration of KUTTL tests to Chainsaw.
chainsaw migrate kuttl config
migrates a KUTTL TestSuite to the corresponding Chainsaw Configuration
chainsaw migrate kuttl tests
migrates KUTTL tests to the corresponding Chainsaw Tests
See below for an example test and the corresponding built docs.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: basic\nspec:\n description: This is a very simple test that creates a configmap and checks the content is as expected.\n steps:\n - description: This steps applies the configmap in the cluster and checks the configmap content.\n try:\n - description: Create the configmap.\n apply:\n file: configmap.yaml\n - description: Check the configmap content.\n assert:\n file: configmap-assert.yaml\n
While it is syntactically possible to create an operation with multiple actions, Chainsaw will verify and reject tests if operations containing multiple actions are found.
The reasoning behind this intentional choice is that it becomes harder to understand in which order actions will be executed when an operation consists of multiple actions. For this reason, operations consisting of multiple actions are not allowed.
"},{"location":"operations/#common-fields","title":"Common fields","text":""},{"location":"operations/#continue-on-error","title":"Continue on error","text":"
The continueOnError field determines whether a test step should continue executing or not if the operation fails (in any case the test will be marked as failed).
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n # in case of error the test will be marked as failed\n # but the step will not stop execution and will\n # continue executing the following operations\n - continueOnError: true\n apply:\n resource:\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: quick-start\n data:\n foo: bar\n
The apply operation defines resources that should be applied to a Kubernetes cluster. If the resource does not exist yet it will be created, otherwise, it will be configured to match the provided configuration.
The full structure of the Apply is documented here.
"},{"location":"operations/apply/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/apply/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Assert is documented here.
"},{"location":"operations/assert/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support | Operation checks support"},{"location":"operations/assert/#templating","title":"Templating","text":"
When working with assert and error operations, the content is already an assertion tree and therefore mostly represents a logical operation. An exception to this rule is for fields participating in the resource selection process.
For this reason, only elements used for looking up the resources from the cluster will be considered for templating. That is, apiVersion, kind, name, namespace and labels.
The full structure of the Command is documented here.
"},{"location":"operations/command/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/command/#kubeconfig","title":"KUBECONFIG","text":"
Unless --no-cluster is specified, Chainsaw always executes commands in the context of a temporary KUBECONFIG, built from the configured target cluster.
This specific KUBECONFIG has a single cluster, auth info and context configured (all named chainsaw).
The full structure of the Create is documented here.
"},{"location":"operations/create/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/create/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - create:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The delete operation defines resources that should be deleted from a Kubernetes cluster.
Warning
The propagation policy is forced to Background because some types default to Orphan (this is the case for unmanaged jobs for example) and we don't want to let dangling pods run in the cluster after cleanup.
The full structure of the Delete is documented here.
"},{"location":"operations/delete/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/delete/#examples","title":"Examples","text":"
The error operation lets you define a set of expected errors for a test step. If any of these errors occur during the test, they are treated as expected outcomes. However, if an error that's not on this list occurs, it will be treated as a test failure.
The full structure of the Error is documented here.
"},{"location":"operations/error/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support | Operation checks support"},{"location":"operations/error/#templating","title":"Templating","text":"
When working with assert and error operations, the content is already an assertion tree and therefore mostly represents a logical operation. An exception to this rule is for fields participating in the resource selection process.
For this reason, only elements used for looking up the resources from the cluster will be considered for templating. That is, apiVersion, kind, name, namespace and labels.
The full structure of the Patch is documented here.
"},{"location":"operations/patch/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/patch/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - patch:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Script is documented here.
"},{"location":"operations/script/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/script/#kubeconfig","title":"KUBECONFIG","text":"
Unless --no-cluster is specified, Chainsaw always executes commands in the context of a temporary KUBECONFIG, built from the configured target cluster.
This specific KUBECONFIG has a single cluster, auth info and context configured (all named chainsaw).
The full structure of the Sleep is documented here.
"},{"location":"operations/sleep/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/sleep/#examples","title":"Examples","text":"
The full structure of the Update is documented here.
"},{"location":"operations/update/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/update/#examples","title":"Examples","text":"
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - update:\n file: my-configmap.yaml\n expect:\n - match:\n # this check applies only if the match\n # statement below evaluates to `true`\n apiVersion: v1\n kind: ConfigMap\n check:\n # an error is expected, this will:\n # - succeed if the operation failed\n # - fail if the operation succeeded\n ($error != null): true\n
The full structure of the Describe resource is documented here.
"},{"location":"operations/helpers/describe/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/describe/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
The full structure of the Events resource is documented here.
"},{"location":"operations/helpers/events/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/events/#test-namespace","title":"Test namespace","text":"
When used with a namespaced resource, Chainsaw will default the scope to the ephemeral test namespace.
The full structure of the Get resource is documented here.
"},{"location":"operations/helpers/get/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/get/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
The full structure of the PodLogs resource is documented here.
"},{"location":"operations/helpers/logs/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/logs/#test-namespace","title":"Test namespace","text":"
Chainsaw will default the scope to the ephemeral test namespace.
The full structure of the Wait resource is documented here.
"},{"location":"operations/helpers/wait/#features","title":"Features","text":"Supported features Bindings support Outputs support Templating support Operation checks support"},{"location":"operations/helpers/wait/#clustered-resources","title":"Clustered resources","text":"
When used with a clustered resource, the namespace is ignored and is not added to the corresponding kubectl command.
Chainsaw allows declaring complex assertions with a simple and no-code approach, allowing assertions based on comparisons beyond simple equality, working with arrays, and other scenarios that could not be achieved before.
Tip
Under the hood, Chainsaw uses kyverno-json assertion trees. Refer to the assertion trees documentation for more details on the supported syntax.
When asking Chainsaw to execute the assertion above, it will look for a deployment named coredns in the kube-system namespace and will compare the existing resource with the (partial) resource definition contained in the assertion.
In this specific case, if the field spec.replicas is set to 2 in the existing resource, the assertion will be considered valid. If it is not equal to 2 the assertion will be considered failed.
This is the most basic assertion Chainsaw can evaluate.
"},{"location":"quick-start/assertion-trees/#slightly-less-basic-assertion","title":"Slightly less basic assertion","text":"
Chainsaw will look up all deployments with the k8s-app: kube-dns label in the kube-system namespace. The assertion will be considered valid if at least one deployment matches the (partial) resource definition contained in the assertion. If none match, the assertion will be considered failed.
Apart from the resource lookup process being a little bit more interesting, this kind of assertion is essentially the same as the previous one. Chainsaw is basically making a decision by comparing an actual and expected resource.
The assertion below will check that the number of replicas for a deployment is greater than 1 AND less than 4.
Chainsaw doesn't need to know the exact expected number of replicas. The (replicas > 1 && replicas < 4) expression will be evaluated until the result is true or the operation timeout expires (making the assertion fail).
Chainsaw offers detailed resource diffs upon assertion failures.
In the example below, the assertion failure message (metadata.annotations.foo: Invalid value: \"null\": Expected value: \"bar\") is augmented with a resource diff.
It provides a clear view of discrepancies between expected and actual resources and gives more context around the specific failure (we can easily identify the owner of the offending pod for example).
You can think of bindings as a side context where you can store and retrieve data based on keys.
This is particularly useful when some data is only known at runtime. For example, to pass data from one operation to another, to implement resource templating, to fetch data from an external system, etc.
Chainsaw offers some built-in bindings you can directly use in your tests but you can also create your own bindings if needed.
The $namespace binding is a good example of a built-in binding provided by Chainsaw. It contains the name of the ephemeral namespace used to execute a test (by default Chainsaw will create an ephemeral namespace for each test).
In the operation below, we are assigning the value of the $namespace binding to an environment variable, and echo it in a script:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n env:\n # assign the value of the `$namespace` binding\n # to the environment variable `FOO`\n - name: FOO\n value: ($namespace)\n content: echo $FOO\n
On top of built-in bindings, you can also create your own ones, combine bindings together, call JMESPath functions using bindings as arguments, etc.
In the test below we create custom bindings at different levels in the test, combine them by calling the join function, assign the result to an environment variable, and echo it in a script:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # bindings can be declared at the test level\n bindings:\n - name: chainsaw\n value: chainsaw\n steps:\n # bindings can also be declared at the step level\n - bindings:\n - name: hello\n value: hello\n try:\n - script:\n # bindings can also be declared at the operation level\n bindings:\n - name: awesome\n value: awesome\n env:\n # combined bindings together using the `join` functions and\n # assign the result to the GREETINGS environment variable\n - name: GREETINGS\n value: (join(' ', [$hello, $chainsaw, 'is', $awesome]))\n content: echo $GREETINGS\n
Let's see how bindings can be useful with resource templating.
"},{"location":"quick-start/cleanup/","title":"Control your cleanup","text":"
Unless configured differently, by default Chainsaw will automatically remove the resources it created after a test finishes.
Cleanup happens in reverse order of creation (created last, cleaned up first). This is important, especially when the controller being tested makes use of finalizers.
Overriding cleanup timeout
Note that Chainsaw performs a blocking deletion, that is, it will wait until the resource is not present anymore in the cluster before proceeding with the next resource cleanup.
Under certain circumstances, automatic cleanup is not enough and we want to execute custom operations.
Chainsaw allows registering cleanup operations that will be run after automatic cleanup. Custom cleanup operations live at the test step level:
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # this step will create a local cluster\n - try:\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n # at cleanup time, we want to delete the local cluster we created\n # and remove the associated kubeconfig\n cleanup:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n
Once installed, use chainsaw completion command to generate and register the autocompletion script for the specified shell.
Supported shells are:
bash
fish
powershell
zsh
"},{"location":"quick-start/first-test/","title":"Create a test","text":"
To create a Chainsaw test all you need to do is to create one (or more) YAML file(s).
The recommended approach is to create one folder per test, with a chainsaw-test.yaml file containing one (or more) test definition(s). The test definition can reference other files in the same folder or anywhere else on the file system as needed.
Tip
While chainsaw supports other syntaxes, we strongly recommend the explicit approach.
"},{"location":"quick-start/first-test/#what-is-a-test","title":"What is a test?","text":"
To put it simply, a test can be represented as an ordered sequence of test steps.
In turn, a test step can be represented as an ordered sequence of operations.
When one of the operations fails the test is considered failed.
If all operations succeed the test is considered successful.
"},{"location":"quick-start/first-test/#lets-write-our-first-test","title":"Let's write our first test","text":"
For this quick start, we will create a (very simple) Test with one step and two operations:
Create a ConfigMap from a manifest
Verify the ConfigMap was created and contains the expected data
Follow the instructions below to create the folder and files defining our first test.
"},{"location":"quick-start/first-test/#create-a-test-folder","title":"Create a test folder","text":"
# create test folder\nmkdir chainsaw-quick-start\n\n# enter test folder\ncd chainsaw-quick-start\n
"},{"location":"quick-start/first-test/#create-a-configmap-manifest","title":"Create a ConfigMap manifest","text":"
"},{"location":"quick-start/first-test/#create-a-test-manifest","title":"Create a test manifest","text":"
By default, Chainsaw will look for a file named chainsaw-test.yaml in every folder.
# create test file\ncat > chainsaw-test.yaml << EOF\napiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: quick-start\nspec:\n steps:\n - try:\n # first operation: create the config map\n - apply:\n # file is relative to the test folder\n file: configmap.yaml\n # second operation: verify the config map exists and contains the expected data\n - assert:\n # file is relative to the test folder\n file: configmap.yaml\nEOF\n
You can install the pre-compiled binary (in several ways), compile from sources, or run with Docker.
We also provide a GitHub action to easily install Chainsaw in your workflows.
"},{"location":"quick-start/install/#install-the-pre-compiled-binary","title":"Install the pre-compiled binary","text":""},{"location":"quick-start/install/#homebrew-tap","title":"Homebrew tap","text":"
add tap:
brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw\n
install chainsaw:
brew install kyverno/chainsaw/chainsaw\n
Don't forget to specify the tap name
Homebrew core already has a tool named chainsaw.
Be sure that you specify the tap name when installing to install the right tool.
Download the pre-compiled binaries for your system from the releases page and copy them to the desired location.
"},{"location":"quick-start/install/#install-using-go-install","title":"Install using go install","text":"
You can install with go install with:
go install github.com/kyverno/chainsaw@latest\n
"},{"location":"quick-start/install/#run-with-docker","title":"Run with Docker","text":"
Chainsaw is also available as a Docker image which you can pull and run:
docker pull ghcr.io/kyverno/chainsaw:<version>\n
Warning
Since Chainsaw relies on files for its operation (like test definitions), you will need to bind mount the necessary directories when running it via Docker.
Using nix-env permanently modifies a local profile of installed packages. This must be updated and maintained by the user in the same way as with a traditional package manager, foregoing many of the benefits that make Nix uniquely powerful. Using nix-shell or a NixOS configuration is recommended instead.
A nix-shell will temporarily modify your $PATH environment variable. This can be used to try a piece of software before deciding to permanently install it. Use the following command to install kyverno-chainsaw :
An output supports an optional match field. The match is used to conditionally create a binding.
In the case of applying a file, for example, the file may contain multiple resources. The match can be used to select the resource to use for creating the binding.
"},{"location":"quick-start/operation-outputs/#load-an-existing-resource","title":"Load an existing resource","text":"
The example below invokes a kubectl command to get a configmap from the cluster in json format.
The json output is then parsed and added to the $cm binding and the next operation performs an assertion on it by reading the binding instead of querying the cluster.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - script:\n content: kubectl get cm quick-start -n $NAMESPACE -o json\n outputs:\n # parse stdout json output and bind the result to `$cm`\n - name: cm\n value: (json_parse($stdout))\n - assert:\n resource:\n ($cm):\n metadata:\n (uid != null): true\n
"},{"location":"quick-start/operation-outputs/#match-a-resource","title":"Match a resource","text":"
The example below applies resources from a file.
When the resource being applied is a configmap, we bind the resource to an output to print its UID in the next operation.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n file: ./resources.yaml\n outputs:\n # match the configmap resource and bind it to `$cm`\n - match:\n apiVersion: v1\n kind: ConfigMap\n name: cm\n value: (@)\n - script:\n env:\n - name: UID\n value: ($cm.metadata.uid)\n content: echo $UID\n
Chainsaw simplifies dynamic resource configuration with native resource templating support.
Sometimes things we need to create resources or assertions are only known at runtime.
In the past, users have created all sorts of hacks using tools like envsubst for dynamic substitution of env-variables. Those workarounds usually lack flexibility and introduce new problems like hiding the real resources from Chainsaw, preventing it from cleaning resources properly.
Tip
Resource templating is heavily based on bindings and uses JMESPath language.
In the template below, we are using the $namespace binding at two different places, effectively injecting the ephemeral namespace name in the name and the data.foo fields:
After installing chainsaw and writing tests, the next natural step is to run Chainsaw to execute the tests.
"},{"location":"quick-start/run-tests/#create-a-local-cluster","title":"Create a local cluster","text":"
To use Chainsaw you will need a Kubernetes cluster, Chainsaw won't create one for you.
Not a cluster management tool
We consider this is not the responsibility of Chainsaw to manage clusters. There are plenty of solutions to create and manage local clusters that will do that better than Chainsaw.
The command below will create a local cluster using kind. Use the tool of your choice or directly jump to the next section if you already have a KUBECONFIG configured and pointing to a valid cluster.
Chainsaw expects a path to the test folder and will discover tests by analyzing files recursively. When no path is provided Chainsaw will use the current path by default (.).
The test above demonstrates the most basic usage of Chainsaw. In the next sections, we will look at the main features that make Chainsaw a very unique tool.
"},{"location":"quick-start/timeouts/","title":"Control your timeouts","text":"
Timeouts in Chainsaw are specified per type of operation. This is handy because the timeout varies greatly depending on the nature of an operation.
For example, applying a manifest in a cluster is expected to be reasonably fast, while validating a resource can be a long operation.
Timeouts can be configured globally and at the test, step or individual operation level.
All timeouts configured at a given level are automatically inherited in child levels. When looking up a timeout, the most specific one takes precedence over the others.
Info
To learn more about timeouts and how to configure global values, see the timeouts configuration page.
"},{"location":"quick-start/timeouts/#at-the-test-level","title":"At the test level","text":"
When a timeout is configured at the test level it will apply to all operations and steps in the test, unless overridden at a more specific level.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # timeouts configured at the test level will apply to all operations and steps\n # unless overriden at the step level and/or individual operation level\n timeouts:\n apply: 5s\n assert: 1m\n # ...\n steps:\n - try:\n - apply:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n - assert:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
"},{"location":"quick-start/timeouts/#at-the-step-level","title":"At the step level","text":"
When a timeout is configured at the step level it will apply to all operations in the step, unless overridden at a more specific level.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # timeouts configured at the step level will apply to all operations\n # in the step unless overriden at the individual operation level\n - timeouts:\n apply: 5s\n # ...\n try:\n - apply:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n # timeouts configured at the step level will apply to all operations\n # in the step unless overriden at the individual operation level\n - timeouts:\n assert: 1m\n # ...\n try:\n - assert:\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
"},{"location":"quick-start/timeouts/#at-the-operation-level","title":"At the operation level","text":"
When a timeout is configured at the operation level, it takes precedence over all timeouts configured at upper levels.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n - try:\n - apply:\n # timeout configured at the operation level takes precedence\n # over timeouts configured at upper levels\n timeout: 5s\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n spec:\n storage:\n secret:\n name: minio\n type: s3\n # ...\n - assert:\n # timeout configured at the operation level takes precedence\n # over timeouts configured at upper levels\n timeout: 1m\n resource:\n apiVersion: tempo.grafana.com/v1alpha1\n kind: TempoStack\n metadata:\n name: simplest\n status:\n (conditions[?type == 'Ready']):\n - status: 'True'\n
In the next section, we will see how Chainsaw manages cleanup.
"},{"location":"quick-start/try-catch/","title":"Use try, catch and finally","text":"
A test step is made of 3 main blocks used to determine the actions Chainsaw will perform when executing the step, depending on operations outcome.
The try block (required)
The catch block (optional)
The finally block (optional)
Operations defined in the try block are executed first, then:
If an operation fails to execute, Chainsaw won't execute the remaining operations and will execute all operations defined in the catch block instead (if any).
If all operations succeed, Chainsaw will NOT execute operations defined in the catch block (if any).
Regardless of the step outcome (success or failure), Chainsaw will execute all operations defined in the finally block (if any).
Note
Note that all operations coming from the catch or finally blocks are executed. If one operation fails, Chainsaw will mark the test as failed and continue executing with the next operation.
At the end of a test, Chainsaw automatically cleans up the resources created during the test (cleanup is done in the opposite order of creation).
All operations from the catch and finally blocks are executed before the cleanup process kicks in. This order allows analyzing the resources that potentially caused the step failure before they are deleted.
Operations in a finally block will always execute regardless of the success or failure of the test step.
This is particularly useful to perform manual cleanup.
In the example below we create a local cluster in a script operation. The cluster deletion script is added to the finally block, guaranteeing the cluster will be deleted regardless of the test outcome.
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # create a local cluster\n - try:\n - script:\n timeout: 1m\n content: |\n kind create cluster --name dynamic --kubeconfig ./dynamic\n - apply:\n # ...\n - assert:\n # ...\n # add cluster deletion script in the `finally` block\n # to guarantee the cluster will be deleted after the test\n finally:\n - script:\n content: |\n kind delete cluster --name dynamic\n - script:\n content: |\n rm -f ./dynamic\n
"},{"location":"reference/builtins/#common","title":"Common","text":"Name Purpose Type $values Values provided when invoking chainsaw with --values flag any$namespace Name of the current test namespace string$client Kubernetes client chainsaw is connected to (if not running with --no-cluster) object$config Kubernetes client config chainsaw is connected to (if not running with --no-cluster) object"},{"location":"reference/builtins/#in-tests","title":"In tests","text":"Name Purpose Type $test.id Current test id int$test.metadata Current test metadata metav1.ObjectMeta
Note
$test.id starts at 1 for the first test
"},{"location":"reference/builtins/#in-steps","title":"In steps","text":"Name Purpose Type $step.id Current step id int
Note
$step.id starts at 1 for the first step
"},{"location":"reference/builtins/#in-operations","title":"In operations","text":"Name Purpose Type $operation.id Current operation id int$operation.resourceId Current resource id int
Note
$operation.id starts at 1 for the first operation
$operation.resourceId maps to the resource id (starting at 1) in case the operation loads a file that contains multiple resources (the same operation is repeated once per resource)
"},{"location":"reference/builtins/#in-checks-and-outputs","title":"In checks and outputs","text":"Name Purpose Type @ The state of the resource (if any) at the end of the operation any$error The error message (if any) at the end of the operation string$stdout The content of the standard console output (if any) at the end of the operation string$stderr The content of the standard console error output (if any) at the end of the operation string
Note
$stdout and $stderr are only available in script and command operations
ActionObject contains object selector options for an action.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ActionObjectSelectorActionObjectSelector No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ActionObjectSelector","title":"ActionObjectSelector","text":"
Appears in:
ActionObject
Events
PodLogs
ActionObjectSelector contains object selector options for an action.
Field Type Required Inline Description ObjectNameObjectName No description provided. selectorstring
Apply represents a set of configurations or resources that should be applied during testing.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Assert","title":"Assert","text":"
Appears in:
Operation
Assert represents a test condition that is expected to hold true during the testing process.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckRefActionCheckRef No description provided. ActionClustersActionClusters No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Binding","title":"Binding","text":"
Appears in:
ActionBindings
ActionEnv
Output
Scenario
StepTemplateSpec
TestSpec
TestStepSpec
Binding represents a key/value set as a binding in an executing test.
Command describes a command to run as a part of a test step.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckActionCheck No description provided. ActionClustersActionClusters No description provided. ActionEnvActionEnv No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. entrypointstring
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation.
reportFormatReportFormatType
ReportFormat determines test report format (JSON reportPathstring
ReportPath defines the path.
reportNamestring
ReportName defines the name of report to create. It defaults to \"chainsaw-report\".
namespacestring
Namespace defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
namespaceTemplatepolicy/v1alpha1.Any
NamespaceTemplate defines a template to create the test namespace.
fullNamebool
FullName makes use of the full test case folder path instead of the folder name.
excludeTestRegexstring
ExcludeTestRegex is used to exclude tests based on a regular expression.
includeTestRegexstring
IncludeTestRegex is used to include tests based on a regular expression.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
testFilestring
TestFile is the name of the file containing the test to run. If no extension is provided, chainsaw will try with .yaml first and .yml if needed.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
catch[]CatchFinally
Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels.
Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Delete","title":"Delete","text":"
Appears in:
CatchFinally
Operation
Delete is a reference to an object that should be deleted
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionExpectationsActionExpectations No description provided. ActionTimeoutActionTimeout No description provided. templatebool
Template determines whether resources should be considered for templating.
filestring
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided. showEventsbool
Show Events indicates whether to include related events.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckRefActionCheckRef No description provided. ActionClustersActionClusters No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Events","title":"Events","text":"
Appears in:
CatchFinally
Operation
Events defines how to collect events.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionFormatActionFormat No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Expectation","title":"Expectation","text":"
Appears in:
ActionExpectations
Expectation represents a check to be applied on the result of an operation with a match filter to determine if the verification should be considered.
Field Type Required Inline Description matchpolicy/v1alpha1.Any
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-ObjectName","title":"ObjectName","text":"
Appears in:
ActionObjectSelector
ObjectReference
Proxy
ObjectName represents an object namespace and name.
Field Type Required Inline Description namespacestring
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
namestring
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ObjectNameObjectName No description provided. labelsmap[string]string
Operation defines a single operation, only one action is permitted for a given operation.
Field Type Required Inline Description OperationBaseOperationBase
OperationBase defines common elements to all operations.
applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a deletion operation.
describeDescribe
Describe determines the resource describe collector to execute.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
eventsEvents
Events determines the events collector to execute.
getGet
Get determines the resource get collector to execute.
patchPatch
Patch represents a patch operation.
podLogsPodLogs
PodLogs determines the pod logs collector to execute.
proxyProxy
Proxy runs a proxy request.
scriptScript
Script defines a script to run.
sleepSleep
Sleep defines zzzz.
updateUpdate
Update represents an update operation.
waitWait
Wait determines the resource wait collector to execute.
OperationBase defines common elements to all operations.
Field Type Required Inline Description descriptionstring
Description contains a description of the operation.
continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-PodLogs","title":"PodLogs","text":"
Appears in:
CatchFinally
Operation
PodLogs defines how to collect pod logs.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided. containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Field Type Required Inline Description ActionClustersActionClusters No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. ObjectNameObjectName No description provided. ObjectTypeObjectType No description provided. portstring
TargetPort defines the target port to proxy the request.
pathstring
TargetPath defines the target path to proxy the request.
Script describes a script to run as a part of a test step.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionCheckActionCheck No description provided. ActionClustersActionClusters No description provided. ActionEnvActionEnv No description provided. ActionOutputsActionOutputs No description provided. ActionTimeoutActionTimeout No description provided. contentstring
Content defines a shell script (run with \"sh -c ...\").
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
clusterstring
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
skipDeletebool
SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
templatebool
Template determines whether resources should be considered for templating.
bindings[]Binding
Bindings defines additional binding key/values.
try[]Operation
Try defines what the step will try to execute.
catch[]CatchFinally
Catch defines what the step will execute when an error happens.
finally[]CatchFinally
Finally defines what the step will execute after the step is terminated.
cleanup[]CatchFinally
Cleanup defines what will be executed after the test is terminated.
Update represents a set of resources that should be updated. If a resource does not exist in the cluster it will fail.
Field Type Required Inline Description ActionBindingsActionBindings No description provided. ActionClustersActionClusters No description provided. ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionOutputsActionOutputs No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha1/#chainsaw-kyverno-io-v1alpha1-Wait","title":"Wait","text":"
Appears in:
CatchFinally
Operation
Wait specifies how to perform wait operations on resources.
Field Type Required Inline Description ActionTimeoutActionTimeout No description provided. ActionFormatActionFormat No description provided. ActionClustersActionClusters No description provided. ActionObjectActionObject No description provided. forWaitFor
ActionObject contains object selector options for an action.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ActionObjectSelectorActionObjectSelector No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ActionObjectSelector","title":"ActionObjectSelector","text":"
Appears in:
ActionObject
Events
PodLogs
ActionObjectSelector contains object selector options for an action.
Field Type Required Inline Description ObjectNameObjectName No description provided. selectorstring
Apply represents a set of configurations or resources that should be applied during testing.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Assert","title":"Assert","text":"
Appears in:
OperationAction
Assert represents a test condition that is expected to hold true during the testing process.
Field Type Required Inline Description ActionCheckRefActionCheckRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-CleanupOptions","title":"CleanupOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
CleanupOptions contains the configuration used for cleaning up resources.
Field Type Required Inline Description skipDeletebool
If set, do not delete the resources after running a test.
delayBeforeCleanupmeta/v1.Duration
DelayBeforeCleanup adds a delay between the time a test ends and the time cleanup starts.
Command describes a command to run as a part of a test step.
Field Type Required Inline Description ActionCheckActionCheck No description provided. ActionEnvActionEnv No description provided. ActionTimeoutActionTimeout No description provided. entrypointstring
Create represents a set of resources that should be created. If a resource already exists in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Delete","title":"Delete","text":"
Appears in:
OperationAction
Delete is a reference to an object that should be deleted
Field Type Required Inline Description ActionExpectationsActionExpectations No description provided. ActionTimeoutActionTimeout No description provided. templatebool
Template determines whether resources should be considered for templating.
filestring
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in the Configuration, the Test and the TestStep.
Field Type Required Inline Description ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided. showEventsbool
Show Events indicates whether to include related events.
Error represents an anticipated error condition that may arise during testing. Instead of treating such an error as a test failure, it acknowledges it as expected.
Field Type Required Inline Description ActionCheckRefActionCheckRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ErrorOptions","title":"ErrorOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
ErrorOptions contains the global error configuration.
Field Type Required Inline Description catch[]CatchFinally
Catch defines what the tests steps will execute when an error happens. This will be combined with catch handlers defined at the test and step levels.
Field Type Required Inline Description ActionFormatActionFormat No description provided. ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-ExecutionOptions","title":"ExecutionOptions","text":"
Appears in:
ConfigurationSpec
ExecutionOptions determines how tests are run.
Field Type Required Inline Description failFastbool
FailFast determines whether the test should stop upon encountering the first failure.
parallelint
The maximum number of tests to run at once.
repeatCountint
RepeatCount indicates how many times the tests should be executed.
forceTerminationGracePeriodmeta/v1.Duration
ForceTerminationGracePeriod forces the termination grace period on pods, statefulsets, daemonsets and deployments.
File is the path to the referenced file. This can be a direct path to a file or an expression that matches multiple files, such as \"manifest/*.yaml\" for all YAML files within the \"manifest\" directory.
Field Type Required Inline Description ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-NamespaceOptions","title":"NamespaceOptions","text":"
Appears in:
ConfigurationSpec
TestSpec
NamespaceOptions contains the configuration used to allocate a namespace for each test.
Field Type Required Inline Description namestring
Name defines the namespace to use for tests. If not specified, every test will execute in a random ephemeral namespace unless the namespace is overridden in a the test spec.
templatepolicy/v1alpha1.Any
Template defines a template to create the test namespace.
ObjectReference represents one or more objects with a specific apiVersion and kind. For a single object name and namespace are used to identify the object. For multiple objects use labels.
Field Type Required Inline Description ObjectTypeObjectType No description provided. ObjectNameObjectName No description provided. labelSelectormeta/v1.LabelSelector
Field Type Required Inline Description OperationActionOperationAction No description provided. OperationBindingsOperationBindings No description provided. OperationClustersOperationClusters No description provided. OperationOutputsOperationOutputs No description provided. descriptionstring
Description contains a description of the operation.
OperationAction defines an operation action, only one action should be specified per operation.
Field Type Required Inline Description applyApply
Apply represents resources that should be applied for this test step. This can include things like configuration settings or any other resources that need to be available during the test.
assertAssert
Assert represents an assertion to be made. It checks whether the conditions specified in the assertion hold true.
commandCommand
Command defines a command to run.
createCreate
Create represents a creation operation.
deleteDelete
Delete represents a deletion operation.
describeDescribe
Describe determines the resource describe collector to execute.
errorError
Error represents the expected errors for this test step. If any of these errors occur, the test will consider them as expected; otherwise, they will be treated as test failures.
eventsEvents
Events determines the events collector to execute.
getGet
Get determines the resource get collector to execute.
patchPatch
Patch represents a patch operation.
podLogsPodLogs
PodLogs determines the pod logs collector to execute.
scriptScript
Script defines a script to run.
sleepSleep
Sleep defines zzzz.
updateUpdate
Update represents an update operation.
waitWait
Wait determines the resource wait collector to execute.
Patch represents a set of resources that should be patched. If a resource doesn't exist yet in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-PodLogs","title":"PodLogs","text":"
Appears in:
OperationAction
PodLogs defines how to collect pod logs.
Field Type Required Inline Description ActionObjectSelectorActionObjectSelector No description provided. ActionTimeoutActionTimeout No description provided. containerstring
Container in pod to get logs from else --all-containers is used.
tailint
Tail is the number of last lines to collect from pods. If omitted or zero, then the default is 10 if you use a selector, or -1 (all) if you use a pod name. This matches default behavior of kubectl logs.
Script describes a script to run as a part of a test step.
Field Type Required Inline Description ActionCheckActionCheck No description provided. ActionEnvActionEnv No description provided. ActionTimeoutActionTimeout No description provided. contentstring
Content defines a shell script (run with \"sh -c ...\").
DeletionPropagationPolicy decides if a deletion will propagate to the dependents of the object, and how the garbage collector will handle the propagation. Overrides the deletion propagation policy set in both the Configuration and the Test.
clusterstring
Cluster defines the target cluster (default cluster will be used if not specified and/or overridden).
clustersClusters
Clusters holds a registry to clusters to support multi-cluster tests.
skipDeletebool
SkipDelete determines whether the resources created by the step should be deleted after the test step is executed.
templatebool
Template determines whether resources should be considered for templating.
bindings[]Binding
Bindings defines additional binding key/values.
try[]TryOperation
Try defines what the step will try to execute.
catch[]Operation
Catch defines what the step will execute when an error happens.
finally[]Operation
Finally defines what the step will execute after the step is terminated.
cleanup[]Operation
Cleanup defines what will be executed after the test is terminated.
Field Type Required Inline Description OperationOperation No description provided. continueOnErrorbool
ContinueOnError determines whether a test should continue or not in case the operation was not successful. Even if the test continues executing, it will still be reported as failed.
Update represents a set of resources that should be updated. If a resource does not exist in the cluster it will fail.
Field Type Required Inline Description ActionDryRunActionDryRun No description provided. ActionExpectationsActionExpectations No description provided. ActionResourceRefActionResourceRef No description provided. ActionTimeoutActionTimeout No description provided."},{"location":"reference/apis/chainsaw.v1alpha2/#chainsaw-kyverno-io-v1alpha2-Wait","title":"Wait","text":"
Appears in:
OperationAction
Wait specifies how to perform wait operations on resources.
Field Type Required Inline Description ActionTimeoutActionTimeout No description provided. ActionFormatActionFormat No description provided. ActionObjectActionObject No description provided. forWaitFor
--clustered Defines if the resource is clustered (only applies when resource is loaded from a file)\n -f, --file string Path to the file to assert or '-' to read from stdin\n -h, --help help for assert\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use (default \"default\")\n --no-color Removes output colors\n -r, --resource string Path to the file containing the resource\n --timeout duration The assert timeout to use (default 30s)\n
--catalog string Path to the built test catalog file\n -h, --help help for docs\n --readme-file string Name of the built docs file (default \"README.md\")\n --test-dir stringArray Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test\")\n
--description If set, adds description when applicable (default true)\n --force If set, existing test will be deleted if needed\n -h, --help help for test\n --save If set, created test will be saved\n
--autogenTag Determines if the generated docs should contain a timestamp (default true)\n -h, --help help for docs\n -o, --output string Output path (default \".\")\n --website Website version\n
--apply-timeout duration The apply timeout to use as default for configuration (default 5s)\n --assert-timeout duration The assert timeout to use as default for configuration (default 30s)\n --cleanup-delay duration Adds a delay between the time a test ends and the time cleanup starts\n --cleanup-timeout duration The cleanup timeout to use as default for configuration (default 30s)\n --cluster strings Register cluster (format <cluster name>=<kubeconfig path>:[context name])\n --config string Chainsaw configuration file\n --delete-timeout duration The delete timeout to use as default for configuration (default 15s)\n --deletion-propagation-policy string The deletion propagation policy (Foreground|Background|Orphan) (default \"Background\")\n --error-timeout duration The error timeout to use as default for configuration (default 30s)\n --exclude-test-regex string Regular expression to exclude tests\n --exec-timeout duration The exec timeout to use as default for configuration (default 5s)\n --fail-fast Stop the test upon encountering the first failure\n --force-termination-grace-period duration If specified, overrides termination grace periods in applicable resources\n --full-name Use full test case folder path instead of folder name\n -h, --help help for test\n --include-test-regex string Regular expression to include tests\n --kube-as string Username to impersonate for the operation\n --kube-as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups.\n --kube-as-uid string UID to impersonate for the operation\n --kube-certificate-authority string Path to a cert file for the certificate authority\n --kube-client-certificate string Path to a client certificate file for TLS\n --kube-client-key string Path to a client key file for TLS\n --kube-cluster string The name of the kubeconfig cluster to use\n --kube-context string The name of the kubeconfig context to use\n --kube-disable-compression If true, opt-out of response compression for all requests to the server\n --kube-insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure\n -n, --kube-namespace string If present, the namespace scope for this CLI request\n --kube-password string Password for basic authentication to the API server\n --kube-proxy-url string If provided, this URL will be used to connect via proxy\n --kube-request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default \"0\")\n --kube-server string The address and port of the Kubernetes API server\n --kube-tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.\n --kube-token string Bearer token for authentication to the API server\n --kube-user string The name of the kubeconfig user to use\n --kube-username string Username for basic authentication to the API server\n --namespace string Namespace to use for tests\n --no-cluster Runs without cluster\n --no-color Removes output colors\n --parallel int The maximum number of tests to run at once\n --pause-on-failure Pause test execution failure (implies no concurrency)\n --remarshal Remarshals tests yaml to apply anchors before parsing\n --repeat-count int Number of times to repeat each test (default 1)\n --report-format string Test report format (JSON|XML|nil)\n --report-name string The name of the report to create (default \"chainsaw-report\")\n --report-path string The path of the report to create\n --selector strings Selector (label query) to filter on\n --skip-delete If set, do not delete the resources after running the tests\n --template If set, resources will be considered for templating (default true)\n --test-dir strings Directories containing test cases to run\n --test-file string Name of the test file (default \"chainsaw-test\")\n --values strings Values passed to the tests\n
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n steps:\n # `try` defines operations to execute in the step\n - try: [...]\n # `catch` defines operations to execute when the step fails\n catch: [...]\n # `finally` defines operations to execute at the end of the step\n finally: [...]\n # `cleanup` defines operations to execute at the end of the test\n cleanup: [...]\n
Operations defined in the try block are executed first, then:
If an operation fails to execute, Chainsaw won't execute the remaining operations and will execute all operations defined in the catch block instead (if any).
If all operations succeed, Chainsaw will NOT execute operations defined in the catch block (if any).
Regardless of the step outcome (success or failure), Chainsaw will execute all operations defined in the finally block (if any).
Tip
Note that all operations coming from the catch or finally blocks are executed. If one operation fails, Chainsaw will mark the test as failed and continue executing with the next operations.
sequenceDiagram\n autonumber\n\n participant S as Step N\n\n box Try block\n participant T1 as Op 1\n participant T2 as Op N\n end\n box Catch block\n end\n box Finally block\n participant F1 as Op 1\n participant F2 as Op N\n end\n participant S1 as Step N+1\n\n S -->> T1 : try\n T1 ->> T2 : success\n T2 -->> S : done\n S -->> F1 : finally\n F1 ->> F2 : done\n F2 -->> S : done\n S -->> S1 : next step
Step starts by executing operations in the try block
Operations in the try block execute sequentially
All operations in the try block terminate
Step starts executing operations in the finally block
Operations in the finally block execute sequentially
sequenceDiagram\n autonumber\n\n participant S as Step N\n\n box Try block\n participant T1 as Op 1\n participant T2 as Op N\n end\n box Catch block\n participant C1 as Op 1\n participant C2 as Op N\n end\n box Finally block\n participant F1 as Op 1\n participant F2 as Op N\n end\n\n S -->> T1 : try\n T1 ->> T2 : success\n T2 -->> S : error\n S -->> C1 : catch\n C1 ->> C2 : done\n C2 -->> S : done\n S -->> F1 : finally\n F1 ->> F2 : done\n F2 -->> S : done
Step starts by executing operations in the try block
Operations in the try block execute sequentially until an error happens
Operations in the try block stop when an error occurs
Step starts executing operations in the catch block
Operations in the catch block execute sequentially
All operations in the catch block terminate
Step starts executing operations in the finally block
Operations in the finally block execute sequentially
Under certain circumstances, it can be useful to configure catch blocks at a higher level than the step grain. At the test or configuration level.
This allows for declaring common catch statements we want to execute when an error occurs. Those catch blocks are combined to produce the final catch block in the following order:
catch statements from the configuration level are executed first (if any)
catch statements from the test level are executed next (if any)
catch statements from the step level are executed last (if any)
A cleanup statement is similar to a finally statement but will execute after the test finishes executing, while finally executes after the step finishes executing.
Tip
All operations of a cleanup statement will be executed regardless of the success or failure of each of them.
A finally statement is similar to a catch statement but will always execute after the try and eventual catch statements finished executing regardless of the success or failure of the test step.
Tip
All operations of a finally statement will be executed regardless of the success or failure of each of them.
While this syntax is simple, it suffers lots of limitations. It doesn't support deletion operations, commands, scripts, and all Chainsaw helpers.
It is also impossible to specify additional configuration per test, step or individual operation (timeouts, additional verifications, etc...), making this approach highly limited.
It also relies a lot on file naming conventions which can be error prone.
Finally, this approach doesn't encourage reusing files across tests and leads to duplication, making maintenance harder.
The manifest below contains an assertion statement in a file called 02-assert.yaml. Chainsaw will associate this manifest with an assert operation in step 02.
The manifest below contains an error statement in a file called 03-errors.yaml. Chainsaw will associate this manifest with an error operation in step 03.
This test will first create a config map, then assert the content of the config map contains the foo: bar data, and then verify that the config map does not contain the lorem: ipsum data.
For such a simple test, the conventional approach works reasonably well but will quickly become limited when the test scenarios get more complex.
Look at the explicit approach for a lot more flexible solution.
The explicit is a bit more verbose than the conventional one but offers far more flexibility and features:
It does not rely on file naming conventions for operations ordering
It encourages file reuse across tests, reducing duplication and maintenance
It offers the flexibility to provide additional configurations like timeouts, complex logic, etc...
It supports all operations without restrictions
"},{"location":"test/explicit/#the-test-resource","title":"The Test resource","text":"
A Test resource, like any other Kubernetes resource, has an apiVersion, kind and metadata section.
It also comes with a spec section used to declaratively represent the test logic, steps and operations, as well as other configuration elements belonging to the test being defined.
Reference documentation
The full structure of the Test resource is documented here.
The Test below illustrates a simple test. Chainsaw will load the Test and steps defined in its spec section.
It's worth noting that:
The test defines its own timeouts
It also states that this test should not be executed in parallel with other tests
It has multiple steps, most of them reference files that can be used in other tests if needed
It uses an arbitrary shell script
apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n name: example\nspec:\n # state that this test should not be executed in parallel with other tests\n concurrent: false\n # timeouts for this specific test\n timeouts:\n apply: 10s\n assert: 10s\n error: 10s\n steps:\n # step 1\n # apply a configmap to the cluster\n # the path to the configmap is relative to the folder\n # containing the test, hence allow reusing manifests\n # across multiple tests\n - try:\n - apply:\n file: ../resources/configmap.yaml\n # step 2\n # execute assert statements against existing resources\n # in the cluster\n - try:\n - assert:\n file: ../resources/configmap-assert.yaml\n # step 3\n # execute error statements against existing resources\n # in the cluster\n - try:\n - error:\n file: ../resources/configmap-error.yaml\n # step 4\n # execute an arbitrary shell script\n - try:\n - script:\n content: echo \"goodbye\"\n
At the end of the test, Chainsaw cleans up resources it created during the test, in the opposite order of creation.
By default, when a step fails, Chainsaw stops the execution and the remaining steps are not executed. The cleanup process starts at the moment the test stops executing.
Tip
Note that when a failure happens during cleanup, the test is marked as failed and Chainsaw continues executing cleanup for the remaining steps.
sequenceDiagram\n autonumber\n participant T as Test\n participant S1 as Step 1\n participant S2 as Step 2\n participant S3 as Step 3\n\n T ->> S1: execute\n S1 ->> S2: execute (fail)\n\n Note left of S3: Step 3 is NOT executed\n\n S2 -->> S1: cleanup\n S1 -->> T: cleanup
Test starts by executing Step 1
Step 1 terminates -> Step 2 starts executing
Step 2 fails -> Cleanup for Step 2 starts
Cleanup for Step 2 terminates -> Cleanup for Step 1 is executed
