Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update parallel-execution.md with clarifications on Feature behavior with method-level parallelism #364

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/execution/parallel-execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,12 @@ When using Reqnroll we can consider the parallel scheduling on the level of scen
### Execution Behavior

* `[BeforeTestRun]` and `[AfterTestRun]` hooks (events) are executed only once on the first thread that initializes the framework. Executing tests in the other threads is blocked until the hooks have been fully executed on the first thread.
* As a general guideline, **we do not suggest using `[BeforeFeature]` and `[AfterFeature]` hooks and the `FeatureContext` when running the tests parallel**, because in that case it is not guaranteed that these hooks will be executed only once and there will be only one instance of `FeatureContext` per feature. The lifetime of the `FeatureContext` (that starts and finishes by invoking the `[BeforeFeature]` and `[AfterFeature]` hooks) is the consecutive execution of scenarios of a feature on the same parallel execution worker thread. In case of running the scenarios parallel, the scenarios of a feature might be distributed to multiple workers and therefore might have their own dedicated `FeatureContext`. Because of this behavior the `FeatureContext` is never shared between parallel threads so it does not have to be handled in a thread-safe way. If you wish to have a singleton `FeatureContext` and `[BeforeFeature]` and `[AfterFeature]` hook execution, scenarios in a feature must be executed on the **same thread**.
* As a general guideline, **we do not suggest using `[BeforeFeature]` and `[AfterFeature]` hooks and the `FeatureContext` when running the tests with method-level parallelism**, because in that case it is not guaranteed that these hooks will be executed only once per-feature and that there will be only one instance of `FeatureContext` per feature. (Class-level parallelism, which is the default in most modern test runners, _should_ operate as expected and only create one `FeatureContext` per feature and only call feature hooks once.) The lifetime of the `FeatureContext` (that starts and finishes by invoking the `[BeforeFeature]` and `[AfterFeature]` hooks) is the consecutive execution of scenarios of a feature on the same parallel execution worker thread. So in case of running the scenarios with full method-level parallelism, the scenarios of a feature might be distributed to multiple workers and therefore might have their own dedicated `FeatureContext`. Additionally, it's also possible for _some_ scenarios of a feature to get the same `FeatureContext`, if they are executed on the same thread as a previous "sibling". It all depends on how the test runner distributes scenarios amongst the worker threads - which is not easily predictable or controllable. Because of this behavior, Reqnroll does not attempt to share the `FeatureContext` between parallel threads so it does not have to handle them in a thread-safe way. If you wish to have a truly singleton `FeatureContext` and `[BeforeFeature]` and `[AfterFeature]` hook execution, you must use either class-level parallelism or disable parallelism entirely so scenarios of a feature are all executed on the **same thread**.
* All that said, if you still want to use method-level parallelism for your test suite, **the following things will be true**:
* The `FeatureContext` and feature-level DI container will remain consistent **per feature, per test thread**. This means in anything you register in the Feature container will be resolvable in the `[AfterFeature]` **per test thread**. The same applies to dictionary values assigned to the `FeatureContext`.
* The same `FeatureContext` and feature-level DI container will be provided to all scenarios of a feature that get scheduled **on the same test thread**.
* A given `[BeforeFeature]` or `[AfterFeature]` will only be executed once **per test thread** that runs a scenario of a feature.
* Types you register in the feature-level DI container that implement `IDisposable` will still be disposed **per feature, per test thread**. (Keep this in mind if you try to work around this parallelism behavior to regain singleton-like behavior. E.g. by using static instances, `Lazy<>`, thread-safe collections, etc.)
* Scenarios and their related hooks (Before/After scenario, scenario block, step) are isolated in the different threads during execution and do not block each other. Each thread has a separate (and isolated) `ScenarioContext`.
* The test trace listener (that outputs the scenario execution trace to the console by default) is invoked asynchronously from the multiple threads and the trace messages are queued and passed to the listener in serialized form. If the test trace listener implements `Reqnroll.Tracing.IThreadSafeTraceListener`, the messages are sent directly from the threads.

Expand Down