ci: enable snapshot testing #5037
Conversation
|
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
agostbiro
had a problem deploying
to
github-action-benchmark
GitHub Actions
Failure
agostbiro
added
no changeset needed
| @@ -89,6 +89,14 @@ async function verify(benchmarkResultPath) { | |||
| )); | |||
|
|
|||
| for (let scenarioName in snapshotResult) { | |||
| // Snapshot testing is unreliable for these scenarios | |||
| if ( | |||
| scenarioName.includes("openzeppelin") || | |||
There was a problem hiding this comment.
Could this be related to #5036 (comment)?
There was a problem hiding this comment.
Thanks good idea, but I don't think so. I updated the openzeppelin scenario to a version in this PR that was collected after Franco's PR to fix the issue in openzeppelin landed: OpenZeppelin/openzeppelin-contracts#4942
There was a problem hiding this comment.
Just an FYI, when I enabled logging for a repo - that suffers from the described problem - and compared the generated logs, the order of calls wasn't deterministic. This was a sign to me that tests were being executed in parallel.
This might be a relatively straightforward way of testing the hypothesis for other repos.
agostbiro
temporarily deployed
to
github-action-benchmark
GitHub Actions
Inactive
This PR enables snapshot testing in the CI by checking that the same requests succeed and fail between benchmark runs.
The
openzeppelin-contractsandneptune-mutual-blue-protocolscenarios are excluded from snapshot testing, because there is some variance in which calls and transactions succeed or fail. A time-limited investigation found that this variance is related to block timestamps. Since it's difficult to know the expected behavior based on the snapshots, as a follow up we added NomicFoundation/edr#240This PR also updates the
openzeppelin-contractsscenario to a version that was collected after some sporadically failing tests were fixed in theopenzeppelin-contractsrepo.