Skip to content

Latest commit

 

History

History
55 lines (32 loc) · 4.94 KB

how-to-judge-a-contest.md

File metadata and controls

55 lines (32 loc) · 4.94 KB
Raw

How to judge an audit

Timeline

Ideally we would like audits to be judged in 48 hours after handoff.

We ask that you try to complete the judging process quickly so that we can distribute awards to wardens promptly. If you need more time, please communicate that to C4 as soon as possible.

Here’s how the process works leading up to judging

C4 kicks off the code competition and establishes a private repo to receive incoming issues. Typically, most findings come in on the last day of the audit. When the audit ends, you will get access to both the validation repo and the findings repo. A group of Validators will triage submissions from all wardens below a set accuracy threshold, and submissions they deem satisfactory will be added to the findings repo.

Sponsors are then invited to review the findings, comment, and provide feedback on issues within the findings repo. Sponsor input is non-binding, and do note that sponsors are heavily biased against having a report that includes very many vulnerabilities. Focus your work as a judge on protecting users and providing feedback to wardens.

C4 staff will hand off to the judge when the sponsor review phase concludes. That said, judges may begin work anytime after the submission period ends.

Before you get started

Read the Judging Criteria, Submission Policy, and review the audit readme as provided by the sponsor.

You may also be interested in browsing past audits, and reviewing open issues in the Rulebook repo, in order to see how other judges have handled issues.

Reviewing submissions

When your judge application is approved, C4 staff will contact you to invite you to our Github organization and provide you with technical documentation on our judging tools. Those documents also include all information regarding de-duping, grading QA/Gas and other judging tasks.

Notes on judging

  • Review the Judging criteria.
  • Consider the sponsor’s feedback, but keep in mind that it’s not always going to be objective.
  • Any submissions that do not apply specifically to the functionality of the smart contract logic itself should be considered QA.
  • When weighing in on severity or validity of an issue, leave a comment describing your justification for any changes you make to the warden's assessment of severity.
  • When necessary, cross reference the submission with the codebase to validate the legitimacy of the proposed submission.
  • Unless there is something uniquely novel created by combining vectors, most submissions regarding vulnerabilities that are inherent to a particular system or the Ethereum network as a whole should be considered QA. Examples of such vulnerabilities include front running, sandwich attacks, and MEV. In such events, leave a comment on the issue:

“Sandwich attacks are inherent to AMMs, so this isn’t a unique issue presented by the MarginSwap implementation. With this in mind, I’m downgrading the risk from a proposed medium severity to QA.”

One important caveat to all of the above: unless otherwise specified by the audit sponsor or intended to be handled by the code. For example, flash loans are generally unavoidable, but since MarginSwap had a safeguard against them, we considered these findings relevant in their audit.

Dealing with spam / repeated low-quality submissions

Please see the Good Citizen policy.

Note: this policy was instated after this proposal from the Autumn 2023 Supreme Court session.

Discussing issues with the sponsor

Ultimately the judge has the final word, but we want your decisions to be well-informed. In a typical C4 audit, there will be a few issues that benefit from discussion with the sponsor; the judge may find that their understanding of the system is incomplete and you need to ask for clarification, or where there is room for misunderstanding. Don’t hesitate to connect directly with the sponsor, either in the Github comments (where you can tag them in if needed), or via Discord.

If you have questions

Do not hesitate to post in the #judges Discord channel, or reach out to the C4 Civics Administrators with questions as you're working on judging. Any questions or feedback you can add to this documentation, or comments/questions on items above are highly welcome and essential for us improving our process. Thank you! 🙏

When you’re done reviewing

Ping a C4 Civics Administrator and let us know you’re ready to hand off the results for post-judging QA and then award distribution.