fairness-and-validity

Fairness, validity, and consistency

Fundamental principles

These are the fundamental principles that underly how we look at the question of “fairness” in Code4rena.

1. Code4rena aims to be a fair and impartial system.
2. Where the system is insufficient or vague, we depend on the judgment of fair and impartial individuals.
3. When we depend too heavily on the judgment of individuals, we work to improve the system long-term in iterative and sustainable ways.
4. Because we are working every day within the constraints of the systems we have, we aim to be patient with the time and consideration that improvement takes and gracious toward the individuals tasked by the system with making difficult decisions.

It may be worth reading this longer piece on the topic of how our system has evolved.

Expectations of participants

• Sponsors should be able to trust that Code4rena as a system is working to help them secure their code and that their funds are a good investment toward that end.
• Wardens should be able to have clear rule expectations of contests they contribute to—as clear as possible within the constraints.
• Judges should be impartial and free to act independently to do what they see best in a given contest within the guidelines they are provided.

Role of hired staff (Code4 Corporation)

The role of staff is regulatory, supportive, and administrative:

• Code4 is a neutral party in contests dedicated to serving and collaborating with all sides of the market in driving the success of Code4rena as a platform.
• Code4 is responsible for improving documentation, process, and tools in support of the goals and expectations of each of the parties involved in Code4rena, providing information, context, and guidance to sponsors, judges, and wardens within the rules.
• Code4 has no role in determining the outcomes of findings and does not put its hand on the scale in individual contests.
• Code4 does have a role to provide sponsors, judges, and wardens with historical context on the intent of rules so that those rules can be applied appropriately when ambiguity is present.

What constitutes a ‘valid’ report’?

The validity of an audit report submission is not based on whether it is ‘true’ or not. A report may contain a finding which is factually 'true' (the most literal interpretation of 'valid'), but if it does not add value or if it is not presented in such a way that adds value to a sponsor, it may be deemed invalid by a judge.

This may seem harsh and exclusive, but it is essential to consider that Code4rena runs audit contests, not gotcha-hunts, and Code4rena offers guaranteed payout for valid submissions. This means that wardens are providing a service to sponsors and the product of those services should meet what judges feel is a minimum standard in order to be deemed of value.

Auditing is serious, disciplined work that should provide high value consultative expertise to the people paying for the work.

In that light, judges are right to have high standards. Some judges have always had higher standards than others, and some judges have applied higher standards in later contests than they did in earlier ones.

While this may be seen as ‘inconsistent’, it is also true that standards within a specific contest will always be informed by the overall quality of a contest’s submissions, and that the standard in a judge’s mind is always going to be evolving based on the aggregate quality of submissions that judge has been exposed to and the decisions other judges have made.

The correct assessment when this happens is not that a judge is being inconsistent, it is that they have objectively observed that the quality of competition has increased, and that observation shapes their view of the whole set of submissions; they are consistent in valuing submissions in the context of each other, which is a central way that performance in a competition is measured.

Per the Autumn 2023 Supreme Court verdicts, it is within the judge’s discretion to invalidate all of a warden’s findings in a particular contest in the case of repeated low-quality submissions. For more details on this, please see "Good citizenship" in the Submission guidelines.

If you disagree with a judge's decision

If you disagree with a decision, and you do not have the C4 SR role (formerly +backstage role), there's nothing further that can be done or changed; the judge's decisions are final.

However, if the concern regarding judging is focused on a matter of inconsistency or process or lack of clarity in the rules, you are encouraged to review the issues in https://github.com/code-423n4/org/issues and:

1. See if one of the problems described there matches the type of issue you have experienced. If so, add a purely fact-based comment with additional information and another point of evidence of it being a challenge.

2. See if any of the suggestions described there would be useful to improving the case you have in mind. If so, feel free to add your thoughts in support.

3. IF a relevant type of issue is not already addressed there which doesn't represent the categorical concern you have, you can feel free to open an issue.

The purpose of issues in that repo is not to post grievances about specific issues but about to identify places where the process can be improved and ways we can improve it.