Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: actively raised errors are also considered as negative feedback. #268

Merged
merged 1 commit into from
Sep 18, 2024
Merged

fix: actively raised errors are also considered as negative feedback. #268

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion rdagent/components/coder/factor_coder/prompts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ evaluator_final_decision_v1_system: |-
The implementation final decision is considered in the following logic:
1. If the value and the ground truth value are exactly the same under a small tolerance, the implementation is considered correct.
2. If the value and the ground truth value have a high correlation on ic or rank ic, the implementation is considered correct.
3. If no ground truth value is not provided, the implementation is considered correct if the code execution is successful and the code feedback is align with the scenario and factor description.
3. If no ground truth value is provided, the implementation is considered correct if the code executes successfully (assuming the data provided is correct). Any exceptions, including those actively raised, are considered faults of the code. Additionally, the code feedback must align with the scenario and factor description.

Please response the critic in the json format. Here is an example structure for the JSON output, please strictly follow the format:
{
Expand Down
Loading