-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Evals SDK #175
Update Evals SDK #175
Conversation
|
||
|
||
@dataclass | ||
class NamedExpectedToolCall: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copy of what ExpectedToolCall
used to be.
Codecov ReportAll modified and coverable lines are covered by tests ✅ 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding tests!
@@ -2,13 +2,15 @@ | |||
from arcade_code_sandbox.tools.e2b import create_static_matplotlib_chart, run_code | |||
from arcade_code_sandbox.tools.models import E2BSupportedLanguage | |||
|
|||
from arcade.core.catalog import ToolCatalog | |||
from arcade.sdk import ToolCatalog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks 💯
PR Description
This PR renames
ExpectedToolCall
toNamedExpectedToolCall
and then creates a new dataclass calledExpectedToolCall
.ExpectedToolCall
can be passed to theEvalSuite.add_case
andEvalSuite.extend_case
methods.EvalSuite.add_case
andEvalSuite.extend_case
by accepting a list ofExpectedToolCall
as theirexpected_tool_calls
input parameter. This helps create a scaffolding for developers. Previously, the expected type waslist[tuple[Callable, dict[str, Any]]]
, which is still valid for backward compatibility.