Skip to content

Implement Evaluator Versions #1679

Implement Evaluator Versions

Implement Evaluator Versions #1679

Triggered via pull request January 8, 2026 00:25
Status Failure
Total duration 10m 29s
Artifacts

ci.yml

on: pull_request
Lint & Type Check
1m 14s
Lint & Type Check
Batch Evaluation Tests
1m 11s
Batch Evaluation Tests
MCP End-to-End Tests
46s
MCP End-to-End Tests
Matrix: test-core
Fit to window
Zoom out
Zoom in

Annotations

17 errors and 2 warnings
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L143
Variable not allowed in type expression (reportInvalidTypeForm)
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L115
Argument of type "None" cannot be assigned to parameter "logger" of type "DatasetLogger" in function "__init__"   "None" is not assignable to "DatasetLogger" (reportArgumentType)
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L74
Argument to class must be a base class (reportGeneralTypeIssues)
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L54
Variable not allowed in type expression (reportInvalidTypeForm)
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L36
Variable not allowed in type expression (reportInvalidTypeForm)
Lint & Type Check: eval_protocol/integrations/tinker_cookbook.py#L32
Argument to class must be a base class (reportGeneralTypeIssues)
Lint & Type Check: eval_protocol/evaluation.py#L350
Cannot access attribute "evaluator_versions" for class "Fireworks"   Attribute "evaluator_versions" is unknown (reportAttributeAccessIssue)
Lint & Type Check: eval_protocol/evaluation.py#L268
Cannot access attribute "evaluator_versions" for class "Fireworks"   Attribute "evaluator_versions" is unknown (reportAttributeAccessIssue)
Lint & Type Check: eval_protocol/evaluation.py#L246
Cannot access attribute "evaluator_versions" for class "Fireworks"   Attribute "evaluator_versions" is unknown (reportAttributeAccessIssue)
Lint & Type Check: eval_protocol/evaluation.py#L7
"EvaluatorVersionParam" is unknown import symbol (reportAttributeAccessIssue)
Core Tests (Python 3.11, Shard 3/4)
Process completed with exit code 1.
Core Tests (Python 3.12, Shard 3/4)
Process completed with exit code 1.
Core Tests (Python 3.12, Shard 4/4)
Process completed with exit code 1.
Core Tests (Python 3.10, Shard 3/4)
Process completed with exit code 1.
Core Tests (Python 3.10, Shard 4/4)
Process completed with exit code 1.
Core Tests (Python 3.11, Shard 2/4)
Process completed with exit code 1.
Core Tests (Python 3.11, Shard 4/4)
Process completed with exit code 1.
Lint & Type Check: eval_protocol/mcp/__init__.py#L49
Operation on "__all__" is not supported, so exported symbol list may be incorrect (reportUnsupportedDunderAll)
Lint & Type Check: eval_protocol/mcp/__init__.py#L49
Operation on "__all__" is not supported, so exported symbol list may be incorrect (reportUnsupportedDunderAll)