Fix pyright type errors

[benjibc]

---

name: Pull Request
about: Propose changes to the codebase
title: "Fix: Address initial Pyright type errors in eval_protocol"
labels: ''
assignees: ''

---

Description

This PR addresses and resolves several initial Pyright type errors within the eval_protocol package, reducing the overall type error count by more than five. The changes focus on improving type accuracy and handling optional values and external imports.

Key changes include:

• eval_protocol/playback_policy.py: Removed openai.types.CompletionUsage dependency, updated the return signature of generate_tool_calls to use Optional[Dict[str, int]] for usage and Optional[str] for correlation ID, and removed a duplicate method definition.
• eval_protocol/utils/static_policy.py: Updated StaticPolicy and RandomPolicy to align with the new PlaybackPolicyBase return signature for generate_tool_calls and made info parameters Optional[Dict[str, Any]].
• eval_protocol/rewards/accuracy_length.py: Improved robustness by coercing response.content to str, handling optional ground_truth values, and adding a cast for accuracy_reward to resolve decorator-related type inference issues.
• eval_protocol/server.py: Added a safe cast for legacy tuple return types from reward functions and introduced type: ignore hints for missing external imports (uvicorn, fastapi, pydantic) to suppress errors related to uninstalled development dependencies.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactoring/Code cleanup
• Build/CI/CD related changes
• Other (please describe):

How Has This Been Tested?

The changes were verified by running the basedpyright type checker:

• Individually on each modified file (eval_protocol/playback_policy.py, eval_protocol/utils/static_policy.py, eval_protocol/rewards/accuracy_length.py, eval_protocol/server.py).
• On the entire eval_protocol package to confirm a reduction in overall type errors.

Test Configuration:

• Toolchain: basedpyright (via pre-commit hook)

Checklist:

• My code follows the style guidelines of this project (ran black ., isort ., flake8 .)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works (type checks)
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

Screenshots (if applicable)

N/A

Additional context

This PR addresses the initial set of type errors as requested. Further work could include:

• Addressing additional Union[str | List[ChatCompletionContentPartTextParam]] string-coercion issues in other reward files (e.g., length.py, math.py, language_consistency.py).
• Optionally ignoring or vendor-gating external import checks in adapter/agent paths where runtime dependencies are not available locally.

---

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}

[Copilot]

Pull Request Overview

This PR addresses and resolves several Pyright type errors within the eval_protocol package to improve type accuracy and handle optional values and external imports. The changes focus on removing problematic import dependencies, updating return type signatures, and improving type handling for optional values.

• Removed dependency on openai.types.CompletionUsage and updated method signatures to use native Python types
• Improved type safety by handling optional values and adding proper type coercion
• Added type ignore comments for external dependencies that may not be available during development

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
eval_protocol/playback_policy.py	Removed OpenAI dependency, updated return signature to use Optional types, removed duplicate method
eval_protocol/utils/static_policy.py	Updated method signatures to align with base class changes and handle optional parameters
eval_protocol/rewards/accuracy_length.py	Added robust string coercion for response content and type casting for accuracy reward
eval_protocol/server.py	Added type ignore comments for external imports and safe casting for legacy tuple returns

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

This import is unused and creates an unnecessary alias. The regular Optional import on line 19 is sufficient for the usage in this file.

Suggested change

from typing import Optional as _Optional

[Copilot]

The type ignore comment suggests uncertainty about the structure. Consider adding a more specific type check or explicit attribute validation before accessing .text to make the code more robust.

Suggested change

	text = " ".join(part.text for part in response.content) # type: ignore[union-attr]
	text = " ".join(
	getattr(part, "text", "") for part in response.content if hasattr(part, "text")
	)

[Copilot]

Casting to Any defeats the purpose of type checking. Consider using a more specific type or ensuring the decorator preserves the original function signature instead of using cast(Any, ...).

Suggested change

	accuracy_eval_result = cast(Any, accuracy_reward)(
	accuracy_eval_result = accuracy_reward(