Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces LangGraph support to the eval_protocol library by adding a new rollout processor and example implementations. The changes enable evaluation of LangGraph-based applications through a dedicated processor that handles conversion between eval_protocol and LangChain message formats.
- Adds
LangGraphRolloutProcessorfor processing LangGraph-based evaluations - Implements database query example using Chinook database with LangGraph
- Provides simple chat example with Fireworks model integration
Reviewed Changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| eval_protocol/pytest/default_langchain_rollout_processor.py | Complete rewrite to support LangGraph with message conversion and error handling |
| eval_protocol/adapters/langchain.py | Simplified message serialization and added EP to LC conversion function |
| tests/pytest/test_langgraph_processor.py | Unit tests for the new LangGraph processor functionality |
| tests/chinook/langgraph/test_langgraph_chinook.py | Integration test using Chinook database with LangGraph |
| tests/chinook/langgraph/graph.py | LangGraph implementation for database queries |
| examples/langgraph/test_langgraph_rollout.py | Example evaluation test using LangGraph |
| examples/langgraph/simple_graph.py | Simple LangGraph implementation with Fireworks integration |
| examples/langgraph/data/simple_prompts.jsonl | Test data for LangGraph examples |
| requirements-dev.txt | Added LangGraph and LangChain dependencies |
| eval_protocol/pytest/handle_persist_flow.py | Fixed dataset name generation for Fireworks API compatibility |
| eval_protocol/adapters/bigquery.py | Consolidated exception handling and removed unused import |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| @pytest.mark.skipif(os.getenv("FIREWORKS_API_KEY") in (None, ""), reason="FIREWORKS_API_KEY not set") | ||
| @evaluation_test( | ||
| input_messages=[[[Message(role="user", content="What is the total number of tracks in the database?")]]], | ||
| completion_params=[{"model": "accounts/fireworks/models/kimi-k2-instruct", "provider": "fireworks"}], |
There was a problem hiding this comment.
"provider" is a pydantic thing, is it necessary here?
There was a problem hiding this comment.
model,
model_provider=model_provider,
temperature=temperature,
reasoning_effort=reasoning_effort,
)
```, it is a langgraph thing.
| rollout_processor=LangGraphRolloutProcessor( | ||
| graph_factory=lambda _: build_graph(), | ||
| build_graph_kwargs=build_graph_kwargs, | ||
| input_key="messages", | ||
| output_key="messages", |
There was a problem hiding this comment.
reading up on langgraph to understand this—its confusing at first
| graph_factory=lambda _: build_graph(), | ||
| build_graph_kwargs=build_graph_kwargs, |
There was a problem hiding this comment.
do these need to be two separate things? why not just have an agent factory:
Callable[[RolloutProcessorConfig], Graph]
|
i think we need to track |
…add langgraph/langgraph_tools extras; relax extras to >= versions
Uh oh!
There was an error while loading. Please reload this page.