Skip to content

LangGraph simple example#167

Merged
benjibc merged 5 commits intomainfrom
langgraph_simple_example
Sep 10, 2025
Merged

LangGraph simple example#167
benjibc merged 5 commits intomainfrom
langgraph_simple_example

Conversation

@benjibc
Copy link
Copy Markdown
Contributor

@benjibc benjibc commented Sep 9, 2025

  • Simple database example
  • Chat example

@benjibc benjibc requested review from Copilot, dphuang2 and xzrderek and removed request for xzrderek September 9, 2025 20:48
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces LangGraph support to the eval_protocol library by adding a new rollout processor and example implementations. The changes enable evaluation of LangGraph-based applications through a dedicated processor that handles conversion between eval_protocol and LangChain message formats.

  • Adds LangGraphRolloutProcessor for processing LangGraph-based evaluations
  • Implements database query example using Chinook database with LangGraph
  • Provides simple chat example with Fireworks model integration

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
eval_protocol/pytest/default_langchain_rollout_processor.py Complete rewrite to support LangGraph with message conversion and error handling
eval_protocol/adapters/langchain.py Simplified message serialization and added EP to LC conversion function
tests/pytest/test_langgraph_processor.py Unit tests for the new LangGraph processor functionality
tests/chinook/langgraph/test_langgraph_chinook.py Integration test using Chinook database with LangGraph
tests/chinook/langgraph/graph.py LangGraph implementation for database queries
examples/langgraph/test_langgraph_rollout.py Example evaluation test using LangGraph
examples/langgraph/simple_graph.py Simple LangGraph implementation with Fireworks integration
examples/langgraph/data/simple_prompts.jsonl Test data for LangGraph examples
requirements-dev.txt Added LangGraph and LangChain dependencies
eval_protocol/pytest/handle_persist_flow.py Fixed dataset name generation for Fireworks API compatibility
eval_protocol/adapters/bigquery.py Consolidated exception handling and removed unused import

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@pytest.mark.skipif(os.getenv("FIREWORKS_API_KEY") in (None, ""), reason="FIREWORKS_API_KEY not set")
@evaluation_test(
input_messages=[[[Message(role="user", content="What is the total number of tracks in the database?")]]],
completion_params=[{"model": "accounts/fireworks/models/kimi-k2-instruct", "provider": "fireworks"}],
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"provider" is a pydantic thing, is it necessary here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

        model,
        model_provider=model_provider,
        temperature=temperature,
        reasoning_effort=reasoning_effort,
    )
```, it is a langgraph thing. 

Comment on lines +56 to +60
rollout_processor=LangGraphRolloutProcessor(
graph_factory=lambda _: build_graph(),
build_graph_kwargs=build_graph_kwargs,
input_key="messages",
output_key="messages",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reading up on langgraph to understand this—its confusing at first

Comment on lines +57 to +58
graph_factory=lambda _: build_graph(),
build_graph_kwargs=build_graph_kwargs,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these need to be two separate things? why not just have an agent factory:

Callable[[RolloutProcessorConfig], Graph]

@xzrderek
Copy link
Copy Markdown
Contributor

xzrderek commented Sep 9, 2025

i think we need to track row.tools, i just made similar change for pydantic processor: #168

@benjibc benjibc merged commit 8101180 into main Sep 10, 2025
7 checks passed
@benjibc benjibc deleted the langgraph_simple_example branch September 10, 2025 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants