Skip to content

Ensure LangSmith traces include tool runs#184

Merged
benjibc merged 1 commit intomainfrom
codex/clean-up-langsmith-example-and-consolidate
Sep 22, 2025
Merged

Ensure LangSmith traces include tool runs#184
benjibc merged 1 commit intomainfrom
codex/clean-up-langsmith-example-and-consolidate

Conversation

@benjibc
Copy link
Copy Markdown
Contributor

@benjibc benjibc commented Sep 17, 2025

Summary

  • instrument the Chinook LangSmith trace generator with RunTree so traces include structured message events, tool-call child runs, and run metadata for evaluation reuse
  • add assertions to the LangSmith evaluation test to guarantee fetched traces expose tool schemas and assistant tool calls before scoring
  • verified the newly generated traces appear in LangSmith with tool metadata and tool-call child runs for inspection

Testing

  • pytest tests/chinook/langsmith/generate_traces.py -q
  • pytest tests/chinook/langsmith/test_langsmith_chinook.py -q
  • pre-commit run --all-files

Logging example:
image
image


https://chatgpt.com/codex/tasks/task_e_68ca61db52608333ad575ab7512012df

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

@xzrderek
Copy link
Copy Markdown
Contributor

should i update langfuse / braintrust examples to work for chinook like this? i haven't updated those

@benjibc benjibc force-pushed the codex/clean-up-langsmith-example-and-consolidate branch from 4047083 to 80b39ad Compare September 22, 2025 03:34
@benjibc benjibc merged commit 9196f83 into main Sep 22, 2025
7 checks passed
@benjibc benjibc deleted the codex/clean-up-langsmith-example-and-consolidate branch September 22, 2025 03:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants