Python: Fix: AgentTelemetryLayer captures context-provider-extended instructions in gen_ai.system_instructions

[benke520]

Summary

Fixes #5291

AgentTelemetryLayer._trace_agent_invocation() records gen_ai.system_instructions from merged_options before calling execute(), but context providers run inside execute() and extend instructions via context.extend_instructions(). This means the invoke_agent span only captures the base instructions, not the dynamically extended ones that are actually sent to the LLM.

Changes

observability.py

• Added AGENT_MERGED_INSTRUCTIONS ContextVar to carry the final merged instructions from the agent layer to the telemetry layer
• Non-streaming path: After await super_run(), reads the ContextVar and re-sets gen_ai.system_instructions on the span with the full instructions (base + provider-extended)
• Streaming path: Same re-capture in _finalize_stream() after the stream is consumed
• Properly resets the ContextVar token in both paths' finally blocks

_agents.py

• In _prepare_session_and_messages(), sets AGENT_MERGED_INSTRUCTIONS after context providers have run and their instructions are merged into chat_options

How it works

Before (broken):
  AgentTelemetryLayer:
    1. Create span
    2. Record gen_ai.system_instructions  <- only base instructions
    3. execute() -> providers extend instructions -> LLM call
    4. End span

After (fixed):
  AgentTelemetryLayer:
    1. Create span
    2. Record gen_ai.system_instructions  <- base instructions (initial capture)
    3. execute() -> providers extend instructions -> set ContextVar -> LLM call
    4. Read ContextVar -> re-set gen_ai.system_instructions  <- full instructions
    5. End span

Testing

• All existing tests pass (124 observability, 9 telemetry, 103 agent tests)
• The fix uses the same ContextVar pattern already established for INNER_ACCUMULATED_USAGE and INNER_RESPONSE_TELEMETRY_CAPTURED_FIELDS

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes telemetry capture so gen_ai.system_instructions reflects context-provider-extended instructions (not just the base instructions) for both streaming and non-streaming agent invocations.

Changes:

• Introduces a new ContextVar to carry final merged instructions from agent execution to telemetry.
• Re-captures gen_ai.system_instructions on the invoke_agent span after execution/stream consumption.
• Publishes merged instructions from _prepare_session_and_messages() after context providers have contributed.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
python/packages/core/agent_framework/observability.py	Adds AGENT_MERGED_INSTRUCTIONS ContextVar and updates telemetry to re-capture final system instructions post-execution (streaming + non-streaming).
python/packages/core/agent_framework/_agents.py	Sets AGENT_MERGED_INSTRUCTIONS after providers have extended and merged instructions into chat_options.

[moonbox3]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
_agents.py	415	52	87%	461, 470, 525, 1020, 1065, 1138–1142, 1202, 1230, 1267, 1288, 1308–1309, 1314, 1361, 1403, 1425, 1427, 1440, 1446, 1498, 1500, 1509–1514, 1519, 1521, 1527–1528, 1535, 1537–1538, 1546–1547, 1550–1552, 1562–1567, 1571, 1576, 1578
observability.py	775	92	88%	126–129, 403, 405–406, 409, 412, 415–416, 421–422, 428–429, 435–436, 443, 445–447, 450–452, 457–458, 464–465, 471–472, 479, 636–637, 765, 769–771, 773, 777–778, 782, 820, 822, 833–835, 837–839, 843, 851, 975–976, 1138, 1381–1382, 1480–1485, 1492–1495, 1499–1507, 1514, 1580–1584, 1604, 1647–1648, 1661–1663, 1798, 1890, 2087, 2305, 2307
TOTAL	27713	3206	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
5611	20 💤	0 ❌	0 🔥	1m 35s ⏱️