feat: add configurable token cache for customer and lob flows

[vitalykumov]

Description

Add in-process token cache for both customer and LoB agent flows. Previously every list_mcp_tools() / call_mcp_tool() call fetched fresh IAS token via mTLS - unnecessary latency in agentic loops.

Changes:

• _token_cache.py (new): _TokenCache - TTL + LRU eviction for system tokens (key: client_id) and user tokens (key: sha256(user_jwt+"|"+client_id)[:16]). Expiry is from expires_in, id_token exp claim, or fallback TTL.
• _customer.py: get_system_token_mtls / exchange_user_token to consult/populate cache. 401 response from MCP server → invalidate stale token + retry once
• _lob.py: get_system_auth / get_user_auth now check cache before calling BTP Destination Service. get_mcp_tools_lob reuses system auth across all fragments (single fetch per discovery call). call_mcp_tool_lob caches user token per (fragment, tenant) scope. Both functions retry once on 401 with cache invalidation - same pattern as customer flow
• agw_client.py: AgentGatewayClient passes _TokenCache to both customer and LoB functions
• config.py: 4 new ClientConfig fields added - token_expiry_buffer_seconds (60 s), max_system_token_cache_size (10 s), max_user_token_cache_size (10 s), fallback_token_ttl_seconds (300 s)

Type of Change

Please check the relevant option:

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Code refactoring
• Dependency update

How to Test

Describe how reviewers can test your changes:

1. Programmatically:

   async def verify_token_caching(agw_client) -> bool:
       """Verify system token is cached across list_mcp_tools calls.
   
       Patches _request_token_mtls to count real HTTP token requests.
       First call must fetch at least one token; second call must reuse cache.
       Returns True if caching works correctly.
       """
       original = _customer_mod._request_token_mtls
       call_count = 0
   
       def counting(*args, **kwargs):
           nonlocal call_count
           call_count += 1
           return original(*args, **kwargs)
   
       _customer_mod._request_token_mtls = counting
       try:
           await agw_client.list_mcp_tools()
           after_first = call_count
   
           await agw_client.list_mcp_tools()
           after_second = call_count
   
           agw_client.clear_token_cache()
           await agw_client.list_mcp_tools()
           after_clear = call_count
       finally:
           _customer_mod._request_token_mtls = original
   
       cache_hit = after_second == after_first
       invalidate_works = after_clear > after_second
   
       print("\nToken Caching")
       print("-" * 40)
       print(f"  1st call  token requests : {after_first}")
       print(f"  2nd call  token requests : {after_second - after_first}  {'✓ cache hit' if cache_hit else '✗ expected 0'}")
       print(f"  post-clear token requests: {after_clear - after_second}  {'✓ re-fetched' if invalidate_works else '✗ expected ≥1'}")
   
       return cache_hit and invalidate_works
   ...
   async def main():
       agw_client = create_client()
   
       ok = await verify_token_caching(agw_client)
       if not ok:
           print("\nWARNING: token caching check failed")
   ...

2. Unit tests:

   pytest tests/agentgateway/unit/test_token_cache.py tests/agentgateway/unit/test_customer.py tests/agentgateway/unit/test_agw_client.py

Checklist

Before submitting your PR, please review and check the following:

• I have read the Contributing Guidelines
• I have verified that my changes solve the issue
• I have added/updated automated tests to cover my changes
• All tests pass locally
• I have verified that my code follows the Code Guidelines
• I have updated documentation (if applicable)
• I have added type hints for all public APIs
• My code does not contain sensitive information (credentials, tokens, etc.)
• I have followed Conventional Commits for commit messages

Breaking Changes

None. Cache internal to AgentGatewayClient - existing create_client() calls get caching automatically. ClientConfig new fields all have defaults.

Additional Notes

Thread safety: GIL makes individual OrderedDict ops atomic, but check-then-set is not. Concurrent coroutines on same key may both miss and both fetch - redundant requests, not corruption. Acceptable for agentic loop use case.

401 retry: Both get_mcp_tools_customer and call_mcp_tool_customer invalidate + retry once on 401, handling server-side revocation before token expiry.

[cla-assistant]

All committers have signed the CLA.

[prashantrakheja]

changes look good, some minor feedback, in particular i would like to remove usage of app_tid from token caching because its an optional parameter and likely to be removed going forward..

[prashantrakheja]

i am not sure if app_tid would be needed for fetching the tokens, is there any other way to do this?

[vitalykumov]

switched to use client_id

[prashantrakheja]

do u feel the exception block is unusually big? can we put this stuff inside a method?

[vitalykumov]

Updated logic here - request system_token when needed. Added helper closure to refetch token

[prashantrakheja]

i feel we should not swallow the exception here, if there is a failure we should fail-fast, what's ur take?

[vitalykumov]

I'll extract logic to a method, good catch.
Regarding exception - it basically follows what was implemented before - log with logger.exception and continue

[cassiofariasmachado]

@vitalykumov, could you take a look at the conflicts?

[vitalykumov]

@vitalykumov, could you take a look at the conflicts?

I'll try but seems like it would require to change quite a lot...

Quick look:
Token fetching is separate now, doesn't seem like there's any reason to keep caching in customer/lob flows. I'll need to re-write pretty much everything. I'll continue on Monday

[vitalykumov]

closing in favor of #142