Stabilize high priority usage tracking tests without LangChain#7785
Stabilize high priority usage tracking tests without LangChain#7785tianmind-studio wants to merge 2 commits into
Conversation
Greptile SummaryThis PR fixes test collection failures on Windows (and other environments without LangChain installed) by stubbing
Confidence Score: 3/5Safe to merge in environments where LangChain is not installed, but the stub setup can silently replace real LangChain classes in the same pytest session when the package is present. The stub attribute assignments (lines 72–75) are unconditional — they run whether backend/tests/unit/test_high_priority_usage_tracking.py — the LangChain stub setup block (lines 34–75) needs a guard to skip attribute assignments when the real package is already loaded. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["pytest collects test file"] --> B{"langchain_core\nin sys.modules?"}
B -- "No (target env)" --> C["_stub_module creates\nfresh ModuleType objects"]
C --> D["Set stub classes on\nnewly-created modules\n(safe)"]
B -- "Yes (LangChain installed)" --> E["_stub_module returns\nreal module objects"]
E --> F["Stub classes assigned\nonto real modules\n⚠️ overwrites real classes"]
D --> G["usage_tracker imports\nBaseCallbackHandler, LLMResult\nfrom stub modules"]
F --> G
G --> H["Test body runs:\ncontext-capture functions\nreplace mock_llm_mini.invoke"]
H --> I["Production function called\n(suggest_goal, get_goal_advice, ...)"]
I --> J["track_usage context manager\nsets _usage_context ContextVar"]
J --> K["LLM mock invoke called;\ntest asserts captured context"]
Reviews (1): Last reviewed commit: "Stabilize high priority usage tracking t..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| langchain_core_mod = _stub_module("langchain_core") | ||
| if not hasattr(langchain_core_mod, '__path__'): | ||
| langchain_core_mod.__path__ = [] | ||
|
|
||
| callbacks_mod = _stub_module("langchain_core.callbacks") | ||
| outputs_mod = _stub_module("langchain_core.outputs") | ||
| output_parsers_mod = _stub_module("langchain_core.output_parsers") | ||
| prompts_mod = _stub_module("langchain_core.prompts") | ||
|
|
||
|
|
||
| class BaseCallbackHandler: | ||
| pass | ||
|
|
||
|
|
||
| class LLMResult: | ||
| pass | ||
|
|
||
|
|
||
| class PydanticOutputParser: | ||
| def __init__(self, pydantic_object): | ||
| self.pydantic_object = pydantic_object | ||
|
|
||
| def get_format_instructions(self): | ||
| return "" | ||
|
|
||
| def parse(self, text): | ||
| return self.pydantic_object(**json.loads(text)) | ||
|
|
||
|
|
||
| class ChatPromptTemplate: | ||
| @classmethod | ||
| def from_messages(cls, _messages): | ||
| return cls() | ||
|
|
||
| def __or__(self, other): | ||
| return other | ||
|
|
||
|
|
||
| callbacks_mod.BaseCallbackHandler = BaseCallbackHandler | ||
| outputs_mod.LLMResult = LLMResult | ||
| output_parsers_mod.PydanticOutputParser = PydanticOutputParser | ||
| prompts_mod.ChatPromptTemplate = ChatPromptTemplate |
There was a problem hiding this comment.
_stub_module returns the real langchain_core.callbacks module when langchain_core is already in sys.modules (i.e., the package is installed). The subsequent attribute assignments on lines 72–75 then replace the real BaseCallbackHandler, LLMResult, PydanticOutputParser, and ChatPromptTemplate with stub implementations on the live module objects. Any other test that later does from langchain_core.callbacks import BaseCallbackHandler in the same pytest session would receive the stub class, silently producing wrong behavior or unexpected test failures.
A minimal guard avoids this: capture whether the stub was freshly created before assigning attributes.
There was a problem hiding this comment.
Addressed in f483ea6b2: optional LangChain/pytz modules now go through _optional_stub_module, which imports real modules when available and only injects stub attributes when the module was newly created as a stub. Added a regression test proving already-loaded modules are preserved. Local validation: 21 passed.
| def _get_llm_stub(name, **_kwargs): | ||
| if name in {'goals', 'knowledge_graph', 'daily_summary_simple'}: | ||
| return mock_llm_mini | ||
| if name in {'goals_advice', 'notifications'}: | ||
| return mock_llm_medium | ||
| if name == 'daily_summary': | ||
| return mock_llm_medium_experiment | ||
| return mock_llm_mini | ||
|
|
||
|
|
||
| clients_mod.get_llm = _get_llm_stub |
There was a problem hiding this comment.
_get_llm_stub masks uncovered get_llm call sites
The final return mock_llm_mini fallback means any new get_llm(name) call whose name is not enumerated in the stub (e.g., after a future feature is added) silently gets mock_llm_mini instead of failing loudly. The context-capture tests then pass even though the wrong mock is in use, so tracking regressions for new features would go undetected. Raising ValueError for unknown names (or at minimum warnings.warn) would make the mismatch visible immediately.
There was a problem hiding this comment.
Addressed in f483ea6b2: _get_llm_stub now raises ValueError for unknown feature names instead of silently falling back to mock_llm_mini, and there is a regression test for that failure path. Local validation: 21 passed.
1 participant
Summary
langchain_coreandpytzsurfaces needed bytest_high_priority_usage_tracking.pyso the source-level usage tracking tests run in lightweight backend test environments.database.users.get_user_language_preferenceandutils.llm.clients.get_llmtest stubs expected by imported modules.get_llmtest stub fail loudly on unknown feature names.Windows reproduction
Before this change on this Windows test environment:
python -m pytest tests\unit\test_high_priority_usage_tracking.py -q-> collection errorModuleNotFoundError: No module named 'langchain_core'while importingutils.llm.usage_trackerTesting
python -m pytest tests\unit\test_high_priority_usage_tracking.py -q->21 passedpython -m black --line-length 120 --skip-string-normalization tests\unit\test_high_priority_usage_tracking.py --checkpython -m py_compile tests\unit\test_high_priority_usage_tracking.pygit diff --check