Repository navigation
OfficeQA Keenable search: redact the key, enforce the round deadline, tighten tests - #381
Open
Ilya Bogin (ilya-bogin-keenable) wants to merge 1 commit into
Conversation
- Redact KEENABLE_API_KEY from the HTTP error text before it is cut and reaches fail_reason, results.jsonl and the reflection prompt. - Check the round deadline again after each response: httpx applies the timeout per phase, so a slow last response could end past search_timeout_seconds unchecked. - Fall back to the URL for a missing or non-string result title. - Tests: a 302 is not followed, a non-chat backend is refused before any request, a non-JSON body fails, a key echoed in a 401 body (and one straddling the 1000-character cut) is redacted, a response past the deadline fails the round. test_fresh_config_never_calls_keenable now has the model ask for search and asserts the transport saw no request; before, the model answered on the first turn and an AssertionError inside process_one was swallowed, so it could not fail. - Docs: Keenable's search_max_num_results is per query (16 results per round with the defaults), and a generated query can repeat text from the document context or the skill, not only the question.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from the #368 review, one item per point.
KEENABLE_API_KEYis replaced with[redacted]in the HTTP error body, before the 1000-character cut, so a key straddling the cut can't leak a prefix either. Tests cover a key echoed in a 401 body (checked infail_reasonandconversation.json) and one at the cut.time.monotonic() > deadlineis checked again after each response, so a slow last query now fails the round too, as the docs say. Test: a single query whose response lands past the deadline.test_fresh_config_never_calls_keenablenow has the model ask for search, then asserts the mock transport saw no request.follow_redirects=True, no backend check, invalid JSON read as[], no redaction, no post-response deadline check, andautoselecting Keenable (the old fresh-config test passed with that one).search_max_num_resultsis per query for Keenable: 16 results, about 97k characters, per round with the defaults.titlefalls back to the URL.tests/test_officeqa_keenable_search.py: 42 passed. The full suite passes on 3.10 and 3.12, apart from failures that also happen on an unmodifiedccfdcdfin my environment:test_backend_pi::test_default_model_from_env, twotest_sleep_adopt_skill_subsettampered-manifest tests, and, on 3.10 only,test_officeqa_parallel_search::test_whole_mcp_request_has_a_deadline.