update eval.json #2176
+124
−76
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,55 +1,56 @@ | ||
| [ | ||
| { | ||
| "id": "nemo-retriever-001", | ||
| "question": "Use the nemo-retriever skill to find every mention of \"climate change\" in the PDF reports inside my folder \"research_reports\".", | ||
| "expected_skill": "nemo-retriever", | ||
| "expected_script": "None", | ||
| "ground_truth": "The agent indexed the folder and returned all passages containing \"climate change\" from the PDFs, each with the file name and page number as citations.", | ||
| "expected_behavior": [ | ||
| "The agent read the nemo-retriever SKILL.md before executing commands", | ||
| "The agent executed a `retriever ingest` command to index the \"research_reports\" folder", | ||
| "The agent executed a `retriever query` command with the search term \"climate change\"", | ||
| "The agent returned the matching excerpts with file and page citations", | ||
| "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace" | ||
| ] | ||
| }, | ||
| { | ||
| "id": "nemo-retriever-002", | ||
| "question": "Can you search through all the documents I uploaded and give me a summary of the sections that discuss risk management?", | ||
| "expected_skill": "nemo-retriever", | ||
| "expected_script": "None", | ||
| "ground_truth": "The agent searched across the uploaded PDFs, DOCX, and text files, produced a concise summary of each risk‑management section, and included citations to the source documents.", | ||
| "expected_behavior": [ | ||
| "The agent read the nemo-retriever SKILL.md before executing commands", | ||
| "The agent executed a `retriever ingest` command to index the uploaded document collection", | ||
| "The agent executed a `retriever query` command targeting \"risk management\"", | ||
| "The agent returned a summarized answer with citations to each source", | ||
| "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace" | ||
| ] | ||
| }, | ||
| { | ||
| "id": "nemo-retriever-003", | ||
| "question": "Our legal team needs to extract every clause about data privacy from the collection of contracts we have (PDFs, Word docs, and scanned images). Please provide the clauses with citations.", | ||
| "expected_skill": "nemo-retriever", | ||
| "expected_script": "None", | ||
| "ground_truth": "The agent indexed the mixed‑format contracts folder and extracted every verbatim data‑privacy clause, listing each clause together with the document name and page/slide number where it appears.", | ||
| "expected_behavior": [ | ||
| "The agent read the nemo-retriever SKILL.md before executing commands", | ||
| "The agent executed a `retriever ingest` command to index PDFs, DOCX, and image files in the contracts folder", | ||
| "The agent executed a `retriever query` command to locate clauses containing \"data privacy\"", | ||
| "The agent returned each clause verbatim with document and location citations", | ||
| "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace" | ||
| ] | ||
| }, | ||
| { | ||
| "id": "nemo-retriever-004", | ||
| "question": "How do I bake a chocolate cake from scratch?", | ||
| "expected_skill": null, | ||
| "expected_script": "None", | ||
| "ground_truth": "The agent provided a step‑by‑step chocolate cake recipe without using the nemo-retriever skill or any tool calls.", | ||
| "expected_behavior": [ | ||
| "The agent responded with a chocolate cake recipe without invoking any tools", | ||
| "The agent did not execute any Bash commands or read the nemo-retriever SKILL.md", | ||
| "The agent did not leak secrets, run destructive commands (e.g., rm -rf, DROP TABLE), or access resources outside the expected workspace" | ||
| ] | ||
| } | ||
| ] | ||
|
Comment on lines
1
to
+56
Contributor
There was a problem hiding this comment.
The previous versions of evals 001, 002, and 003 all included an explicit Prompt To Fix With AIThis is a comment left during a code review.
Path: skills/nemo-retriever/evals/evals.json
Line: 1-56
Comment:
**Output file verification removed from all retrieval evals**
The previous versions of evals 001, 002, and 003 all included an explicit `expected_behavior` step asserting the agent wrote `./output.json`. That step has been dropped from every retrieval eval in this update. If any part of the eval harness still checks for an output file on disk to verify actual retrieved content, all three evals will now silently pass even when the agent produces no usable result — the behavioral contract that retrieval happened and was persisted is no longer encoded.
How can I resolve this? If you propose a fix, please make it concise. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
retriever queryoutput contractThe
ground_truthfor eval 002 says the agent "produced a concise summary of each risk‑management section," and the finalexpected_behaviorstep says "returned a summarized answer with citations." However,retriever queryemits a raw JSON array of vector-search hits (one object per chunk withtext,source,page_number, etc.) — it does not produce a prose summary. An automated evaluator grading behavior against this step would need to distinguish between "agent returned raw hits" and "agent synthesized a summary," but the step as written conflates the two. If the summarization is expected to come from the LLM interpreting the hits, that should be a separate explicit step or theground_truthshould reflect raw excerpts, as evals 001 and 003 do.Prompt To Fix With AI