Skip to content

Issue 2: Validator approved a clear non-answer #3

@sanjaigridsandguides

Description

@sanjaigridsandguides

Problem

  • For FAQ 4 ("Who is your favourite actor?") the saved answer is the LLM's disclaimer:

    "I don't have personal preferences or feelings, so I don't have a favourite actor. I can provide information about various actors..."

  • The validator marked this is_valid=true with iterationCount=1.

Impact

  • The validator is failing exactly the case it exists to catch: an answer that does not address the question.
  • The bar for "valid" is far too low — any well-formed English sentence currently passes.

Required Fix

  • Add an explicit rule: "If the answer is a disclaimer or refusal to answer, set is_valid=false."
  • Add an explicit rule: "If the answer does not directly state a fact answering the question, set is_valid=false."

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions