Update ADK doc according to issue #1587 - 11

[adk-bot]

11. Eval Scenario Generation

Doc file: docs/evaluate/user-sim.md

Current state:

The docs/evaluate/user-sim.md document explains how to add eval cases manually using adk eval_set add_eval_case but does not mention automated generation.

Proposed Change:

Add documentation for the new adk eval_set generate_eval_cases command. Explain that this command uses the Vertex AI Eval SDK (ScenarioGenerator) to dynamically generate a suite of conversation scenarios and automatically adds them to the specified eval set based on a user simulation config file. Provide an example of how to use it: adk eval_set generate_eval_cases <agent_path> <eval_set_id> --user_simulation_config_I file <path>.

Reasoning:
This is a powerful new CLI capability that automates the tedious process of writing manual conversation scenarios for evaluation by leveraging an LLM to generate them.

Reference: src/google/adk/cli/cli_tools_click.py

[netlify]

✅ Deploy Preview for adk-docs-preview ready!

Name	Link
🔨 Latest commit	68a3794
🔍 Latest deploy log	https://app.netlify.com/projects/adk-docs-preview/deploys/69d80b5281b3ca00081ced46
😎 Deploy Preview	https://deploy-preview-1606--adk-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[jcwriter74]

Eval Scenario Generation

Doc file: docs/evaluate/user-sim.md

Proposed Documentation (human: Juan Carlos González Reséndiz)

Prerequisites:

• Agent Platform Credentials
• API Access
• Simulation Config

Documentation for adk eval_set generate_eval_cases

The adk eval_set generate_eval_cases is a new command that takes the place of the adk eval_set add_eval_cases. It creates sets of cases dynamically, meaning they don’t need to be predefined or manually added. These groups of conversation scenarios are generated using the Vertex AI Eval SDK (ScenarioGenerator), and added to the evaluation set; built around a user simulation config file.

The adk eval_set generate_eval_cases command can be defined as:

adk eval_set generate_eval_cases \
    <AGENT_MODULE_FILE_PATH> \
    <EVAL_SET_ID> \
    --user_simulation_config_file=<PATH_TO_CONFIG_FILE>

The arguments for that command are the next ones:

• AGENT_MODULE_FILE_PATH
  Required. The path to the Python file containing your agent definition (e.g., app/agent.py).
• EVAL_SET_ID
  Required. The unique identifier for the evaluation set. If the set does not exist, ADK will create it automatically.

The definition of a method working with the adk eval_set generate_eval_cases command is as follows:

def cli_generate_eval_cases(
    agent_module_file_path: str,
    eval_set_id: str,
    user_simulation_config_file: str,
    eval_storage_uri: Optional[str] = None,
    log_level: str = "INFO",
)

The method is called cli_generate_eval_cases, it has the prefix cli because this is a new feature of the CLI utilised for creating dynamic suites of conversations for user simulation exercises.

Its arguments are:

• agent_module_file_path: The path to the agent module file.
• eval_set_id: The id of the eval set to generate cases for.
• user_simulation_config_file: The path to the user simulation config file.
• eval_storage_uri: The eval storage uri.
• log_level: The log level

Example of Configuration

The --user_simulation_config_file is the core of the generation process. It tells the LLM what kind of users to simulate.
Example user_sim_config.json

{
  "scenarios": [
    {
      "starting_prompt": "I want to book a flight to London.",
      "conversation_plan": "The user should try to book a business class seat and ask about the refund policy.",
      "user_persona": {
        "id": "BUSINESS_TRAVELER",
        "description": "A busy professional who values efficiency and clear information.",
        "behaviors": [
          {
            "name": "Concise",
            "description": "User gives short, direct answers."
          }
        ]
      }
    }
  ],
  "num_cases": 5
}

** Arguments **

• starting_prompt: How the simulated user begins the chat.
• conversation_plan: The specific goals the user must attempt to achieve.
• user_persona: (Optional) Detailed behavioral traits to test how the agent handles different personalities (e.g., impatient, confused, or technical users).

Example Command

To generate 10 new evaluation cases for a travel_agent and save them to a set named v1_test_suite:

Bash
adk eval_set generate_eval_cases \
  app/agent.py \
  v1_test_suite \
  --user_simulation_config_file configs/user_sim.json

What happens next?

• ADK loads your agent's tools and system instructions to understand its capabilities.
• The Vertex AI Eval SDK generates conversation trajectories based on your user_sim.json.
• The generated cases are saved to your local or remote storage.

You can now run the evaluation using:

Bash
adk eval app/agent.py v1_test_suite

Definition of how to use adk eval_set generate_eval_cases <agent_path> <eval_set_id> --user_simulation_config_I file <path>

Explanation of the components

Command Component Definitions:

• adk: CLI tool for the ADK.
• eval_set: subcommands for evaluation sets’ management.
• generate_eval_cases: subcommand that generates new evaluation cases and adds them to a specified evaluation set.
• <agent_path>: required positional argument that represents the file system path to the directory or main file defining the agent you want to evaluate (e.g., my_agent/ or my_agent/agent.py).
• <eval_set_id>: required positional argument: the path to the JSON file where the evaluation set is stored (e.g., my_agent/evals/my_test_cases.evalset.json). The generated cases will be added to this file.
  -* -user_simulation_config_file <path>: optional flag followed by a path to a configuration file (likely YAML or JSON) that provides parameters and guidelines for the user simulation engine to generate diverse and realistic conversational test cases.

Example of Usage

Let's assume you have:

• An agent defined in ~/dev/my_customer_service_agent/agent.py.
• An existing evaluation set file at ~/dev/my_customer_service_agent/evals/regression_tests.evalset.json.
• A user simulation configuration file at ~/dev/my_customer_service_agent/evals/simulation_config.yaml.

The command would look like this:

BASH
adk eval_set generate_eval_cases \
  ~/dev/my_customer_service_agent/agent.py \
  ~/dev/my_customer_service_agent/evals/regression_tests.evalset.json \
  --user_simulation_config_file ~/dev/my_customer_service_agent/evals/simulation_config.yaml

Implementation of the above definition to the YAML file below:

# simulation_config.yaml
num_cases: 20
max_turns: 8
user_personas:
  -  id: tech_savvy
    description: "A user who is comfortable with technical terms and asks specific questions."
    initial_queries:
      -  "What's the API rate limit?"
      -  "How do I integrate this with my existing system?"
      -  "Tell me about the security protocols."
  -  id: non_technical
    description: "A user who needs simpler explanations and might ask broader questions."
    initial_queries:
      -  "How does this help me?"
      -  "Is it easy to use?"
      -  "What are the main benefits?"
topics:
  -  "pricing"
  -  "features"
  -  "support"
  -  "integration"
goal_oriented:
  -  description: "User wants to understand the difference between plan A and plan B."
  -  description: "User is trying to troubleshoot an error message."

Reviewed with:
Duckie 1
Duckie 2
Google AI Studio 1
Google AI Studio 2

[AI recommendation: This documentation is designed to fit into the existing ADK CLI reference or the "Evaluation" section of the developer guide.]

Current state:

The docs/evaluate/user-sim.md document explains how to add eval cases manually using adk eval_set add_eval_case but does not mention automated generation.

AI Proposed Change:

Add documentation for the new adk eval_set generate_eval_cases command.
Explain that this command uses the Vertex AI Eval SDK (ScenarioGenerator) to dynamically generate a suite of conversation scenarios and automatically adds them to the specified eval set based on a user simulation config file.
Provide an example of how to use it: adk eval_set generate_eval_cases <agent_path> <eval_set_id> --user_simulation_config_I file .

AI Reasoning:
This is a powerful new CLI capability that automates the tedious process of writing manual conversation scenarios for evaluation by leveraging an LLM to generate them.

Reference: src/google/adk/cli/cli_tools_click.py