From 008d78f3640bde7b4b0bd15e665a67e32a99d255 Mon Sep 17 00:00:00 2001 From: Daniel Medina Date: Fri, 6 Mar 2026 13:41:45 -0600 Subject: [PATCH 1/3] Add test_seeds generator --- README.md | 36 ++-- .../agent_e2e/install/install_generator.rb | 14 +- .../agent_e2e/install/templates/e2e | 21 ++- .../test_seeds/test_seeds_generator.rb | 176 ++++++++++++++++++ 4 files changed, 229 insertions(+), 18 deletions(-) create mode 100644 lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb diff --git a/README.md b/README.md index 23330ec..6c921ad 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,7 @@ This will: 5. Update `.gitignore` to exclude `node_modules`, `failures.md`, and `screenshots` 6. Run `npm install` in `agent-tests/` 7. Install the Chromium browser for Playwright +8. **Generate `db/test_seeds.rb`** by analyzing your models with OpenAI (requires `OPENAI_API_KEY`) ### Step 3: Set up your environment @@ -68,21 +69,20 @@ OPENAI_API_KEY=sk-proj-... > **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). -### Step 4: Create a QA test user +### Step 4: Generate test seeds -Add a seed user for the agent to log in with. In `db/seeds.rb`: +The install generator automatically analyzes all your models and `db/schema.rb`, then calls OpenAI to generate `db/test_seeds.rb` with realistic seed data for E2E testing — including a QA user, associated records, and proper handling of validations, enums, and Devise. -```ruby -if Rails.env.local? - User.find_or_create_by!(email: "qa@example.com") do |user| - user.password = "Password123!" - user.password_confirmation = "Password123!" - # If using Devise confirmable: - user.confirmed_at = Time.current - end -end +If the API key wasn't available during install, or you want to regenerate seeds after adding new models, run: + +```sh +bin/rails generate agent_e2e:test_seeds ``` +The generated file uses `find_or_create_by!` so it's safe to run multiple times. Review and adjust it as needed. + +You can override the AI model used with the `SEED_AI_MODEL` environment variable (defaults to `o3`). + ### Step 5: Add `data-testid` attributes to your views The agent can interact with elements by visible text, labels, and roles — but `data-testid` attributes make interactions more reliable, especially for buttons and form elements. @@ -140,7 +140,7 @@ bin/e2e ``` This will: -1. Prepare and seed the test database +1. Prepare the test database and seed it with `db/test_seeds.rb` 2. Start a Rails server on port 3001 (configurable via `PORT`) 3. Run each test case with the AI agent 4. Print a summary of results @@ -148,6 +148,8 @@ This will: 6. Save screenshots on failure to `agent-tests/screenshots/` 7. Clean up: stop the server and reset the database +> If `db/test_seeds.rb` doesn't exist, the script will stop and show you the steps to generate it. + ## Configuration All configuration is via environment variables. Set them in `.env` or pass them directly: @@ -155,7 +157,8 @@ All configuration is via environment variables. Set them in `.env` or pass them | Variable | Default | Description | |---|---|---| | `OPENAI_API_KEY` | *(required)* | Your OpenAI API key | -| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the agent | +| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the test agent | +| `SEED_AI_MODEL` | `o3` | OpenAI model to use for generating test seeds | | `BASE_URL` | `http://localhost:3000` | Base URL of the app (overridden to port 3001 by `bin/e2e`) | | `MAX_STEPS` | `25` | Maximum steps per test before timeout | | `ACTION_TIMEOUT` | `8000` | Timeout in ms for each browser action | @@ -176,12 +179,16 @@ All configuration is via environment variables. Set them in `.env` or pass them | File | Description | |---|---| +| `db/test_seeds.rb` | AI-generated seed data for E2E testing (generated by `agent_e2e:test_seeds`) | | `agent-tests/tests.md` | Your test cases (you write this) | | `agent-tests/failures.md` | Detailed failure reports with action history (auto-generated, gitignored) | | `agent-tests/screenshots/` | Screenshots captured on failure (auto-generated, gitignored) | ## Troubleshooting +**"db/test_seeds.rb not found"** +Run `bin/rails generate agent_e2e:test_seeds` to generate it. Make sure `OPENAI_API_KEY` is set in your `.env`. + **"No test cases found in tests.md"** Add at least one test case line (not starting with `#`) to `agent-tests/tests.md`. @@ -197,6 +204,9 @@ Add `data-testid` attributes to ambiguous elements. The agent prefers `data-test **Asset compilation issues** Uncomment the asset build step in `bin/e2e` that matches your setup. +**Regenerating seeds after model changes** +Run `bin/rails generate agent_e2e:test_seeds` again. It will overwrite the existing file. + ## License MIT diff --git a/lib/generators/agent_e2e/install/install_generator.rb b/lib/generators/agent_e2e/install/install_generator.rb index 8f9559e..411f320 100644 --- a/lib/generators/agent_e2e/install/install_generator.rb +++ b/lib/generators/agent_e2e/install/install_generator.rb @@ -106,6 +106,10 @@ def install_playwright_browsers end end + def generate_test_seeds + generate "agent_e2e:test_seeds" + end + def print_next_steps say "" say "=" * 60, :green @@ -120,10 +124,12 @@ def print_next_steps say " 2. Add data-testid attributes to key UI elements:" say ' ' say "" - say " 3. Create a QA seed user (e.g. in db/seeds.rb):" - say " User.find_or_create_by!(email: 'qa@example.com') do |u|" - say " u.password = 'Password123!'" - say " end" + if File.exist?(File.join(destination_root, "db/test_seeds.rb")) + say " 3. Review the generated db/test_seeds.rb and adjust if needed." + else + say " 3. Generate test seeds:" + say " bin/rails generate agent_e2e:test_seeds" + end say "" say " 4. Write test cases in agent-tests/tests.md (one per line):" say " - Log in with qa@example.com and verify the dashboard loads" diff --git a/lib/generators/agent_e2e/install/templates/e2e b/lib/generators/agent_e2e/install/templates/e2e index 7becf4c..5f0ff22 100644 --- a/lib/generators/agent_e2e/install/templates/e2e +++ b/lib/generators/agent_e2e/install/templates/e2e @@ -24,7 +24,26 @@ echo "==> Preparing test database..." bin/rails db:test:prepare echo "==> Seeding test database..." -bin/rails db:seed +if [ -f db/test_seeds.rb ]; then + bin/rails runner db/test_seeds.rb +else + echo "" + echo "ERROR: db/test_seeds.rb not found." + echo "" + if grep -q "OPENAI_API_KEY" .env 2>/dev/null; then + echo "Next steps:" + echo " Run the test seeds generator:" + echo " bin/rails generate agent_e2e:test_seeds" + else + echo "Next steps:" + echo " 1. Add your OpenAI API key to .env:" + echo " OPENAI_API_KEY=sk-..." + echo " 2. Run the test seeds generator:" + echo " bin/rails generate agent_e2e:test_seeds" + fi + echo "" + exit 1 +fi # Uncomment if your app needs asset precompilation (e.g. Tailwind CSS): # echo "==> Compiling assets..." diff --git a/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb new file mode 100644 index 0000000..60b1e26 --- /dev/null +++ b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb @@ -0,0 +1,176 @@ +# frozen_string_literal: true + +require "rails/generators/base" +require "net/http" +require "json" +require "uri" + +module AgentE2e + module Generators + class TestSeedsGenerator < Rails::Generators::Base + desc "Analyzes your models and generates db/test_seeds.rb using OpenAI" + + def generate_test_seeds + model_files = Dir.glob(File.join(destination_root, "app/models/**/*.rb")) + if model_files.empty? + say "No models found in app/models/. Nothing to generate.", :yellow + return + end + + api_key = ENV["OPENAI_API_KEY"] + dotenv_path = File.join(destination_root, ".env") + if !api_key && File.exist?(dotenv_path) + File.readlines(dotenv_path).each do |line| + if line.strip.match?(/\AOPENAI_API_KEY=/) + api_key = line.strip.sub("OPENAI_API_KEY=", "") + break + end + end + end + + unless api_key && !api_key.empty? + say "OPENAI_API_KEY not found.", :red + say "" + say "Add it to your .env file and try again:", :yellow + say " echo 'OPENAI_API_KEY=sk-...' >> .env" + say " bin/rails generate agent_e2e:test_seeds" + return + end + + say "Found #{model_files.size} model(s). Analyzing...", :green + + schema_path = File.join(destination_root, "db/schema.rb") + schema_content = File.exist?(schema_path) ? File.read(schema_path) : nil + + models_content = model_files.map { |f| + relative = f.sub("#{destination_root}/", "") + "# #{relative}\n#{File.read(f)}" + }.join("\n\n") + + user_model = detect_user_model(model_files) + qa_user_block = build_qa_user_block(user_model, models_content) + + prompt = build_seed_prompt(models_content, schema_content) + ai_seeds = call_openai(api_key, prompt) + + seeds_code = <<~RUBY + # Auto-generated by agent_e2e:test_seeds + # QA user for E2E testing (guaranteed) + + #{qa_user_block} + + # AI-generated seed data for all models + + #{ai_seeds || "# Could not generate seed data. Add your seeds here."} + RUBY + + create_file "db/test_seeds.rb", seeds_code, force: true + say "" + say "Generated db/test_seeds.rb", :green + unless ai_seeds + say "WARNING: AI seed generation failed. QA user was added but other models need manual seeds.", :yellow + end + say "Review it and run your tests with: bin/e2e", :cyan + end + + private + + def detect_user_model(model_files) + model_files.each do |f| + content = File.read(f) + basename = File.basename(f, ".rb") + return basename.camelize if content.match?(/devise|has_secure_password|authenticate/i) + end + model_files.each do |f| + basename = File.basename(f, ".rb") + return basename.camelize if basename.match?(/\Auser\z|\Aaccount\z|\Aadmin\z/i) + end + "User" + end + + def build_qa_user_block(user_model, models_content) + uses_devise = models_content.match?(/devise/) + uses_secure_password = models_content.match?(/has_secure_password/) + + lines = [] + lines << "qa_user = #{user_model}.find_or_create_by!(email: \"qa@example.com\") do |u|" + lines << " u.password = \"Password123!\"" + lines << " u.password_confirmation = \"Password123!\"" if uses_devise + lines << " u.confirmed_at = Time.current" if models_content.match?(/confirmable/i) + lines << "end" + lines.join("\n") + end + + def build_seed_prompt(models_content, schema_content) + <<~PROMPT + You are a Ruby on Rails expert. Analyze the following models and database schema, + then generate seed data for end-to-end (E2E) browser testing. + + IMPORTANT: Do NOT create the QA user (qa@example.com) — it is already created separately. + Use the variable `qa_user` when you need to reference the user (e.g. for associations). + + Requirements: + - Use `find_or_create_by!` to make the script idempotent (safe to run multiple times) + - Create realistic sample data for ALL models (except the QA user), respecting associations and validations + - Handle belongs_to associations by creating parent records first (topological order) + - Include enough variety (at least 2-3 records per model where it makes sense) + - Add brief comments explaining what each section seeds + - Wrap everything in an ActiveRecord::Base.transaction block + - Handle any enum values you find in the models + - If there's a role/admin system, create an admin user (different from qa_user) + - The data should be meaningful and realistic, not placeholder text + - If models use Devise, handle `confirmed_at`, `password_confirmation`, etc. + - If models have file attachments (ActiveStorage), skip those fields + - Output ONLY valid Ruby code — no markdown fences, no explanations + + ## Models: + + #{models_content} + + #{schema_content ? "## Database Schema:\n\n#{schema_content}" : ""} + + Generate the seed code: + PROMPT + end + + def call_openai(api_key, prompt) + model = ENV["SEED_AI_MODEL"] || "o3" + uri = URI("https://api.openai.com/v1/responses") + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = true + http.read_timeout = 120 + + request = Net::HTTP::Post.new(uri) + request["Content-Type"] = "application/json" + request["Authorization"] = "Bearer #{api_key}" + + request.body = JSON.generate({ + model: model, + input: prompt + }) + + say "Calling #{model} (this may take a moment)...", :cyan + response = http.request(request) + + if response.code.to_i == 200 + data = JSON.parse(response.body) + extract_ruby_code(data["output_text"]) + else + say "OpenAI API error (#{response.code}): #{response.body[0..300]}", :red + nil + end + rescue => e + say "Error calling OpenAI API: #{e.message}", :red + nil + end + + def extract_ruby_code(content) + return nil if content.nil? || content.strip.empty? + code = content.strip + code = code.sub(/\A```ruby\n?/, "").sub(/\n?```\z/, "") + code = code.sub(/\A```\n?/, "").sub(/\n?```\z/, "") + code.strip + "\n" + end + end + end +end From 594371411fcf6ca521f29d80232c6fbee727a6e4 Mon Sep 17 00:00:00 2001 From: Daniel Medina Date: Fri, 6 Mar 2026 13:46:22 -0600 Subject: [PATCH 2/3] Improve Readme file --- README.md | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 6c921ad..ce07d60 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,17 @@ end bundle install ``` -### Step 2: Run the install generator +### Step 2: Set up your environment + +Add your OpenAI API key to your `.env` file in the Rails root: + +```env +OPENAI_API_KEY=sk-proj-... +``` + +> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). This key is needed both for the install generator (to generate test seeds) and for running the E2E tests. + +### Step 3: Run the install generator ```sh bin/rails generate agent_e2e:install @@ -57,23 +67,11 @@ This will: 5. Update `.gitignore` to exclude `node_modules`, `failures.md`, and `screenshots` 6. Run `npm install` in `agent-tests/` 7. Install the Chromium browser for Playwright -8. **Generate `db/test_seeds.rb`** by analyzing your models with OpenAI (requires `OPENAI_API_KEY`) - -### Step 3: Set up your environment - -Add your OpenAI API key to your `.env` file in the Rails root: - -```env -OPENAI_API_KEY=sk-proj-... -``` - -> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). - -### Step 4: Generate test seeds +8. **Generate `db/test_seeds.rb`** by analyzing your models with OpenAI -The install generator automatically analyzes all your models and `db/schema.rb`, then calls OpenAI to generate `db/test_seeds.rb` with realistic seed data for E2E testing — including a QA user, associated records, and proper handling of validations, enums, and Devise. +The generator reads all your models and `db/schema.rb`, then calls OpenAI to generate `db/test_seeds.rb` with realistic seed data for E2E testing — including a QA user (`qa@example.com` / `Password123!`), associated records, and proper handling of validations, enums, and Devise. -If the API key wasn't available during install, or you want to regenerate seeds after adding new models, run: +To regenerate seeds after adding new models: ```sh bin/rails generate agent_e2e:test_seeds @@ -83,7 +81,7 @@ The generated file uses `find_or_create_by!` so it's safe to run multiple times. You can override the AI model used with the `SEED_AI_MODEL` environment variable (defaults to `o3`). -### Step 5: Add `data-testid` attributes to your views +### Step 4: Add `data-testid` attributes to your views The agent can interact with elements by visible text, labels, and roles — but `data-testid` attributes make interactions more reliable, especially for buttons and form elements. @@ -95,7 +93,7 @@ The agent can interact with elements by visible text, labels, and roles — but **Recommendation:** Add `data-testid` to every interactive element (buttons, links, inputs, selects). This doesn't affect your production HTML and makes tests much more stable. -### Step 6: Handle asset compilation (if needed) +### Step 5: Handle asset compilation (if needed) If your app uses Tailwind CSS, esbuild, or another build step, uncomment the relevant line in `bin/e2e`: From 7de067fc6916bac7625fdaa527ade13cc70667a9 Mon Sep 17 00:00:00 2001 From: Daniel Medina Date: Fri, 6 Mar 2026 14:00:36 -0600 Subject: [PATCH 3/3] Improve seeds generator logic --- .../test_seeds/test_seeds_generator.rb | 57 ++++++++++--------- 1 file changed, 31 insertions(+), 26 deletions(-) diff --git a/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb index 60b1e26..7184adb 100644 --- a/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb +++ b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb @@ -48,27 +48,25 @@ def generate_test_seeds }.join("\n\n") user_model = detect_user_model(model_files) - qa_user_block = build_qa_user_block(user_model, models_content) - prompt = build_seed_prompt(models_content, schema_content) + prompt = build_seed_prompt(models_content, schema_content, user_model) ai_seeds = call_openai(api_key, prompt) - seeds_code = <<~RUBY - # Auto-generated by agent_e2e:test_seeds - # QA user for E2E testing (guaranteed) + if ai_seeds + seeds_code = <<~RUBY + # Auto-generated by agent_e2e:test_seeds - #{qa_user_block} - - # AI-generated seed data for all models - - #{ai_seeds || "# Could not generate seed data. Add your seeds here."} - RUBY - - create_file "db/test_seeds.rb", seeds_code, force: true - say "" - say "Generated db/test_seeds.rb", :green - unless ai_seeds - say "WARNING: AI seed generation failed. QA user was added but other models need manual seeds.", :yellow + #{ai_seeds} + RUBY + create_file "db/test_seeds.rb", seeds_code, force: true + say "" + say "Generated db/test_seeds.rb", :green + else + fallback = build_fallback_seeds(user_model, models_content) + create_file "db/test_seeds.rb", fallback, force: true + say "" + say "Generated db/test_seeds.rb (fallback — AI generation failed)", :yellow + say "WARNING: Review db/test_seeds.rb and add seeds for your models manually.", :yellow end say "Review it and run your tests with: bin/e2e", :cyan end @@ -88,30 +86,37 @@ def detect_user_model(model_files) "User" end - def build_qa_user_block(user_model, models_content) + def build_fallback_seeds(user_model, models_content) uses_devise = models_content.match?(/devise/) - uses_secure_password = models_content.match?(/has_secure_password/) - lines = [] + lines << "# Auto-generated by agent_e2e:test_seeds (fallback)" + lines << "# AI generation failed — review and complete this file manually." + lines << "" lines << "qa_user = #{user_model}.find_or_create_by!(email: \"qa@example.com\") do |u|" lines << " u.password = \"Password123!\"" lines << " u.password_confirmation = \"Password123!\"" if uses_devise lines << " u.confirmed_at = Time.current" if models_content.match?(/confirmable/i) + lines << " # TODO: add any other required fields for #{user_model}" lines << "end" + lines << "" lines.join("\n") end - def build_seed_prompt(models_content, schema_content) + def build_seed_prompt(models_content, schema_content, user_model) <<~PROMPT You are a Ruby on Rails expert. Analyze the following models and database schema, then generate seed data for end-to-end (E2E) browser testing. - IMPORTANT: Do NOT create the QA user (qa@example.com) — it is already created separately. - Use the variable `qa_user` when you need to reference the user (e.g. for associations). - Requirements: + - The FIRST thing in the output must be creating a QA user for E2E testing: + * Model: #{user_model} + * Email: qa@example.com + * Password: Password123! + * Assign the result to a variable called `qa_user` + * Fill ALL required fields (from validations and null:false columns) with realistic values + * If the model uses Devise, set password_confirmation and confirmed_at - Use `find_or_create_by!` to make the script idempotent (safe to run multiple times) - - Create realistic sample data for ALL models (except the QA user), respecting associations and validations + - Create realistic sample data for ALL models, respecting associations and validations - Handle belongs_to associations by creating parent records first (topological order) - Include enough variety (at least 2-3 records per model where it makes sense) - Add brief comments explaining what each section seeds @@ -134,7 +139,7 @@ def build_seed_prompt(models_content, schema_content) end def call_openai(api_key, prompt) - model = ENV["SEED_AI_MODEL"] || "o3" + model = ENV["SEED_AI_MODEL"] || "gpt-5.3-codex" uri = URI("https://api.openai.com/v1/responses") http = Net::HTTP.new(uri.host, uri.port) http.use_ssl = true