Add test_seeds generator for automatic E2E seed data

README.md

@@ -42,7 +42,17 @@ end
 bundle install
 ```
 
-### Step 2: Run the install generator
+### Step 2: Set up your environment
+
+Add your OpenAI API key to your `.env` file in the Rails root:
+
+```env
+OPENAI_API_KEY=sk-proj-...
+```
+
+> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). This key is needed both for the install generator (to generate test seeds) and for running the E2E tests.
+
+### Step 3: Run the install generator
 
 ```sh
 bin/rails generate agent_e2e:install
@@ -57,33 +67,21 @@ This will:
 5. Update `.gitignore` to exclude `node_modules`, `failures.md`, and `screenshots`
 6. Run `npm install` in `agent-tests/`
 7. Install the Chromium browser for Playwright
+8. **Generate `db/test_seeds.rb`** by analyzing your models with OpenAI
 
-### Step 3: Set up your environment
+The generator reads all your models and `db/schema.rb`, then calls OpenAI to generate `db/test_seeds.rb` with realistic seed data for E2E testing — including a QA user (`qa@example.com` / `Password123!`), associated records, and proper handling of validations, enums, and Devise.
 
-Add your OpenAI API key to your `.env` file in the Rails root:
+To regenerate seeds after adding new models:
 
-```env
-OPENAI_API_KEY=sk-proj-...
+```sh
+bin/rails generate agent_e2e:test_seeds
 ```
 
-> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default).
-
-### Step 4: Create a QA test user
+The generated file uses `find_or_create_by!` so it's safe to run multiple times. Review and adjust it as needed.
 
-Add a seed user for the agent to log in with. In `db/seeds.rb`:
-
-```ruby
-if Rails.env.local?
-  User.find_or_create_by!(email: "qa@example.com") do |user|
-    user.password = "Password123!"
-    user.password_confirmation = "Password123!"
-    # If using Devise confirmable:
-    user.confirmed_at = Time.current
-  end
-end
-```
+You can override the AI model used with the `SEED_AI_MODEL` environment variable (defaults to `o3`).
 
-### Step 5: Add `data-testid` attributes to your views
+### Step 4: Add `data-testid` attributes to your views
 
 The agent can interact with elements by visible text, labels, and roles — but `data-testid` attributes make interactions more reliable, especially for buttons and form elements.
 
@@ -95,7 +93,7 @@ The agent can interact with elements by visible text, labels, and roles — but
 
 **Recommendation:** Add `data-testid` to every interactive element (buttons, links, inputs, selects). This doesn't affect your production HTML and makes tests much more stable.
 
-### Step 6: Handle asset compilation (if needed)
+### Step 5: Handle asset compilation (if needed)
 
 If your app uses Tailwind CSS, esbuild, or another build step, uncomment the relevant line in `bin/e2e`:
 
@@ -140,22 +138,25 @@ bin/e2e
 ```
 
 This will:
-1. Prepare and seed the test database
+1. Prepare the test database and seed it with `db/test_seeds.rb`
 2. Start a Rails server on port 3001 (configurable via `PORT`)
 3. Run each test case with the AI agent
 4. Print a summary of results
 5. Write `agent-tests/failures.md` with detailed failure reports (if any)
 6. Save screenshots on failure to `agent-tests/screenshots/`
 7. Clean up: stop the server and reset the database
 
+> If `db/test_seeds.rb` doesn't exist, the script will stop and show you the steps to generate it.
+
 ## Configuration
 
 All configuration is via environment variables. Set them in `.env` or pass them directly:
 
 | Variable | Default | Description |
 |---|---|---|
 | `OPENAI_API_KEY` | *(required)* | Your OpenAI API key |
-| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the agent |
+| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the test agent |
+| `SEED_AI_MODEL` | `o3` | OpenAI model to use for generating test seeds |
 | `BASE_URL` | `http://localhost:3000` | Base URL of the app (overridden to port 3001 by `bin/e2e`) |
 | `MAX_STEPS` | `25` | Maximum steps per test before timeout |
 | `ACTION_TIMEOUT` | `8000` | Timeout in ms for each browser action |
@@ -176,12 +177,16 @@ All configuration is via environment variables. Set them in `.env` or pass them
 
 | File | Description |
 |---|---|
+| `db/test_seeds.rb` | AI-generated seed data for E2E testing (generated by `agent_e2e:test_seeds`) |
 | `agent-tests/tests.md` | Your test cases (you write this) |
 | `agent-tests/failures.md` | Detailed failure reports with action history (auto-generated, gitignored) |
 | `agent-tests/screenshots/` | Screenshots captured on failure (auto-generated, gitignored) |
 
 ## Troubleshooting
 
+**"db/test_seeds.rb not found"**
+Run `bin/rails generate agent_e2e:test_seeds` to generate it. Make sure `OPENAI_API_KEY` is set in your `.env`.
+
 **"No test cases found in tests.md"**
 Add at least one test case line (not starting with `#`) to `agent-tests/tests.md`.
 
@@ -197,6 +202,9 @@ Add `data-testid` attributes to ambiguous elements. The agent prefers `data-test
 **Asset compilation issues**
 Uncomment the asset build step in `bin/e2e` that matches your setup.
 
+**Regenerating seeds after model changes**
+Run `bin/rails generate agent_e2e:test_seeds` again. It will overwrite the existing file.
+
 ## License
 
 MIT

lib/generators/agent_e2e/install/install_generator.rb

@@ -106,6 +106,10 @@ def install_playwright_browsers
         end
       end
 
+      def generate_test_seeds
+        generate "agent_e2e:test_seeds"
+      end
+
       def print_next_steps
         say ""
         say "=" * 60, :green
@@ -120,10 +124,12 @@ def print_next_steps
         say "  2. Add data-testid attributes to key UI elements:"
         say '     <button data-testid="submit-login">Log in</button>'
         say ""
-        say "  3. Create a QA seed user (e.g. in db/seeds.rb):"
-        say "     User.find_or_create_by!(email: 'qa@example.com') do |u|"
-        say "       u.password = 'Password123!'"
-        say "     end"
+        if File.exist?(File.join(destination_root, "db/test_seeds.rb"))
+          say "  3. Review the generated db/test_seeds.rb and adjust if needed."
+        else
+          say "  3. Generate test seeds:"
+          say "     bin/rails generate agent_e2e:test_seeds"
+        end
         say ""
         say "  4. Write test cases in agent-tests/tests.md (one per line):"
         say "     - Log in with qa@example.com and verify the dashboard loads"

lib/generators/agent_e2e/install/templates/e2e

@@ -24,7 +24,26 @@ echo "==> Preparing test database..."
 bin/rails db:test:prepare
 
 echo "==> Seeding test database..."
-bin/rails db:seed
+if [ -f db/test_seeds.rb ]; then
+  bin/rails runner db/test_seeds.rb
+else
+  echo ""
+  echo "ERROR: db/test_seeds.rb not found."
+  echo ""
+  if grep -q "OPENAI_API_KEY" .env 2>/dev/null; then
+    echo "Next steps:"
+    echo "  Run the test seeds generator:"
+    echo "    bin/rails generate agent_e2e:test_seeds"
+  else
+    echo "Next steps:"
+    echo "  1. Add your OpenAI API key to .env:"
+    echo "     OPENAI_API_KEY=sk-..."
+    echo "  2. Run the test seeds generator:"
+    echo "     bin/rails generate agent_e2e:test_seeds"
+  fi
+  echo ""
+  exit 1
+fi
 
 # Uncomment if your app needs asset precompilation (e.g. Tailwind CSS):
 # echo "==> Compiling assets..."

lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb

@@ -0,0 +1,181 @@
+# frozen_string_literal: true
+
+require "rails/generators/base"
+require "net/http"
+require "json"
+require "uri"
+
+module AgentE2e
+  module Generators
+    class TestSeedsGenerator < Rails::Generators::Base
+      desc "Analyzes your models and generates db/test_seeds.rb using OpenAI"
+
+      def generate_test_seeds
+        model_files = Dir.glob(File.join(destination_root, "app/models/**/*.rb"))
+        if model_files.empty?
+          say "No models found in app/models/. Nothing to generate.", :yellow
+          return
+        end
+
+        api_key = ENV["OPENAI_API_KEY"]
+        dotenv_path = File.join(destination_root, ".env")
+        if !api_key && File.exist?(dotenv_path)
+          File.readlines(dotenv_path).each do |line|
+            if line.strip.match?(/\AOPENAI_API_KEY=/)
+              api_key = line.strip.sub("OPENAI_API_KEY=", "")
+              break
+            end
+          end
+        end
+
+        unless api_key && !api_key.empty?
+          say "OPENAI_API_KEY not found.", :red
+          say ""
+          say "Add it to your .env file and try again:", :yellow
+          say "  echo 'OPENAI_API_KEY=sk-...' >> .env"
+          say "  bin/rails generate agent_e2e:test_seeds"
+          return
+        end
+
+        say "Found #{model_files.size} model(s). Analyzing...", :green
+
+        schema_path = File.join(destination_root, "db/schema.rb")
+        schema_content = File.exist?(schema_path) ? File.read(schema_path) : nil
+
+        models_content = model_files.map { |f|
+          relative = f.sub("#{destination_root}/", "")
+          "# #{relative}\n#{File.read(f)}"
+        }.join("\n\n")
+
+        user_model = detect_user_model(model_files)
+
+        prompt = build_seed_prompt(models_content, schema_content, user_model)
+        ai_seeds = call_openai(api_key, prompt)
+
+        if ai_seeds
+          seeds_code = <<~RUBY
+            # Auto-generated by agent_e2e:test_seeds
+
+            #{ai_seeds}
+          RUBY
+          create_file "db/test_seeds.rb", seeds_code, force: true
+          say ""
+          say "Generated db/test_seeds.rb", :green
+        else
+          fallback = build_fallback_seeds(user_model, models_content)
+          create_file "db/test_seeds.rb", fallback, force: true
+          say ""
+          say "Generated db/test_seeds.rb (fallback — AI generation failed)", :yellow
+          say "WARNING: Review db/test_seeds.rb and add seeds for your models manually.", :yellow
+        end
+        say "Review it and run your tests with: bin/e2e", :cyan
+      end
+
+      private
+
+      def detect_user_model(model_files)
+        model_files.each do |f|
+          content = File.read(f)
+          basename = File.basename(f, ".rb")
+          return basename.camelize if content.match?(/devise|has_secure_password|authenticate/i)
+        end
+        model_files.each do |f|
+          basename = File.basename(f, ".rb")
+          return basename.camelize if basename.match?(/\Auser\z|\Aaccount\z|\Aadmin\z/i)
+        end
+        "User"
+      end
+
+      def build_fallback_seeds(user_model, models_content)
+        uses_devise = models_content.match?(/devise/)
+        lines = []
+        lines << "# Auto-generated by agent_e2e:test_seeds (fallback)"
+        lines << "# AI generation failed — review and complete this file manually."
+        lines << ""
+        lines << "qa_user = #{user_model}.find_or_create_by!(email: \"qa@example.com\") do |u|"
+        lines << "  u.password = \"Password123!\""
+        lines << "  u.password_confirmation = \"Password123!\"" if uses_devise
+        lines << "  u.confirmed_at = Time.current" if models_content.match?(/confirmable/i)
+        lines << "  # TODO: add any other required fields for #{user_model}"
+        lines << "end"
+        lines << ""
+        lines.join("\n")
+      end
+
+      def build_seed_prompt(models_content, schema_content, user_model)
+        <<~PROMPT
+          You are a Ruby on Rails expert. Analyze the following models and database schema,
+          then generate seed data for end-to-end (E2E) browser testing.
+
+          Requirements:
+          - The FIRST thing in the output must be creating a QA user for E2E testing:
+            * Model: #{user_model}
+            * Email: qa@example.com
+            * Password: Password123!
+            * Assign the result to a variable called `qa_user`
+            * Fill ALL required fields (from validations and null:false columns) with realistic values
+            * If the model uses Devise, set password_confirmation and confirmed_at
+          - Use `find_or_create_by!` to make the script idempotent (safe to run multiple times)
+          - Create realistic sample data for ALL models, respecting associations and validations
+          - Handle belongs_to associations by creating parent records first (topological order)
+          - Include enough variety (at least 2-3 records per model where it makes sense)
+          - Add brief comments explaining what each section seeds
+          - Wrap everything in an ActiveRecord::Base.transaction block
+          - Handle any enum values you find in the models
+          - If there's a role/admin system, create an admin user (different from qa_user)
+          - The data should be meaningful and realistic, not placeholder text
+          - If models use Devise, handle `confirmed_at`, `password_confirmation`, etc.
+          - If models have file attachments (ActiveStorage), skip those fields
+          - Output ONLY valid Ruby code — no markdown fences, no explanations
+
+          ## Models:
+
+          #{models_content}
+
+          #{schema_content ? "## Database Schema:\n\n#{schema_content}" : ""}
+
+          Generate the seed code:
+        PROMPT
+      end
+
+      def call_openai(api_key, prompt)
+        model = ENV["SEED_AI_MODEL"] || "gpt-5.3-codex"
+        uri = URI("https://api.openai.com/v1/responses")
+        http = Net::HTTP.new(uri.host, uri.port)
+        http.use_ssl = true
+        http.read_timeout = 120
+
+        request = Net::HTTP::Post.new(uri)
+        request["Content-Type"] = "application/json"
+        request["Authorization"] = "Bearer #{api_key}"
+
+        request.body = JSON.generate({
+          model: model,
+          input: prompt
+        })
+
+        say "Calling #{model} (this may take a moment)...", :cyan
+        response = http.request(request)
+
+        if response.code.to_i == 200
+          data = JSON.parse(response.body)
+          extract_ruby_code(data["output_text"])
+        else
+          say "OpenAI API error (#{response.code}): #{response.body[0..300]}", :red
+          nil
+        end
+      rescue => e
+        say "Error calling OpenAI API: #{e.message}", :red
+        nil
+      end
+
+      def extract_ruby_code(content)
+        return nil if content.nil? || content.strip.empty?
+        code = content.strip
+        code = code.sub(/\A```ruby\n?/, "").sub(/\n?```\z/, "")
+        code = code.sub(/\A```\n?/, "").sub(/\n?```\z/, "")
+        code.strip + "\n"
+      end
+    end
+  end
+end