diff --git a/README.md b/README.md index 23330ec..ce07d60 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,17 @@ end bundle install ``` -### Step 2: Run the install generator +### Step 2: Set up your environment + +Add your OpenAI API key to your `.env` file in the Rails root: + +```env +OPENAI_API_KEY=sk-proj-... +``` + +> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). This key is needed both for the install generator (to generate test seeds) and for running the E2E tests. + +### Step 3: Run the install generator ```sh bin/rails generate agent_e2e:install @@ -57,33 +67,21 @@ This will: 5. Update `.gitignore` to exclude `node_modules`, `failures.md`, and `screenshots` 6. Run `npm install` in `agent-tests/` 7. Install the Chromium browser for Playwright +8. **Generate `db/test_seeds.rb`** by analyzing your models with OpenAI -### Step 3: Set up your environment +The generator reads all your models and `db/schema.rb`, then calls OpenAI to generate `db/test_seeds.rb` with realistic seed data for E2E testing — including a QA user (`qa@example.com` / `Password123!`), associated records, and proper handling of validations, enums, and Devise. -Add your OpenAI API key to your `.env` file in the Rails root: +To regenerate seeds after adding new models: -```env -OPENAI_API_KEY=sk-proj-... +```sh +bin/rails generate agent_e2e:test_seeds ``` -> **Important:** Make sure `.env` is in your `.gitignore` (Rails apps typically ignore `/.env*` by default). - -### Step 4: Create a QA test user +The generated file uses `find_or_create_by!` so it's safe to run multiple times. Review and adjust it as needed. -Add a seed user for the agent to log in with. In `db/seeds.rb`: - -```ruby -if Rails.env.local? - User.find_or_create_by!(email: "qa@example.com") do |user| - user.password = "Password123!" - user.password_confirmation = "Password123!" - # If using Devise confirmable: - user.confirmed_at = Time.current - end -end -``` +You can override the AI model used with the `SEED_AI_MODEL` environment variable (defaults to `o3`). -### Step 5: Add `data-testid` attributes to your views +### Step 4: Add `data-testid` attributes to your views The agent can interact with elements by visible text, labels, and roles — but `data-testid` attributes make interactions more reliable, especially for buttons and form elements. @@ -95,7 +93,7 @@ The agent can interact with elements by visible text, labels, and roles — but **Recommendation:** Add `data-testid` to every interactive element (buttons, links, inputs, selects). This doesn't affect your production HTML and makes tests much more stable. -### Step 6: Handle asset compilation (if needed) +### Step 5: Handle asset compilation (if needed) If your app uses Tailwind CSS, esbuild, or another build step, uncomment the relevant line in `bin/e2e`: @@ -140,7 +138,7 @@ bin/e2e ``` This will: -1. Prepare and seed the test database +1. Prepare the test database and seed it with `db/test_seeds.rb` 2. Start a Rails server on port 3001 (configurable via `PORT`) 3. Run each test case with the AI agent 4. Print a summary of results @@ -148,6 +146,8 @@ This will: 6. Save screenshots on failure to `agent-tests/screenshots/` 7. Clean up: stop the server and reset the database +> If `db/test_seeds.rb` doesn't exist, the script will stop and show you the steps to generate it. + ## Configuration All configuration is via environment variables. Set them in `.env` or pass them directly: @@ -155,7 +155,8 @@ All configuration is via environment variables. Set them in `.env` or pass them | Variable | Default | Description | |---|---|---| | `OPENAI_API_KEY` | *(required)* | Your OpenAI API key | -| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the agent | +| `AI_MODEL` | `gpt-5.1` | OpenAI model to use for the test agent | +| `SEED_AI_MODEL` | `o3` | OpenAI model to use for generating test seeds | | `BASE_URL` | `http://localhost:3000` | Base URL of the app (overridden to port 3001 by `bin/e2e`) | | `MAX_STEPS` | `25` | Maximum steps per test before timeout | | `ACTION_TIMEOUT` | `8000` | Timeout in ms for each browser action | @@ -176,12 +177,16 @@ All configuration is via environment variables. Set them in `.env` or pass them | File | Description | |---|---| +| `db/test_seeds.rb` | AI-generated seed data for E2E testing (generated by `agent_e2e:test_seeds`) | | `agent-tests/tests.md` | Your test cases (you write this) | | `agent-tests/failures.md` | Detailed failure reports with action history (auto-generated, gitignored) | | `agent-tests/screenshots/` | Screenshots captured on failure (auto-generated, gitignored) | ## Troubleshooting +**"db/test_seeds.rb not found"** +Run `bin/rails generate agent_e2e:test_seeds` to generate it. Make sure `OPENAI_API_KEY` is set in your `.env`. + **"No test cases found in tests.md"** Add at least one test case line (not starting with `#`) to `agent-tests/tests.md`. @@ -197,6 +202,9 @@ Add `data-testid` attributes to ambiguous elements. The agent prefers `data-test **Asset compilation issues** Uncomment the asset build step in `bin/e2e` that matches your setup. +**Regenerating seeds after model changes** +Run `bin/rails generate agent_e2e:test_seeds` again. It will overwrite the existing file. + ## License MIT diff --git a/lib/generators/agent_e2e/install/install_generator.rb b/lib/generators/agent_e2e/install/install_generator.rb index 8f9559e..411f320 100644 --- a/lib/generators/agent_e2e/install/install_generator.rb +++ b/lib/generators/agent_e2e/install/install_generator.rb @@ -106,6 +106,10 @@ def install_playwright_browsers end end + def generate_test_seeds + generate "agent_e2e:test_seeds" + end + def print_next_steps say "" say "=" * 60, :green @@ -120,10 +124,12 @@ def print_next_steps say " 2. Add data-testid attributes to key UI elements:" say ' ' say "" - say " 3. Create a QA seed user (e.g. in db/seeds.rb):" - say " User.find_or_create_by!(email: 'qa@example.com') do |u|" - say " u.password = 'Password123!'" - say " end" + if File.exist?(File.join(destination_root, "db/test_seeds.rb")) + say " 3. Review the generated db/test_seeds.rb and adjust if needed." + else + say " 3. Generate test seeds:" + say " bin/rails generate agent_e2e:test_seeds" + end say "" say " 4. Write test cases in agent-tests/tests.md (one per line):" say " - Log in with qa@example.com and verify the dashboard loads" diff --git a/lib/generators/agent_e2e/install/templates/e2e b/lib/generators/agent_e2e/install/templates/e2e index 7becf4c..5f0ff22 100644 --- a/lib/generators/agent_e2e/install/templates/e2e +++ b/lib/generators/agent_e2e/install/templates/e2e @@ -24,7 +24,26 @@ echo "==> Preparing test database..." bin/rails db:test:prepare echo "==> Seeding test database..." -bin/rails db:seed +if [ -f db/test_seeds.rb ]; then + bin/rails runner db/test_seeds.rb +else + echo "" + echo "ERROR: db/test_seeds.rb not found." + echo "" + if grep -q "OPENAI_API_KEY" .env 2>/dev/null; then + echo "Next steps:" + echo " Run the test seeds generator:" + echo " bin/rails generate agent_e2e:test_seeds" + else + echo "Next steps:" + echo " 1. Add your OpenAI API key to .env:" + echo " OPENAI_API_KEY=sk-..." + echo " 2. Run the test seeds generator:" + echo " bin/rails generate agent_e2e:test_seeds" + fi + echo "" + exit 1 +fi # Uncomment if your app needs asset precompilation (e.g. Tailwind CSS): # echo "==> Compiling assets..." diff --git a/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb new file mode 100644 index 0000000..7184adb --- /dev/null +++ b/lib/generators/agent_e2e/test_seeds/test_seeds_generator.rb @@ -0,0 +1,181 @@ +# frozen_string_literal: true + +require "rails/generators/base" +require "net/http" +require "json" +require "uri" + +module AgentE2e + module Generators + class TestSeedsGenerator < Rails::Generators::Base + desc "Analyzes your models and generates db/test_seeds.rb using OpenAI" + + def generate_test_seeds + model_files = Dir.glob(File.join(destination_root, "app/models/**/*.rb")) + if model_files.empty? + say "No models found in app/models/. Nothing to generate.", :yellow + return + end + + api_key = ENV["OPENAI_API_KEY"] + dotenv_path = File.join(destination_root, ".env") + if !api_key && File.exist?(dotenv_path) + File.readlines(dotenv_path).each do |line| + if line.strip.match?(/\AOPENAI_API_KEY=/) + api_key = line.strip.sub("OPENAI_API_KEY=", "") + break + end + end + end + + unless api_key && !api_key.empty? + say "OPENAI_API_KEY not found.", :red + say "" + say "Add it to your .env file and try again:", :yellow + say " echo 'OPENAI_API_KEY=sk-...' >> .env" + say " bin/rails generate agent_e2e:test_seeds" + return + end + + say "Found #{model_files.size} model(s). Analyzing...", :green + + schema_path = File.join(destination_root, "db/schema.rb") + schema_content = File.exist?(schema_path) ? File.read(schema_path) : nil + + models_content = model_files.map { |f| + relative = f.sub("#{destination_root}/", "") + "# #{relative}\n#{File.read(f)}" + }.join("\n\n") + + user_model = detect_user_model(model_files) + + prompt = build_seed_prompt(models_content, schema_content, user_model) + ai_seeds = call_openai(api_key, prompt) + + if ai_seeds + seeds_code = <<~RUBY + # Auto-generated by agent_e2e:test_seeds + + #{ai_seeds} + RUBY + create_file "db/test_seeds.rb", seeds_code, force: true + say "" + say "Generated db/test_seeds.rb", :green + else + fallback = build_fallback_seeds(user_model, models_content) + create_file "db/test_seeds.rb", fallback, force: true + say "" + say "Generated db/test_seeds.rb (fallback — AI generation failed)", :yellow + say "WARNING: Review db/test_seeds.rb and add seeds for your models manually.", :yellow + end + say "Review it and run your tests with: bin/e2e", :cyan + end + + private + + def detect_user_model(model_files) + model_files.each do |f| + content = File.read(f) + basename = File.basename(f, ".rb") + return basename.camelize if content.match?(/devise|has_secure_password|authenticate/i) + end + model_files.each do |f| + basename = File.basename(f, ".rb") + return basename.camelize if basename.match?(/\Auser\z|\Aaccount\z|\Aadmin\z/i) + end + "User" + end + + def build_fallback_seeds(user_model, models_content) + uses_devise = models_content.match?(/devise/) + lines = [] + lines << "# Auto-generated by agent_e2e:test_seeds (fallback)" + lines << "# AI generation failed — review and complete this file manually." + lines << "" + lines << "qa_user = #{user_model}.find_or_create_by!(email: \"qa@example.com\") do |u|" + lines << " u.password = \"Password123!\"" + lines << " u.password_confirmation = \"Password123!\"" if uses_devise + lines << " u.confirmed_at = Time.current" if models_content.match?(/confirmable/i) + lines << " # TODO: add any other required fields for #{user_model}" + lines << "end" + lines << "" + lines.join("\n") + end + + def build_seed_prompt(models_content, schema_content, user_model) + <<~PROMPT + You are a Ruby on Rails expert. Analyze the following models and database schema, + then generate seed data for end-to-end (E2E) browser testing. + + Requirements: + - The FIRST thing in the output must be creating a QA user for E2E testing: + * Model: #{user_model} + * Email: qa@example.com + * Password: Password123! + * Assign the result to a variable called `qa_user` + * Fill ALL required fields (from validations and null:false columns) with realistic values + * If the model uses Devise, set password_confirmation and confirmed_at + - Use `find_or_create_by!` to make the script idempotent (safe to run multiple times) + - Create realistic sample data for ALL models, respecting associations and validations + - Handle belongs_to associations by creating parent records first (topological order) + - Include enough variety (at least 2-3 records per model where it makes sense) + - Add brief comments explaining what each section seeds + - Wrap everything in an ActiveRecord::Base.transaction block + - Handle any enum values you find in the models + - If there's a role/admin system, create an admin user (different from qa_user) + - The data should be meaningful and realistic, not placeholder text + - If models use Devise, handle `confirmed_at`, `password_confirmation`, etc. + - If models have file attachments (ActiveStorage), skip those fields + - Output ONLY valid Ruby code — no markdown fences, no explanations + + ## Models: + + #{models_content} + + #{schema_content ? "## Database Schema:\n\n#{schema_content}" : ""} + + Generate the seed code: + PROMPT + end + + def call_openai(api_key, prompt) + model = ENV["SEED_AI_MODEL"] || "gpt-5.3-codex" + uri = URI("https://api.openai.com/v1/responses") + http = Net::HTTP.new(uri.host, uri.port) + http.use_ssl = true + http.read_timeout = 120 + + request = Net::HTTP::Post.new(uri) + request["Content-Type"] = "application/json" + request["Authorization"] = "Bearer #{api_key}" + + request.body = JSON.generate({ + model: model, + input: prompt + }) + + say "Calling #{model} (this may take a moment)...", :cyan + response = http.request(request) + + if response.code.to_i == 200 + data = JSON.parse(response.body) + extract_ruby_code(data["output_text"]) + else + say "OpenAI API error (#{response.code}): #{response.body[0..300]}", :red + nil + end + rescue => e + say "Error calling OpenAI API: #{e.message}", :red + nil + end + + def extract_ruby_code(content) + return nil if content.nil? || content.strip.empty? + code = content.strip + code = code.sub(/\A```ruby\n?/, "").sub(/\n?```\z/, "") + code = code.sub(/\A```\n?/, "").sub(/\n?```\z/, "") + code.strip + "\n" + end + end + end +end