Standardize maxVariation logic and rename difference calculation

AGENTS.md

@@ -1,17 +1,20 @@
 # Doc Detective Core - AI Coding Assistant Guide
 
 ## Project Overview
+
 Doc Detective Core is a low-code documentation testing framework that validates docs through browser automation, shell commands, HTTP requests, and content analysis. It's the engine behind the Doc Detective CLI tool.
 
 ## Architecture
 
 ### Test Execution Flow
+
 1. **Input Resolution** (`doc-detective-resolver`): Detects tests from docs/specs → resolves to executable format
 2. **Test Orchestration** (`src/tests.js`): `runSpecs()` → spec → test → context → step hierarchy
 3. **Browser Automation** (`src/tests.js`): Appium server manages WebDriver sessions (Chrome/Firefox/Safari)
 4. **Step Execution**: Each step type has dedicated handler in `src/tests/` (e.g., `httpRequest.js`, `runShell.js`)
 
 ### Key Components
+
 - **`src/index.js`**: Entry point exposing `runTests()` function
 - **`src/tests.js`**: Core test runner with Appium/WebDriver orchestration (600+ LOC orchestrator)
 - **`src/config.js`**: Configuration validation, environment detection, browser discovery
@@ -22,23 +25,27 @@ Doc Detective Core is a low-code documentation testing framework that validates
 ## Critical Workflows
 
 ### Running Tests Locally
+
 ```bash
 npm test                    # Run full test suite (mocha)
 node dev                    # Development/manual testing
 npm run depcheck            # Check for unused dependencies
 ```
 
 ### CI/CD Pipeline
+
 Three GitHub Actions workflows automate releases and testing:
 
 1. **Auto Dev Release** (`auto-dev-release.yml`): Triggers on push to `main`
+
    - Skips on `[skip ci]` commits, release commits, or docs-only changes
    - Increments dev version (`3.4.0-dev.1` → `3.4.0-dev.2`)
    - Publishes to npm with `dev` tag
    - Users install with `npm install doc-detective-core@dev`
    - **Version strategy**: Checks npm for latest dev number, increments, updates `package.json`, commits with `[skip ci]`, creates git tag
 
 2. **Test & Publish** (`npm-test.yaml`): Cross-platform testing + release publishing
+
    - **Test matrix**: Ubuntu/Windows/macOS × Node 18/20/22/24 (15 min timeout)
    - **Triggers**: Push to `main`, PRs (opened/reopened/synced), manual dispatch
    - **On release publish**: Runs `npm publish` to npm registry
@@ -51,18 +58,21 @@ Three GitHub Actions workflows automate releases and testing:
    - **Release notes**: Aggregates merged PRs since last tag + resolver release notes
 
 ### Browser Management
+
 - **Post-install** (`scripts/postinstall.js`): Auto-downloads Chrome/Firefox/ChromeDriver to `browser-snapshots/`
 - Browsers MUST match platform (detected via `@puppeteer/browsers`)
 - Appium drivers installed: `chromium`, `gecko`, `safari` (Mac only)
 - **Timeout**: All drivers default to 10 minutes (`newCommandTimeout: 600`)
 
 ### Version Management
+
 - **Dev releases**: `X.Y.Z-dev.N` format (auto-incremented on every main push)
 - **Stable releases**: Manual GitHub releases trigger npm publish
 - **Dependency sync**: Resolver updates trigger automated core updates
 - **Commit conventions**: Use `[skip ci]` to avoid triggering auto-dev-release
 
 ### Adding New Step Types
+
 1. Create handler in `src/tests/[actionName].js` exporting async function
 2. Add action to `driverActions` array in `src/tests.js` if requires browser
 3. Add case in `runStep()` switch statement
@@ -72,53 +82,63 @@ Three GitHub Actions workflows automate releases and testing:
 ## Project Conventions
 
 ### Test Structure
+
 Tests follow nested hierarchy:
-```
+
+```text
 spec (file) → test → context (browser/platform combo) → step (action)
 ```
+
 - **Contexts** run serially and skip if platform/browser unsupported
 - **Steps** skip after first failure in context (stepExecutionFailed flag)
 - **Unsafe steps** (`step.unsafe = true`) require `config.allowUnsafeSteps = true`
 
 ### Configuration Patterns
+
 - Config validated via `doc-detective-common` schemas (`validate({ schemaKey: "config_v3", object })`)
 - File types (`markdown`, `asciidoc`, `html`) define inline test detection regexes
 - Environment variables loaded via `loadEnvs()` and replaced via `replaceEnvs()` using `$VAR_NAME` syntax
 - OpenAPI definitions loaded and dereferenced at config time (stored in `config.integrations.openApi[].definition`)
 
 ### Expression System (`src/expressions.js`)
+
 - **Meta values**: `$$response.body.users[0].name` accesses runtime data
 - **Embedded expressions**: `"User ID is {{$$response.body.id}}"` for string interpolation
 - **Operators**: `jq($$response.body, ".users[0].name")`, `extract($$output, "ID: (\d+)")`
 - Variables set via `step.variables = { MY_VAR: "$$response.body.token" }` → stored as env vars
 
 ### OpenAPI Integration
+
 - **Example compilation**: Extracts request/response examples from OpenAPI spec
 - **Schema validation**: Uses AJV to validate payloads against OpenAPI schemas
 - **Mock responses**: Set `step.httpRequest.openApi.mockResponse = true` to skip actual HTTP call
 - Operations referenced by `operationId` (e.g., `step.httpRequest.openApi.operationId = "getUserById"`)
 
 ### Error Handling & Logging
+
 - Use `log(config, level, message)` where level = "debug"|"info"|"warning"|"error"
 - Config object MUST be passed as first param to log functions
 - Step failures should return `{ status: "FAIL", description: "Detailed error message" }`
 - Always handle driver cleanup in try/finally blocks
 
 ## Common Pitfalls
+
 - **Appium must be running** for any driver-based step (auto-started if needed, but check `appiumRequired` flag)
 - **Browser paths are platform-specific**: Use `getAvailableApps()` to detect installed browsers
 - **JSON pointer syntax**: Use `#/path/to/field` after meta value (e.g., `$$response#/body/users/0/name`)
 - **Viewport vs Window size**: `setViewportSize()` calculates delta to set inner dimensions
-- **Percentage variation** (`maxVariation`): Value is decimal (0.1 = 10%), but comparison uses percentage (multiply by 100)
+- **Fractional variation** (`maxVariation`): Value is a decimal fraction (0.1 = 10% tolerance). Comparisons use fractions directly.
 - **File overwrite modes**: "false" (never), "true" (always), "aboveVariation" (only if content differs > maxVariation)
 
 ## Testing Patterns
+
 - Tests in `test/core.test.js` use mocha with `this.timeout(0)` for indefinite timeout
 - Test server runs on port 8092 (`test/server/`) for HTTP request tests
 - Artifacts stored in `test/artifacts/` (specs, configs, test files)
 - Use `fs.writeFileSync()` + `fs.unlinkSync()` for temp test files in try/finally blocks
 
 ## Dependencies to Know
+
 - `webdriverio` (8.45.0): WebDriver protocol implementation
 - `appium`: Browser automation server
 - `@puppeteer/browsers`: Browser binary management
@@ -129,6 +149,7 @@ spec (file) → test → context (browser/platform combo) → step (action)
 - `doc-detective-resolver`: Test detection/resolution
 
 ## Documentation
+
 - Main docs at https://doc-detective.com
 - Schemas at https://doc-detective.com/reference/schemas/
 - Report issues to https://github.com/doc-detective/doc-detective-core/issues

src/tests/httpRequest.js

@@ -4,7 +4,7 @@ const fs = require("fs");
 const path = require("path");
 const Ajv = require("ajv");
 const { getOperation, loadDescription } = require("../openapi");
-const { log, calculatePercentageDifference, replaceEnvs } = require("../utils");
+const { log, calculateFractionalDifference, replaceEnvs } = require("../utils");
 
 exports.httpRequest = httpRequest;
 
@@ -302,16 +302,18 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) {
   // Validate required fields in response
   if (step.httpRequest.response?.required?.length > 0) {
     const missingFields = [];
-    
+
     for (const fieldPath of step.httpRequest.response.required) {
       if (!fieldExistsAtPath(response.data, fieldPath)) {
         missingFields.push(fieldPath);
       }
     }
-    
+
     if (missingFields.length > 0) {
       result.status = "FAIL";
-      result.description += ` Missing required fields: ${missingFields.join(", ")}`;
+      result.description += ` Missing required fields: ${missingFields.join(
+        ", "
+      )}`;
       return result;
     }
   }
@@ -451,31 +453,35 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) {
       // Read existing file
       const existingFile = fs.readFileSync(filePath, "utf8");
 
-      // Calculate percentage diff between existing file content and command output content, not length
-      const percentDiff = calculatePercentageDifference(
+      // Calculate fractional diff between existing file content and command output content, not length
+      const fractionalDiff = calculateFractionalDifference(
         existingFile,
         JSON.stringify(response.data, null, 2)
       );
-      log(config, "debug", `Percentage difference: ${percentDiff}%`);
+      log(config, "debug", `Fractional difference: ${fractionalDiff}`);
 
-      if (percentDiff > step.httpRequest.maxVariation * 100) {
+      if (fractionalDiff > step.httpRequest.maxVariation) {
         if (step.httpRequest.overwrite == "aboveVariation") {
           // Overwrite file
           await fs.promises.writeFile(
             filePath,
             JSON.stringify(response.data, null, 2)
           );
+          result.description += ` Saved response to file.`;
         }
-        result.status = "FAIL";
-        result.description += ` The percentage difference between the existing file content and command output content (${percentDiff}%) is greater than the max accepted variation (${
-          step.httpRequest.maxVariation * 100
-        }%).`;
+        result.status = "WARNING";
+        result.description += ` The difference between the existing saved response and the new response (${fractionalDiff.toFixed(
+          2
+        )}) is greater than the max accepted variation (${
+          step.httpRequest.maxVariation
+        }).`;
         return result;
       }
 
       if (step.httpRequest.overwrite == "true") {
         // Overwrite file
         fs.writeFileSync(filePath, JSON.stringify(response.data, null, 2));
+        result.description += ` Saved response to file.`;
       }
     }
   }
@@ -487,7 +493,7 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) {
 /**
  * Checks if a field exists at the specified path in an object.
  * Supports dot notation and array indices.
- * 
+ *
  * @param {Object} obj - The object to search
  * @param {string} path - The field path (e.g., "user.profile.name" or "items[0].id")
  * @returns {boolean} - True if the field exists, false otherwise
@@ -496,13 +502,13 @@ function fieldExistsAtPath(obj, path) {
   // Parse the path into segments
   // Handle both dot notation and array brackets
   const segments = path.match(/[^.[\]]+/g);
-  
+
   if (!segments) {
     return false;
   }
-  
+
   let current = obj;
-  
+
   // Traverse each segment
   for (const segment of segments) {
     // Treat as array index only if the segment is purely numeric (e.g., "0", "12")
@@ -516,13 +522,17 @@ function fieldExistsAtPath(obj, path) {
     } else {
       // Object property access
       // Use 'in' operator to check existence (works for null/undefined values)
-      if (typeof current !== 'object' || current === null || !(segment in current)) {
+      if (
+        typeof current !== "object" ||
+        current === null ||
+        !(segment in current)
+      ) {
         return false;
       }
       current = current[segment];
     }
   }
-  
+
   return true;
 }
 

src/tests/runShell.js

@@ -2,7 +2,7 @@ const { validate } = require("doc-detective-common");
 const {
   spawnCommand,
   log,
-  calculatePercentageDifference,
+  calculateFractionalDifference,
 } = require("../utils");
 const fs = require("fs");
 const path = require("path");
@@ -94,7 +94,10 @@ async function runShell({ config, step }) {
       step.runShell.stdio.endsWith("/")
     ) {
       const regex = new RegExp(step.runShell.stdio.slice(1, -1));
-      if (!regex.test(result.outputs.stdio.stdout) && !regex.test(result.outputs.stdio.stderr)) {
+      if (
+        !regex.test(result.outputs.stdio.stdout) &&
+        !regex.test(result.outputs.stdio.stderr)
+      ) {
         result.status = "FAIL";
         result.description = `Couldn't find expected output (${step.runShell.stdio}) in actual output (stdout or stderr).`;
       }
@@ -134,28 +137,34 @@ async function runShell({ config, step }) {
       // Read existing file
       const existingFile = fs.readFileSync(filePath, "utf8");
 
-      // Calculate percentage diff between existing file content and command output content, not length
-      const percentDiff = calculatePercentageDifference(
+      // Calculate fractional diff between existing file content and command output content, not length
+      const fractionalDiff = calculateFractionalDifference(
         existingFile,
         result.outputs.stdio.stdout
       );
-      log(config, "debug", `Percentage difference: ${percentDiff}%`);
+      log(config, "debug", `Fractional difference: ${fractionalDiff}`);
 
-      if (percentDiff > step.runShell.maxVariation) {
+      if (fractionalDiff > step.runShell.maxVariation) {
         if (step.runShell.overwrite == "aboveVariation") {
           // Overwrite file
           fs.writeFileSync(filePath, result.outputs.stdio.stdout);
+          result.description += ` Saved output to file.`;
         }
         result.status = "WARNING";
         result.description =
           result.description +
-          ` The percentage difference between the existing file content and command output content (${percentDiff}%) is greater than the max accepted variation (${step.runShell.maxVariation}%).`;
+          ` The difference between the existing output and the new output (${fractionalDiff.toFixed(
+            2
+          )}) is greater than the max accepted variation (${
+            step.runShell.maxVariation
+          }).`;
         return result;
       }
 
       if (step.runShell.overwrite == "true") {
         // Overwrite file
         fs.writeFileSync(filePath, result.outputs.stdio.stdout);
+        result.description += ` Saved output to file.`;
       }
     }
   }