diff --git a/AGENTS.md b/AGENTS.md index 04a43a9..288d052 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,17 +1,20 @@ # Doc Detective Core - AI Coding Assistant Guide ## Project Overview + Doc Detective Core is a low-code documentation testing framework that validates docs through browser automation, shell commands, HTTP requests, and content analysis. It's the engine behind the Doc Detective CLI tool. ## Architecture ### Test Execution Flow + 1. **Input Resolution** (`doc-detective-resolver`): Detects tests from docs/specs → resolves to executable format 2. **Test Orchestration** (`src/tests.js`): `runSpecs()` → spec → test → context → step hierarchy 3. **Browser Automation** (`src/tests.js`): Appium server manages WebDriver sessions (Chrome/Firefox/Safari) 4. **Step Execution**: Each step type has dedicated handler in `src/tests/` (e.g., `httpRequest.js`, `runShell.js`) ### Key Components + - **`src/index.js`**: Entry point exposing `runTests()` function - **`src/tests.js`**: Core test runner with Appium/WebDriver orchestration (600+ LOC orchestrator) - **`src/config.js`**: Configuration validation, environment detection, browser discovery @@ -22,6 +25,7 @@ Doc Detective Core is a low-code documentation testing framework that validates ## Critical Workflows ### Running Tests Locally + ```bash npm test # Run full test suite (mocha) node dev # Development/manual testing @@ -29,9 +33,11 @@ npm run depcheck # Check for unused dependencies ``` ### CI/CD Pipeline + Three GitHub Actions workflows automate releases and testing: 1. **Auto Dev Release** (`auto-dev-release.yml`): Triggers on push to `main` + - Skips on `[skip ci]` commits, release commits, or docs-only changes - Increments dev version (`3.4.0-dev.1` → `3.4.0-dev.2`) - Publishes to npm with `dev` tag @@ -39,6 +45,7 @@ Three GitHub Actions workflows automate releases and testing: - **Version strategy**: Checks npm for latest dev number, increments, updates `package.json`, commits with `[skip ci]`, creates git tag 2. **Test & Publish** (`npm-test.yaml`): Cross-platform testing + release publishing + - **Test matrix**: Ubuntu/Windows/macOS × Node 18/20/22/24 (15 min timeout) - **Triggers**: Push to `main`, PRs (opened/reopened/synced), manual dispatch - **On release publish**: Runs `npm publish` to npm registry @@ -51,18 +58,21 @@ Three GitHub Actions workflows automate releases and testing: - **Release notes**: Aggregates merged PRs since last tag + resolver release notes ### Browser Management + - **Post-install** (`scripts/postinstall.js`): Auto-downloads Chrome/Firefox/ChromeDriver to `browser-snapshots/` - Browsers MUST match platform (detected via `@puppeteer/browsers`) - Appium drivers installed: `chromium`, `gecko`, `safari` (Mac only) - **Timeout**: All drivers default to 10 minutes (`newCommandTimeout: 600`) ### Version Management + - **Dev releases**: `X.Y.Z-dev.N` format (auto-incremented on every main push) - **Stable releases**: Manual GitHub releases trigger npm publish - **Dependency sync**: Resolver updates trigger automated core updates - **Commit conventions**: Use `[skip ci]` to avoid triggering auto-dev-release ### Adding New Step Types + 1. Create handler in `src/tests/[actionName].js` exporting async function 2. Add action to `driverActions` array in `src/tests.js` if requires browser 3. Add case in `runStep()` switch statement @@ -72,53 +82,63 @@ Three GitHub Actions workflows automate releases and testing: ## Project Conventions ### Test Structure + Tests follow nested hierarchy: -``` + +```text spec (file) → test → context (browser/platform combo) → step (action) ``` + - **Contexts** run serially and skip if platform/browser unsupported - **Steps** skip after first failure in context (stepExecutionFailed flag) - **Unsafe steps** (`step.unsafe = true`) require `config.allowUnsafeSteps = true` ### Configuration Patterns + - Config validated via `doc-detective-common` schemas (`validate({ schemaKey: "config_v3", object })`) - File types (`markdown`, `asciidoc`, `html`) define inline test detection regexes - Environment variables loaded via `loadEnvs()` and replaced via `replaceEnvs()` using `$VAR_NAME` syntax - OpenAPI definitions loaded and dereferenced at config time (stored in `config.integrations.openApi[].definition`) ### Expression System (`src/expressions.js`) + - **Meta values**: `$$response.body.users[0].name` accesses runtime data - **Embedded expressions**: `"User ID is {{$$response.body.id}}"` for string interpolation - **Operators**: `jq($$response.body, ".users[0].name")`, `extract($$output, "ID: (\d+)")` - Variables set via `step.variables = { MY_VAR: "$$response.body.token" }` → stored as env vars ### OpenAPI Integration + - **Example compilation**: Extracts request/response examples from OpenAPI spec - **Schema validation**: Uses AJV to validate payloads against OpenAPI schemas - **Mock responses**: Set `step.httpRequest.openApi.mockResponse = true` to skip actual HTTP call - Operations referenced by `operationId` (e.g., `step.httpRequest.openApi.operationId = "getUserById"`) ### Error Handling & Logging + - Use `log(config, level, message)` where level = "debug"|"info"|"warning"|"error" - Config object MUST be passed as first param to log functions - Step failures should return `{ status: "FAIL", description: "Detailed error message" }` - Always handle driver cleanup in try/finally blocks ## Common Pitfalls + - **Appium must be running** for any driver-based step (auto-started if needed, but check `appiumRequired` flag) - **Browser paths are platform-specific**: Use `getAvailableApps()` to detect installed browsers - **JSON pointer syntax**: Use `#/path/to/field` after meta value (e.g., `$$response#/body/users/0/name`) - **Viewport vs Window size**: `setViewportSize()` calculates delta to set inner dimensions -- **Percentage variation** (`maxVariation`): Value is decimal (0.1 = 10%), but comparison uses percentage (multiply by 100) +- **Fractional variation** (`maxVariation`): Value is a decimal fraction (0.1 = 10% tolerance). Comparisons use fractions directly. - **File overwrite modes**: "false" (never), "true" (always), "aboveVariation" (only if content differs > maxVariation) ## Testing Patterns + - Tests in `test/core.test.js` use mocha with `this.timeout(0)` for indefinite timeout - Test server runs on port 8092 (`test/server/`) for HTTP request tests - Artifacts stored in `test/artifacts/` (specs, configs, test files) - Use `fs.writeFileSync()` + `fs.unlinkSync()` for temp test files in try/finally blocks ## Dependencies to Know + - `webdriverio` (8.45.0): WebDriver protocol implementation - `appium`: Browser automation server - `@puppeteer/browsers`: Browser binary management @@ -129,6 +149,7 @@ spec (file) → test → context (browser/platform combo) → step (action) - `doc-detective-resolver`: Test detection/resolution ## Documentation + - Main docs at https://doc-detective.com - Schemas at https://doc-detective.com/reference/schemas/ - Report issues to https://github.com/doc-detective/doc-detective-core/issues diff --git a/src/tests/httpRequest.js b/src/tests/httpRequest.js index e895afe..b2a8fcb 100644 --- a/src/tests/httpRequest.js +++ b/src/tests/httpRequest.js @@ -4,7 +4,7 @@ const fs = require("fs"); const path = require("path"); const Ajv = require("ajv"); const { getOperation, loadDescription } = require("../openapi"); -const { log, calculatePercentageDifference, replaceEnvs } = require("../utils"); +const { log, calculateFractionalDifference, replaceEnvs } = require("../utils"); exports.httpRequest = httpRequest; @@ -302,16 +302,18 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) { // Validate required fields in response if (step.httpRequest.response?.required?.length > 0) { const missingFields = []; - + for (const fieldPath of step.httpRequest.response.required) { if (!fieldExistsAtPath(response.data, fieldPath)) { missingFields.push(fieldPath); } } - + if (missingFields.length > 0) { result.status = "FAIL"; - result.description += ` Missing required fields: ${missingFields.join(", ")}`; + result.description += ` Missing required fields: ${missingFields.join( + ", " + )}`; return result; } } @@ -451,31 +453,35 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) { // Read existing file const existingFile = fs.readFileSync(filePath, "utf8"); - // Calculate percentage diff between existing file content and command output content, not length - const percentDiff = calculatePercentageDifference( + // Calculate fractional diff between existing file content and command output content, not length + const fractionalDiff = calculateFractionalDifference( existingFile, JSON.stringify(response.data, null, 2) ); - log(config, "debug", `Percentage difference: ${percentDiff}%`); + log(config, "debug", `Fractional difference: ${fractionalDiff}`); - if (percentDiff > step.httpRequest.maxVariation * 100) { + if (fractionalDiff > step.httpRequest.maxVariation) { if (step.httpRequest.overwrite == "aboveVariation") { // Overwrite file await fs.promises.writeFile( filePath, JSON.stringify(response.data, null, 2) ); + result.description += ` Saved response to file.`; } - result.status = "FAIL"; - result.description += ` The percentage difference between the existing file content and command output content (${percentDiff}%) is greater than the max accepted variation (${ - step.httpRequest.maxVariation * 100 - }%).`; + result.status = "WARNING"; + result.description += ` The difference between the existing saved response and the new response (${fractionalDiff.toFixed( + 2 + )}) is greater than the max accepted variation (${ + step.httpRequest.maxVariation + }).`; return result; } if (step.httpRequest.overwrite == "true") { // Overwrite file fs.writeFileSync(filePath, JSON.stringify(response.data, null, 2)); + result.description += ` Saved response to file.`; } } } @@ -487,7 +493,7 @@ async function httpRequest({ config, step, openApiDefinitions = [] }) { /** * Checks if a field exists at the specified path in an object. * Supports dot notation and array indices. - * + * * @param {Object} obj - The object to search * @param {string} path - The field path (e.g., "user.profile.name" or "items[0].id") * @returns {boolean} - True if the field exists, false otherwise @@ -496,13 +502,13 @@ function fieldExistsAtPath(obj, path) { // Parse the path into segments // Handle both dot notation and array brackets const segments = path.match(/[^.[\]]+/g); - + if (!segments) { return false; } - + let current = obj; - + // Traverse each segment for (const segment of segments) { // Treat as array index only if the segment is purely numeric (e.g., "0", "12") @@ -516,13 +522,17 @@ function fieldExistsAtPath(obj, path) { } else { // Object property access // Use 'in' operator to check existence (works for null/undefined values) - if (typeof current !== 'object' || current === null || !(segment in current)) { + if ( + typeof current !== "object" || + current === null || + !(segment in current) + ) { return false; } current = current[segment]; } } - + return true; } diff --git a/src/tests/runShell.js b/src/tests/runShell.js index ee2cce0..8e1925e 100644 --- a/src/tests/runShell.js +++ b/src/tests/runShell.js @@ -2,7 +2,7 @@ const { validate } = require("doc-detective-common"); const { spawnCommand, log, - calculatePercentageDifference, + calculateFractionalDifference, } = require("../utils"); const fs = require("fs"); const path = require("path"); @@ -94,7 +94,10 @@ async function runShell({ config, step }) { step.runShell.stdio.endsWith("/") ) { const regex = new RegExp(step.runShell.stdio.slice(1, -1)); - if (!regex.test(result.outputs.stdio.stdout) && !regex.test(result.outputs.stdio.stderr)) { + if ( + !regex.test(result.outputs.stdio.stdout) && + !regex.test(result.outputs.stdio.stderr) + ) { result.status = "FAIL"; result.description = `Couldn't find expected output (${step.runShell.stdio}) in actual output (stdout or stderr).`; } @@ -134,28 +137,34 @@ async function runShell({ config, step }) { // Read existing file const existingFile = fs.readFileSync(filePath, "utf8"); - // Calculate percentage diff between existing file content and command output content, not length - const percentDiff = calculatePercentageDifference( + // Calculate fractional diff between existing file content and command output content, not length + const fractionalDiff = calculateFractionalDifference( existingFile, result.outputs.stdio.stdout ); - log(config, "debug", `Percentage difference: ${percentDiff}%`); + log(config, "debug", `Fractional difference: ${fractionalDiff}`); - if (percentDiff > step.runShell.maxVariation) { + if (fractionalDiff > step.runShell.maxVariation) { if (step.runShell.overwrite == "aboveVariation") { // Overwrite file fs.writeFileSync(filePath, result.outputs.stdio.stdout); + result.description += ` Saved output to file.`; } result.status = "WARNING"; result.description = result.description + - ` The percentage difference between the existing file content and command output content (${percentDiff}%) is greater than the max accepted variation (${step.runShell.maxVariation}%).`; + ` The difference between the existing output and the new output (${fractionalDiff.toFixed( + 2 + )}) is greater than the max accepted variation (${ + step.runShell.maxVariation + }).`; return result; } if (step.runShell.overwrite == "true") { // Overwrite file fs.writeFileSync(filePath, result.outputs.stdio.stdout); + result.description += ` Saved output to file.`; } } } diff --git a/src/tests/saveScreenshot.js b/src/tests/saveScreenshot.js index 0000652..b339df6 100644 --- a/src/tests/saveScreenshot.js +++ b/src/tests/saveScreenshot.js @@ -130,7 +130,10 @@ async function saveScreenshot({ config, step, driver }) { } if (element) result.outputs.element = findResult.outputs.element; // Determine if element bounding box + padding is within viewport - const rect = { ...(await element.getLocation()), ...(await element.getSize()) }; + const rect = { + ...(await element.getLocation()), + ...(await element.getSize()), + }; const viewport = await driver.execute(() => { return { width: window.innerWidth, @@ -162,13 +165,17 @@ async function saveScreenshot({ config, step, driver }) { // Scroll element into view at top-left with padding await driver.execute( (el, pad) => { - el.scrollIntoView({ block: "start", inline: "start", behavior: "instant" }); + el.scrollIntoView({ + block: "start", + inline: "start", + behavior: "instant", + }); window.scrollBy(-pad.left, -pad.top); }, element, padding ); - + // Wait for scroll to complete await driver.pause(100); } @@ -217,7 +224,7 @@ async function saveScreenshot({ config, step, driver }) { x: bounds.left, y: bounds.top, width: bounds.width, - height: bounds.height + height: bounds.height, }; }, element); log(config, "debug", { rect }); @@ -286,7 +293,7 @@ async function saveScreenshot({ config, step, driver }) { fs.renameSync(filePath, existFilePath); return result; } - let percentDiff; + let fractionalDiff; // Perform numerical pixel diff with pixelmatch if (step.screenshot.maxVariation) { @@ -336,28 +343,30 @@ async function saveScreenshot({ config, step, driver }) { height, { threshold: 0.0005 } ); - percentDiff = (numDiffPixels / (width * height)) * 100; + fractionalDiff = numDiffPixels / (width * height); log(config, "debug", { totalPixels: width * height, numDiffPixels, - percentDiff, + fractionalDiff, }); - if (percentDiff > step.screenshot.maxVariation) { + if (fractionalDiff > step.screenshot.maxVariation) { if (step.screenshot.overwrite == "aboveVariation") { // Replace old file with new file fs.renameSync(filePath, existFilePath); } result.status = "WARNING"; - result.description += ` Screenshots are beyond maximum accepted variation: ${percentDiff.toFixed( + result.description += ` The difference between the existing screenshot and new screenshot (${fractionalDiff.toFixed( 2 - )}%.`; + )}) is greater than the max accepted variation (${ + step.screenshot.maxVariation + }).`; return result; } else { - result.description += ` Screenshots are within maximum accepted variation: ${percentDiff.toFixed( + result.description += ` Screenshots are within maximum accepted variation: ${fractionalDiff.toFixed( 2 - )}%.`; + )}.`; if (step.screenshot.overwrite != "true") { fs.unlinkSync(filePath); } diff --git a/src/utils.js b/src/utils.js index a143ea8..3aa545e 100644 --- a/src/utils.js +++ b/src/utils.js @@ -12,7 +12,7 @@ exports.replaceEnvs = replaceEnvs; exports.spawnCommand = spawnCommand; exports.inContainer = inContainer; exports.cleanTemp = cleanTemp; -exports.calculatePercentageDifference = calculatePercentageDifference; +exports.calculateFractionalDifference = calculateFractionalDifference; exports.fetchFile = fetchFile; exports.isRelativeUrl = isRelativeUrl; @@ -264,11 +264,20 @@ async function inContainer() { return false; } -function calculatePercentageDifference(text1, text2) { +/** + * Calculates the fractional difference between two strings using Levenshtein distance. + * @param {string} text1 - First string to compare + * @param {string} text2 - Second string to compare + * @returns {number} Fractional difference between 0 and 1, where 0 means identical + * and 1 means completely different. Compare against maxVariation + * thresholds directly (e.g., 0.1 for 10% tolerance). + */ +function calculateFractionalDifference(text1, text2) { const distance = llevenshteinDistance(text1, text2); const maxLength = Math.max(text1.length, text2.length); - const percentageDiff = (distance / maxLength) * 100; - return percentageDiff.toFixed(2); // Returns the percentage difference as a string with two decimal places + if (maxLength === 0) return 0; // Both strings are empty + const fractionalDiff = distance / maxLength; + return fractionalDiff; } function llevenshteinDistance(s, t) {