diff --git a/docs/get-started/concepts.md b/docs/get-started/concepts.md index c816fdda..83b4aebc 100644 --- a/docs/get-started/concepts.md +++ b/docs/get-started/concepts.md @@ -41,6 +41,17 @@ An action is the task performed in a step. Doc Detective supports a variety of a A [context](/docs/references/schemas/context) consists of an application and platforms that support the tests. +## Debug Mode + +Doc Detective provides comprehensive debugging capabilities through a hierarchical debug mode system. You can control debugging behavior at the config, specification, test, and step levels. Debug modes include: + +- `false` - Disable debugging (default) +- `true` - Enable debugging with breakpoint support +- `"stepThrough"` - Pause at every step waiting for user input + +Learn more about [debugging tests](/docs/get-started/debugging). + ## Next steps - [Create your first test](/docs/get-started/create-your-first-test) +- [Debug your tests](/docs/get-started/debugging) diff --git a/docs/get-started/create-your-first-test.md b/docs/get-started/create-your-first-test.md index 96862e28..47c80873 100644 --- a/docs/get-started/create-your-first-test.md +++ b/docs/get-started/create-your-first-test.md @@ -180,4 +180,5 @@ Additionally, you should see a new image file named `example.png` saved in your - [Explore sample tests](/docs/get-started/sample-tests) to see more examples. - [Learn more about available actions](/docs/category/actions) to expand your test capabilities. +- [Learn about debugging tests](/docs/get-started/debugging) to troubleshoot issues and step through test execution. - Try modifying this test to check for different elements or add more steps. diff --git a/docs/get-started/debugging.md b/docs/get-started/debugging.md new file mode 100644 index 00000000..02666d6a --- /dev/null +++ b/docs/get-started/debugging.md @@ -0,0 +1,308 @@ +--- +sidebar_position: 6 +--- + +# Debugging Tests + +Doc Detective provides comprehensive debugging capabilities to help you troubleshoot and step through your tests. The debug mode system operates on a hierarchical structure, allowing you to control debugging behavior at multiple levels. + +## Debug Mode Hierarchy + +Debug settings follow a hierarchical override system: + +1. **Config level** - Global default (`config.debug`) +2. **Spec level** - Overrides config (`spec.debug`) +3. **Test level** - Overrides spec/config (`test.debug`) +4. **Step level** - Individual breakpoints (`step.breakpoint`) + +Each level can override the settings from higher levels, giving you fine-grained control over debugging behavior. + +## Debug Mode Values + +All debug properties support three values: + +- `false` - Disable debugging (default) +- `true` - Enable debugging with breakpoint support +- `"stepThrough"` - Pause at every step waiting for user input + +## Configuration Level + +Set global debug defaults in your `.doc-detective.json` configuration file: + +```json +{ + "debug": true, + "input": "tests/", + "output": "results/" +} +``` + +## Specification Level + +Override global debug settings for an entire test specification: + +```json +{ + "specId": "api-tests", + "debug": "stepThrough", + "tests": [ + { + "steps": [ + { + "checkLink": "https://api.example.com" + } + ] + } + ] +} +``` + +## Test Level + +Control debugging for individual tests, overriding both config and spec settings: + +```json +{ + "tests": [ + { + "testId": "critical-test", + "debug": true, + "steps": [ + { + "goTo": "https://example.com" + }, + { + "find": { + "selector": "[name='q']" + } + } + ] + } + ] +} +``` + +## Step Level Breakpoints + +Set breakpoints on individual steps regardless of higher-level debug settings: + +```json +{ + "tests": [ + { + "testId": "detailed-test", + "debug": "stepThrough", + "steps": [ + { + "goTo": "https://example.com" + }, + { + "checkLink": "https://api.example.com", + "breakpoint": true + }, + { + "screenshot": true + } + ] + } + ] +} +``` + +## Debugging Workflows + +### Step-Through Debugging + +Use `"stepThrough"` mode to pause at every step: + +```json +{ + "testId": "step-by-step-test", + "debug": "stepThrough", + "steps": [ + { + "goTo": "https://example.com" + }, + { + "find": { + "selector": "input[type='search']" + } + }, + { + "type": "search query" + } + ] +} +``` + +### Selective Debugging + +Enable debugging globally but disable it for specific tests: + +```json +{ + "debug": true, + "tests": [ + { + "testId": "fast-test", + "debug": false, + "steps": [ + { + "checkLink": "https://example.com" + } + ] + }, + { + "testId": "debug-test", + "steps": [ + { + "goTo": "https://example.com", + "breakpoint": true + } + ] + } + ] +} +``` + +### Critical Step Breakpoints + +Set breakpoints only on critical steps: + +```json +{ + "tests": [ + { + "testId": "api-validation", + "steps": [ + { + "loadVariables": ".env" + }, + { + "httpRequest": { + "url": "https://api.example.com/users", + "method": "POST" + }, + "breakpoint": true + }, + { + "screenshot": true + } + ] + } + ] +} +``` + +## Best Practices + +### Development vs Production + +Use different configurations for development and production: + +**Development (.doc-detective.dev.json):** +```json +{ + "debug": "stepThrough", + "input": "tests/", + "logLevel": "debug" +} +``` + +**Production (.doc-detective.json):** +```json +{ + "debug": false, + "input": "tests/", + "logLevel": "info" +} +``` + +### Debugging Complex Tests + +For complex tests, use a combination of approaches: + +```json +{ + "specId": "complex-workflow", + "debug": true, + "tests": [ + { + "testId": "setup-test", + "debug": false, + "steps": [ + { + "loadVariables": ".env" + } + ] + }, + { + "testId": "main-test", + "debug": "stepThrough", + "steps": [ + { + "goTo": "https://app.example.com" + }, + { + "find": { + "selector": "#login-form" + }, + "breakpoint": true + }, + { + "type": { + "keys": ["$USERNAME$"] + } + } + ] + } + ] +} +``` + +### Debugging Failed Tests + +When tests fail, add breakpoints around the failing step: + +```json +{ + "steps": [ + { + "goTo": "https://example.com" + }, + { + "find": { + "selector": "#problematic-element" + }, + "breakpoint": true + }, + { + "click": { + "selector": "#problematic-element" + }, + "breakpoint": true + } + ] +} +``` + +## Running Tests with Debug Mode + +Execute tests with debug configuration: + +```bash +# Use specific config with debug enabled +npx doc-detective --config .doc-detective.dev.json + +# Override debug setting via command line +npx doc-detective --debug true + +# Step through all tests +npx doc-detective --debug stepThrough +``` + +## Next Steps + +- Learn about [test configuration](/docs/get-started/config/contexts) +- Explore [action reference](/docs/get-started/actions) +- See [troubleshooting guide](/docs/get-started/resources) \ No newline at end of file diff --git a/docs/get-started/intro.md b/docs/get-started/intro.md index 699fd9ce..d69bdc20 100644 --- a/docs/get-started/intro.md +++ b/docs/get-started/intro.md @@ -40,6 +40,10 @@ Doc Detective’s core strength is the ability to systematically check each and When Doc Detective runs tests, it can take screenshots and make recordings. You can include this media in the documentation to complement written processes, aiding the readers who prefer visual media while making sure your videos and images are up-to-date. +### Debug and troubleshoot tests + +Doc Detective provides comprehensive debugging capabilities with breakpoint support and step-through debugging. You can control debugging behavior at multiple levels (config, specification, test, and step) to help troubleshoot failing tests and understand test execution flow. + ## What can't Doc Detective do? It’s important to know your limits, and Doc Detective’s too. diff --git a/docs/get-started/resources.md b/docs/get-started/resources.md index 4ee766df..2ed80bd2 100644 --- a/docs/get-started/resources.md +++ b/docs/get-started/resources.md @@ -8,6 +8,7 @@ There are a handful of resources to help you write and run tests: - The [Action Builder](/app) prototype is an interactive tool to help you build test actions, using the same action definitions and validations as Doc Detective itself. - The [Schemas](/docs/category/schemas) include detailed information about the schemas and actions available in Doc Detective. +- The [Debugging Guide](/docs/get-started/debugging) provides comprehensive information about debugging tests with breakpoints and step-through debugging. - The [Discord](https://discord.gg/uAfSjVH7yr) community is a great place to ask questions and get help from other users and the maintainers. - The [Issue tracker](https://github.com/doc-detective/doc-detective/issues) is the best place to report bugs and request new features. - The [Contribution guide](https://github.com/doc-detective/doc-detective/blob/main/CONTRIBUTIONS.md) includes information about how to contribute to the project. diff --git a/docs/references/schemas/common.md b/docs/references/schemas/common.md index a64f609a..3a6ac8fa 100644 --- a/docs/references/schemas/common.md +++ b/docs/references/schemas/common.md @@ -16,6 +16,7 @@ Field | Type | Description | Default $schema | string | Optional. JSON Schema for this object.

Accepted values: `https://raw.githubusercontent.com/doc-detective/common/refs/heads/main/dist/schemas/step_v3.schema.json` | stepId | string | Optional. ID of the step. | description | string | Optional. Description of the step. | +breakpoint | boolean | Optional. Whether to pause execution at this step when debugging is enabled. When `true`, execution will pause at this step regardless of the debug mode setting, waiting for user input before continuing. | `false` unsafe | boolean | Optional. Whether or not the step may be unsafe. Unsafe steps may perform actions that could modify the system or environment in unexpected ways. Unsafe steps are only performed within Docker containers or if unsafe steps are enabled with the `allowUnsafeSteps` config property or the `--allow-unsafe` flag. | `false` outputs | object(Outputs (step)) | Optional. Outputs from step processes and user-defined expressions. Use the `outputs` object to reference outputs in subsequent steps. If a user-defined output matches the key for a step-defined output, the user-defined output takes precedence. | ``{}`` variables | object(Variables (step)) | Optional. Environment variables to set from user-defined expressions. | ``{}`` diff --git a/docs/references/schemas/config.md b/docs/references/schemas/config.md index 86624a47..34ea5835 100644 --- a/docs/references/schemas/config.md +++ b/docs/references/schemas/config.md @@ -20,6 +20,7 @@ beforeAny | one of:
- string
- array of string | Optional. Path(s) to te afterAll | one of:
- string
- array of string | Optional. Path(s) to test specifications to perform after those specified by `input`. Useful for cleaning up testing environments. | detectSteps | boolean | Optional. Whether or not to detect steps in input files based on defined markup. | `true` allowUnsafeSteps | boolean | Optional. Whether or not to run potentially unsafe steps, such as those that might modify files or system state. | +debug | one of:
- boolean
- string | Optional. Enable debugging mode globally. `true` allows pausing on breakpoints, waiting for user input before continuing. `stepThrough` pauses at every step, waiting for user input before continuing. `false` disables all debugging.

Accepted values: `false`, `true`, `stepThrough` | `false` logLevel | string | Optional. Amount of detail to output when performing an operation.

Accepted values: `silent`, `error`, `warning`, `info`, `debug` | `info` runOn | array of object([context](/docs/references/schemas/context)) | Optional. Contexts to run the test in. Overrides contexts defined at the config and spec levels. | fileTypes | array of one of: string, object([File type (custom)](/docs/references/schemas/file-type-custom)), object([File type (executable)](/docs/references/schemas/file-type-executable)) | Optional. Configuration for file types and their markup detection. | ``["markdown","asciidoc","html"]`` @@ -118,3 +119,20 @@ environment | object([Environment details](/docs/references/schemas/environment- } } ``` + +```json +{ + "debug": true, + "input": "tests/", + "output": "results/", + "logLevel": "debug" +} +``` + +```json +{ + "debug": "stepThrough", + "input": "tests/critical/", + "allowUnsafeSteps": true +} +``` diff --git a/docs/references/schemas/specification.md b/docs/references/schemas/specification.md index 29120a65..dba80179 100644 --- a/docs/references/schemas/specification.md +++ b/docs/references/schemas/specification.md @@ -13,6 +13,7 @@ description | string | Optional. Description of the test specification. | specPath | string | Optional. Path to the test specification. | contentPath | string | Optional. Path to the content that the specification is associated with. | runOn | array of object([context](/docs/references/schemas/context)) | Optional. Contexts to run the test in. Overrides contexts defined at the config and spec levels. | +debug | one of:
- boolean
- string | Optional. Enable debugging mode for this specification. Overrides the debug setting from config level. `true` allows pausing on breakpoints, waiting for user input before continuing. `stepThrough` pauses at every step, waiting for user input before continuing. `false` disables all debugging for this specification.

Accepted values: `false`, `true`, `stepThrough` | openApi | array of unknown | Optional. No description provided. | tests | array of object([test](/docs/references/schemas/test)) | Required. [Tests](test) to perform. | @@ -182,3 +183,42 @@ tests | array of object([test](/docs/references/schemas/test)) | Required. [Test ] } ``` + +```json +{ + "specId": "debug-spec-basic", + "description": "Specification with debug mode enabled", + "debug": true, + "tests": [ + { + "steps": [ + { + "checkLink": "https://www.google.com" + } + ] + } + ] +} +``` + +```json +{ + "specId": "debug-spec-step-through", + "description": "Specification with step-through debug mode", + "debug": "stepThrough", + "tests": [ + { + "steps": [ + { + "goTo": "https://www.google.com" + }, + { + "find": { + "selector": "[name='q']" + } + } + ] + } + ] +} +``` diff --git a/docs/references/schemas/test.md b/docs/references/schemas/test.md index 62ecfc09..3d7928f0 100644 --- a/docs/references/schemas/test.md +++ b/docs/references/schemas/test.md @@ -19,6 +19,7 @@ runOn | array of object([context](/docs/references/schemas/context)) | Optional. openApi | array of unknown | Optional. No description provided. | before | string | Optional. Path to a test specification to perform before this test, while maintaining this test's context. Useful for setting up testing environments. Only the `steps` property is used from the first test in the setup spec. | after | string | Optional. Path to a test specification to perform after this test, while maintaining this test's context. Useful for cleaning up testing environments. Only the `steps` property is used from the first test in the cleanup spec. | +debug | one of:
- boolean
- string | Optional. Enable debugging mode for this test. Overrides the debug setting from config and spec levels. `true` allows pausing on breakpoints, waiting for user input before continuing. `stepThrough` pauses at every step, waiting for user input before continuing. `false` disables all debugging for this test.

Accepted values: `false`, `true`, `stepThrough` | steps | array of object(step) | Optional. Steps to perform as part of the test. Performed in the sequence defined. If one or more actions fail, the test fails. By default, if a step fails, the test stops and the remaining steps are not executed. | contexts | array of object([Resolved context](/docs/references/schemas/resolved-context)) | ReadOnly. Resolved contexts to run the test in. This is a resolved version of the `runOn` property. It is not user-defined and should not be used in test specifications. | @@ -196,3 +197,48 @@ contexts | array of object([Resolved context](/docs/references/schemas/resolved- "detectSteps": true } ``` + +```json +{ + "testId": "debug-test-basic", + "description": "Test with debug mode enabled", + "debug": true, + "steps": [ + { + "checkLink": "https://www.google.com" + } + ] +} +``` + +```json +{ + "testId": "debug-test-step-through", + "description": "Test with step-through debug mode", + "debug": "stepThrough", + "steps": [ + { + "goTo": "https://www.google.com" + }, + { + "find": { + "selector": "[name='q']" + } + } + ] +} +``` + +```json +{ + "testId": "critical-test", + "description": "Test with debug mode and step breakpoints", + "debug": "stepThrough", + "steps": [ + { + "checkLink": "https://api.example.com", + "breakpoint": true + } + ] +} +```