Skip to content

docs: add manual testing guide for slash command validation#1955

Merged
mnriem merged 3 commits intogithub:mainfrom
mnriem:docs/add-manual-testing-guide
Mar 24, 2026
Merged

docs: add manual testing guide for slash command validation#1955
mnriem merged 3 commits intogithub:mainfrom
mnriem:docs/add-manual-testing-guide

Conversation

@mnriem
Copy link
Collaborator

@mnriem mnriem commented Mar 24, 2026

Adds a top-level TESTING.md that describes the manual test process PR submitters must follow when their changes affect slash commands.

What's included

  • Process overview — identify affected commands → setup test project → run commands → report results
  • Local setup — editable install via uv pip install -e . + specify init --offline
  • Reporting template — copy-paste block for PR submissions with agent/OS/shell metadata
  • Agent prompt for test selection — PR submitters paste this into their agent, which analyzes git diff --name-only main and maps changed files to affected commands using:
    • Direct command template mapping
    • Transitive script dependencies (e.g., common.shcreate-new-feature.sh/speckit.specify)
    • Extension hook resolution (e.g., extension.yml before_specify hook → /speckit.specify)
    • Template consumption tracking
    • Prerequisite chain resolution

The agent's reasoning output (which files affect which commands, and why) is submitted alongside the test results in the PR.

Why

There was no documented process for validating that slash commands actually work after code changes. Automated tests cover file generation, placeholder replacement, and script execution — but nobody was verifying the full agent-mediated command invocation path. This fills that gap with a lightweight manual process.

Adds a top-level TESTING.md that describes the manual test process PR
submitters must follow when their changes affect slash commands.

Includes:
- Process overview (identify affected commands, setup, run, report)
- Local setup instructions using editable install
- Reporting template for PR submissions
- Agent prompt that analyzes changed files and determines which
  commands need testing, including transitive script dependencies
  and extension hook mappings
Copilot AI review requested due to automatic review settings March 24, 2026 15:40
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new top-level manual testing guide to ensure PRs that impact /speckit.* slash commands include end-to-end, agent-mediated validation steps and a standard reporting format.

Changes:

  • Add TESTING.md describing a manual test workflow for slash command changes.
  • Provide a copy/paste PR reporting template for manual test results.
  • Include an “agent prompt” to map changed files to affected commands and prerequisite test ordering.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mnriem
Copy link
Collaborator Author

mnriem commented Mar 24, 2026

@copilot open a new pull request to apply changes based on the comments in this thread

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 24, 2026 18:46
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

TESTING.md:22

  • /tmp/speckit-test is POSIX-specific and won’t work on Windows (and may be undesirable in some environments). Consider using a placeholder like <test-project-dir> (or showing both a Unix and Windows example), or recommending specify init --here in an empty directory to keep the instructions cross-platform.
# Initialize a test project using your local changes
specify init /tmp/speckit-test --ai <agent> --offline
cd /tmp/speckit-test

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings March 24, 2026 19:07
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

TESTING.md:20

  • The Windows activation hint looks incorrect/incomplete: standard venv activation scripts are .venv\Scripts\Activate.ps1 (PowerShell) or .venv\Scripts\activate.bat (cmd), whereas .venv\Scripts\activate usually won’t exist. This repo’s docs/local-development.md uses the Activate.ps1 form—please align this guide so Windows users can follow it verbatim.
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv pip install -e .

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@mnriem mnriem merged commit fb152eb into github:main Mar 24, 2026
12 checks passed
@mnriem mnriem deleted the docs/add-manual-testing-guide branch March 24, 2026 19:22
jonasbokim added a commit to Jonas-Construction-Software/jonas-spec-kit-dev that referenced this pull request Mar 26, 2026
…nc-main-2026-03-25

* upstream/main: (90 commits)
  fix(ps1): replace null-conditional operator for PowerShell 5.1 compatibility (github#1975)
  chore: bump version to 0.4.2 (github#1973)
  feat: Auto-register ai-skills for extensions whenever applicable (github#1840)
  docs: add manual testing guide for slash command validation (github#1955)
  Add AIDE, Extensify, and Presetify to community extensions (github#1961)
  docs: add community presets section to main README (github#1960)
  docs: move community extensions table to main README for discoverability (github#1959)
  docs(readme): consolidate Community Friends sections and fix ToC anchors (github#1958)
  fix(commands): rename NFR references to success criteria in analyze and clarify (github#1935)
  Add Community Friends section to README (github#1956)
  docs: add Community Friends section with Spec Kit Assistant VS Code extension (github#1944)
  chore: bump version to 0.4.1 (github#1953)
  Add checkpoint extension (github#1947)
  fix(scripts): prioritize .specify over git for repo root detection (github#1933)
  docs: add AIDE extension demo to community projects (github#1943)
  fix(templates): add missing Assumptions section to spec template (github#1939)
  chore: bump version to 0.4.0 (github#1937)
  fix(cli): add allow_unicode=True and encoding="utf-8" to YAML I/O (github#1936)
  fix(codex): native skills fallback refresh + legacy prompt suppression (github#1930)
  feat(cli): embed core pack in wheel for offline/air-gapped deployment (github#1803)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants