Skip to content

feat: [#339] AI training resources - config examples and questionnaire#345

Merged
josecelano merged 15 commits intomainfrom
339-config-examples-and-questionnaire
Feb 13, 2026
Merged

feat: [#339] AI training resources - config examples and questionnaire#345
josecelano merged 15 commits intomainfrom
339-config-examples-and-questionnaire

Conversation

@josecelano
Copy link
Member

Overview

This PR implements Issue #339, providing structured resources to help AI agents guide users through environment configuration creation.

Implementation Summary

All 6 phases completed:

Phase 1: Schema Extension ✅

  • Added optional description field to environment configuration schema
  • Updated JSON schema (schemas/environment-config.json)
  • Updated Rust DTO (src/application/command_handlers/create/config/environment_config.rs)
  • Maintained backward compatibility (configs without description still work)

Phase 2: Questionnaire Template ✅

  • Created comprehensive questionnaire at docs/ai-training/questionnaire.md (494 lines)
  • Full decision tree covering all configuration requirements
  • Includes validation rules and conditional logic

Phase 3-4: Example Configurations ✅

  • Created all 15 example scenarios covering common deployment patterns
  • All examples validated with cargo run -- validate --env-file
  • Each example includes descriptive explanation (2-3 sentences)
  • Examples use fixture keys only (no sensitive data)

Phase 5: Documentation ✅

  • Created comprehensive README at docs/ai-training/README.md (469 lines)
  • Includes scenarios table, usage instructions, customization guide
  • Documents common decision points and best practices

Phase 6: Integration Test ✅

  • Created integration test at tests/validate_ai_training_examples.rs
  • 4 test functions validating all aspects:
    • Configuration validation (all 15 examples)
    • File count verification (exactly 15 examples)
    • Naming convention enforcement (NN-descriptive-name.json pattern)
    • Description field presence validation
  • Added regex dependency for pattern matching
  • All linters passing

Example Scenarios Coverage

ID Scenario Provider Database Trackers HTTPS Monitoring Backup
01 Minimal development LXD SQLite UDP+HTTP No No No
02 Full-stack LXD LXD MySQL 2U+2H Yes Yes Yes
03 Minimal Hetzner Hetzner SQLite UDP+HTTP No No No
04 Full-stack Hetzner Hetzner MySQL 2U+2H Yes Yes Yes
... (11 more scenarios) ... ... ... ... ... ...

Benefits

For AI Agents:

  • Structured guidance through questionnaire
  • Few-shot learning from validated examples
  • Reduced hallucination with real configs as reference
  • Faster iteration with less trial-and-error

For Users:

  • Examples serve as documentation and starting points
  • Clear mapping from requirements to configuration values
  • Understanding of common deployment patterns

For Training/RAG:

  • Curated dataset with high-quality examples
  • Coverage of common and edge cases
  • Machine-readable JSON format
  • Versioned with deployer releases

Testing

  • ✅ All linters pass: cargo run --bin linter all
  • ✅ All examples validated: cargo run -- validate --env-file
  • ✅ Integration tests pass: cargo test --test validate_ai_training_examples
  • ✅ Pre-commit checks pass: ./scripts/pre-commit.sh

Files Changed

  • schemas/environment-config.json - Added description field
  • src/application/command_handlers/create/config/environment_config.rs - EnvironmentSection with description
  • docs/ai-training/questionnaire.md - Decision tree questionnaire (494 lines)
  • docs/ai-training/examples/*.json - 15 validated example configurations
  • docs/ai-training/README.md - Comprehensive documentation (469 lines)
  • tests/validate_ai_training_examples.rs - Integration test suite
  • Cargo.toml - Added regex dev-dependency
  • docs/issues/339-provide-config-examples-and-questionnaire-for-ai-agents.md - Updated progress
  • docs/roadmap.md - Marked task 11.3 as completed

Acceptance Criteria

All criteria met:

  • Optional description field added and validated
  • Backward compatibility maintained
  • Questionnaire template created
  • All 15 examples created with descriptions
  • Each example validated
  • README with scenarios table and guidance
  • Integration test created and passing
  • All linters passing
  • Examples use fixture keys only

Closes #339

@josecelano josecelano force-pushed the 339-config-examples-and-questionnaire branch from dc494af to 06165ca Compare February 13, 2026 09:41
@josecelano josecelano self-assigned this Feb 13, 2026
Core examples (01-05):
- 01-minimal-lxd: Basic LXD setup with SQLite
- 02-full-stack-lxd: Complete LXD with MySQL, HTTPS, monitoring, backups
- 03-minimal-hetzner: Basic Hetzner cloud deployment
- 04-full-stack-hetzner: Production Hetzner with all features
- 05-mysql-development: MySQL-focused development setup

Extended examples (06-15):
- 06-production-https: Production with HTTPS and monitoring
- 07-udp-only: UDP-only minimal tracker
- 08-http-only-https: HTTP API-focused with HTTPS
- 09-monitoring-stack: Full monitoring without HTTPS complexity
- 10-multi-domain: Multi-domain architecture
- 11-private-tracker: Private tracker with user whitelisting
- 12-high-availability: HA production setup with backups
- 13-backup-focused: Aggressive backup strategy
- 14-lightweight-production: Cost-effective cloud deployment
- 15-sqlite-monitoring: SQLite with full monitoring

All configurations include:
- Descriptive environment.description fields
- Realistic SSH credential paths
- Valid schema-compliant structure
- Tested with validate command
Created docs/ai-training/README.md with:
- Overview of questionnaire and example configurations
- Questionnaire workflow with Mermaid diagram
- Complete example configuration tables (core + extended)
- Scenario-to-example mapping guide
- Detailed configuration customization guide
- Common decision points (Provider, Database, HTTPS, Monitoring)
- AI agent best practices
- Troubleshooting section
- Related documentation references
Add comprehensive integration test that validates all 15 AI training
example configurations. The test suite includes:

- Configuration validation using the validate command
- File count verification (ensures exactly 15 examples)
- Naming convention enforcement (NN-descriptive-name.json pattern)
- Description field presence validation

This automated test ensures:
- All examples remain valid as the schema evolves
- CI catches configuration issues before they reach users
- Examples follow consistent naming patterns
- Every example has a descriptive explanation

Dependencies:
- Added regex = "1.0" to dev-dependencies for pattern matching
All 6 implementation phases have been successfully completed:

Phase 1: Description field added to schema ✓
Phase 2: Questionnaire created ✓
Phase 3: Core 6 examples created ✓
Phase 4: Extended 9 examples created ✓
Phase 5: README documentation created ✓
Phase 6: Integration test created ✓

All acceptance criteria met:
- All linters passing
- All 15 examples validated
- Integration test passes (4 test functions)
- Backward compatibility maintained
- Comprehensive documentation included
Task 11.3 (AI agent configuration examples and questionnaire) has been
fully implemented with all deliverables completed:

- Questionnaire template: 494 lines with decision tree
- Example configurations: 15 validated JSON scenarios
- README documentation: 469 lines with comprehensive guidance
- Integration test suite: 4 test functions validating all examples

This completes the work for improving AI agent experience with
configuration creation.
Implement Option B: Generate rendered deployment artifacts for all AI
training examples to complete the dataset with input/output pairs.

Added:
- Script: scripts/generate-ai-training-outputs.sh
  * Iterates over all 15 example configurations
  * Replaces generic SSH paths with fixture paths dynamically
  * Calls render command with placeholder IP (203.0.113.1)
  * Generates artifacts to docs/ai-training/outputs/

- Outputs: docs/ai-training/outputs/ (4.1 MB, 15 subdirectories)
  * Complete rendered templates for all examples
  * Each output includes: Ansible playbooks, Docker Compose,
    configuration files (tracker.toml, prometheus.yml, etc.)
  * Demonstrates deployer transformation: config → artifacts

- Documentation: docs/ai-training/outputs/README.md
  * Explains purpose and structure of outputs
  * Documents placeholder values used
  * Provides regeneration instructions

Updated:
- docs/ai-training/README.md: Added outputs section explaining
  complete input/output pairs for AI training

Benefits:
- AI agents can learn full input/output transformation patterns
- Users can see concrete examples of what gets deployed
- Diff analysis reveals how config options affect rendered templates
- Complete dataset for few-shot learning and pattern recognition

Script features:
- Validates fixture SSH keys exist
- Uses jq for JSON manipulation
- Detailed progress reporting
- Error handling with failure summary
- --clean flag for regeneration
- --help for usage documentation

All linters passing (markdown, yaml, toml, cspell, clippy, rustfmt, shellcheck)
- examples → environment-configs (user input configurations)
- outputs → rendered-templates (generated deployment artifacts)
- Updated all path references throughout codebase

This improves directory naming to clearly distinguish between:
- User-created configuration files (environment-configs)
- System-generated template outputs (rendered-templates)

Files updated:
- scripts/generate-ai-training-outputs.sh
- tests/validate_ai_training_examples.rs
- docs/ai-training/README.md
- docs/ai-training/rendered-templates/README.md
- docs/issues/339-provide-config-examples-and-questionnaire-for-ai-agents.md
- docs/roadmap.md
- Group inputs and outputs under docs/ai-training/dataset/
- environment-configs/ (user configuration inputs)
- rendered-templates/ (generated deployment outputs)

This structure makes it explicit that these directories form a
complete AI training dataset with input/output pairs.

Files updated:
- scripts/generate-ai-training-outputs.sh
- tests/validate_ai_training_examples.rs (added empty assertions, removed verbose output)
- docs/ai-training/README.md
- docs/ai-training/dataset/rendered-templates/README.md
- docs/issues/339-provide-config-examples-and-questionnaire-for-ai-agents.md
- docs/roadmap.md
- Add it_should_render_all_ai_training_example_configurations() test
- Validates that all 15 example configs can be rendered into deployment artifacts
- Uses tempfile for automatic cleanup of test outputs
- Converts fixture paths to absolute paths (required by render command)
- Verifies ansible and docker-compose directories are created
- Add empty assertions to fail when no examples found
- Remove verbose print statements for cleaner test output
- Fix clippy warnings (if_not_else, map_clone)

This ensures the AI training dataset outputs can actually be generated
from the input configurations, not just that configs are valid.
Update remaining path references that were missed in the previous
dataset/ reorganization commit:

- docs/ai-training/README.md: Update all table paths and examples
- docs/ai-training/dataset/rendered-templates/README.md: Update input path reference
- docs/issues/339-*.md: Update validation paths
- docs/roadmap.md: Update component paths
- scripts/generate-ai-training-outputs.sh: Update directory paths in comments

All paths now correctly reference dataset/environment-configs and
dataset/rendered-templates.
…d helper functions

- Replace direct Command usage with ProcessRunner for consistency with E2E tests
- Extract helper functions: get_file_name, setup_test_examples, path_to_str, report_failures_and_panic, has_valid_description
- Decompose render_configuration into focused functions: prepare_test_config, replace_ssh_credentials_with_fixtures, execute_render_command, verify_render_output_directories
- Reduce code duplication and improve readability
- Better error messages with contextual information
- Improve variable naming (obj -> credentials_map)
- Add test support module allow directives for shared utilities

Code reduced from 345 to 299 lines (~13% reduction) while improving maintainability.
Document the problem of stale rendered templates in AI training dataset
and explore potential solutions including timestamp mocking for deterministic
testing. Deferred for future implementation when needed.
@josecelano josecelano force-pushed the 339-config-examples-and-questionnaire branch from d4127ca to 0bd9547 Compare February 13, 2026 11:49
@josecelano
Copy link
Member Author

ACK 0bd9547

@josecelano josecelano merged commit 2807543 into main Feb 13, 2026
41 checks passed
josecelano added a commit that referenced this pull request Feb 13, 2026
The PR #345 has been merged to main. The issue tracking file is no longer needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide Configuration Examples and Questionnaire for AI Agent Guidance

1 participant