diff --git a/.claude/skills/agentic-contribution-skill/SKILL.md b/.claude/skills/agentic-contribution-skill/SKILL.md index d7936a8f..90c3656c 100644 --- a/.claude/skills/agentic-contribution-skill/SKILL.md +++ b/.claude/skills/agentic-contribution-skill/SKILL.md @@ -1,16 +1,18 @@ --- name: agentic-contribution-skill description: | - Interactive skill and agentic pack creation with automated validation and marketplace compliance. + Interactive skill creation and import with automated validation and marketplace compliance. Use when: - "Create a new skill" + - "Import an existing skill" - "Create a new agentic pack" - "Add skill to " - "Build skill for " - - User mentions "skill builder", "contribute", "new skill", or "new pack" + - User mentions "skill builder", "contribute", "new skill", "import skill", or "new pack" - Guides through discovery, definition, generation, and validation. Enforces SKILL_DESIGN_PRINCIPLES.md and agentskills.io spec. + Two modes: create from scratch or import existing SKILL.md. Guides through discovery, definition, generation, and validation. Enforces SKILL_DESIGN_PRINCIPLES.md and agentskills.io spec. +license: Apache-2.0 model: inherit color: green metadata: @@ -58,6 +60,7 @@ If prerequisites fail: **Use when**: - Creating new skill for any pack +- Importing an existing SKILL.md into the repository - Creating new agentic pack collection - Developing or editing skills for this agentic collection - User explicitly invokes `/agentic-contribution-skill` @@ -68,6 +71,13 @@ If prerequisites fail: ## Workflow +### Phase 0: Mode Selection + +Ask: **"Are you creating a new skill from scratch, or importing an existing SKILL.md?"** + +- **Create** → Proceed to Phase 1 (Discovery) +- **Import** → Proceed to Phase 1-Import (Analysis) + ### Phase 1: Discovery (5 questions max) Ask concisely, validate before proceeding. Make additional questions if needed to gather complete context. @@ -112,6 +122,62 @@ Ask concisely, validate before proceeding. Make additional questions if needed t **Quality over Speed**: Focus on gathering complete, accurate information. Validation and iteration will ensure correctness - prioritize quality of final result over generation time. +### Phase 1-Import: Analysis (import mode only) + +1. **Get file**: Ask for the path to the existing SKILL.md +2. **Read & parse**: Read the file with Read tool. Parse YAML frontmatter, extract name, description, workflow steps, MCP tools mentioned +3. **Pack suggestion**: Analyze skill content keywords and suggest the best-fit pack: + + | Keywords in skill content | Suggested pack | + |---|---| + | VM, virtual machine, KubeVirt, snapshot, migration, clone | rh-virt | + | CVE, vulnerability, remediation, compliance, RHEL, SRE | rh-sre | + | deploy, build, S2I, Helm, container, route, BuildConfig | rh-developer | + | cluster, install, Assisted Installer, multi-cluster, ROSA | ocp-admin | + | model, inference, GPU, vLLM, KServe, RHOAI, workbench | rh-ai-engineer | + | Ansible, AAP, playbook, governance, job template | rh-automation | + | CVE explanation, product lifecycle, support severity, diagnostics, patching | rh-basic | + | support case, troubleshooting, customer issue, knowledge base, must-gather | rh-support-engineer | + + - Present top match with reasoning: "This skill mentions X, Y, Z which aligns with `` (persona: )" + - Ask user to confirm or override + +4. **Color inference**: If frontmatter has no `color`, analyze the skill's workflow steps and operations to infer the risk level. Present your conclusion to the user: + ``` + Inferred color: — Reason: + Confirm? (yes/override) + ``` + Use the color mapping table from Phase 1 (Discovery). + +5. **MCP tool verification**: Identify all MCP tools referenced in the skill. Read the target pack's `mcps.json` and verify each tool exists. Flag any tools not found — they may need a new MCP server or the skill may need adaptation. + +6. **Report analysis**: + ``` + Analyzed: + Name: | Lines: | Frontmatter: + Suggested pack: (keywords: ) + Color: () + MCP tools: + Missing sections: + + Proceed with adaptation? (yes/no/try another file) + ``` + + If user says **no**: ask "Would you like to try a different file, or cancel?" and act accordingly. + +### Phase 2-Import: Adaptation (import mode only) + +**Document Consultation** (REQUIRED): Read [SKILL_DESIGN_PRINCIPLES.md](../../../SKILL_DESIGN_PRINCIPLES.md) using Read tool. Output: "I consulted SKILL_DESIGN_PRINCIPLES.md to ensure compliant adaptation." + +1. **Fix frontmatter**: Ensure `model: inherit` and `color` set (use value confirmed by user in Phase 1-Import). Add `metadata` block if missing +2. **Add missing sections**: Per DP6/DP7 — Prerequisites, When to Use, Workflow, Common Issues, Dependencies. Keep existing content, add structure around it +3. **Validate naming**: kebab-case, check uniqueness: `test -d /skills//` +4. **Place file**: Copy to `/skills//SKILL.md` +5. **Update routing**: Add entry to `/CLAUDE.md` intent routing table +6. **Show changes**: Present summary of all modifications to user, wait for confirmation + +After confirmation → proceed to Phase 5 (Validation & Iteration). + ### Phase 3: Pre-Generation Summary & Document Consultation **Document Consultation** (REQUIRED - Execute BEFORE generation): @@ -182,19 +248,7 @@ mkdir -p /skills//docs/ 5. **Update marketplace/rh-agentic-collection.yml**: If new pack (register pack for Lola installation) 6. **Create pack structure**: If new pack (README.md, CLAUDE.md, skills/ directory) -**Mandatory SKILL.md sections** (in order per DP7): -1. Frontmatter (name, description, model, color) -2. `# / Skill` + overview (1-2 sentences) -3. `## Critical: Human-in-the-Loop Requirements` (if applicable - HITL can be here or after Dependencies) -4. `## Prerequisites` (verification + Human Notification Protocol per DP8) -5. `## When to Use This Skill` (use cases + anti-patterns) -6. `## Workflow` (precise parameters per DP2, document consultation per DP1 if needed in steps) -7. `## Common Issues` (min 3, reference docs/ for details) -8. `## Dependencies` (MCP servers, tools, related skills) -9. `## Security Considerations` (if applicable) -10. `## Example Usage` (min 1, reference docs/examples.md for comprehensive cases) - -**Note**: If SKILL.md becomes too long during generation, detailed content moves to `docs/` with references in main file. +Generate SKILL.md following the mandatory section template in SKILL_DESIGN_PRINCIPLES.md (already consulted in Phase 3). If SKILL.md becomes too long, move detailed content to `docs/` with references in main file. ### Phase 5: Validation & Iteration @@ -232,84 +286,29 @@ python scripts/validate_skill_design.py /skills//SKILL.md ### Phase 6: Post-Validation Summary -```markdown -## ✅ Skill Created - -**Files**: -✅ /skills//SKILL.md ( lines) -[If docs/ created]: -✅ /skills//docs/workflow-details.md -✅ /skills//docs/common-issues.md -✅ /skills//docs/examples.md -✅ /skills//docs/external-resources.md (if applicable) -✅ /CLAUDE.md (intent routing updated) -[If new pack]: -✅ /README.md -✅ /CLAUDE.md -✅ /mcps.json (if MCP servers needed) -✅ marketplace/rh-agentic-collection.yml (pack registered) - -**Validation**: -✅ Tier 1: agentskills.io compliant (PASSED [with N warnings]) -✅ Tier 2: Design principles satisfied (PASSED [with N warnings]) -[If iterated]: -✅ Iterations: (quality over speed achieved) - -**Quality**: -- Main skill: lines (under 500-line limit) -- Sections: (all mandatory sections present) -- Workflow steps: -- Common issues: documented -- Supporting docs: files (detailed content preserved) - -**Structure**: -- Main SKILL.md: Concise and actionable -- docs/ folder: Detailed explanations, examples, external resources - -**Lola Installation**: -After merge: `lola install -f ` - -**Opinion**: - -Ready to commit? (yes/no) -``` +Present a concise summary to the user: +- List all created/modified files with line counts +- Validation results (Tier 1 + Tier 2, pass/warning counts) +- Quality metrics (line count, section count, workflow steps, common issues) +- Iteration count if applicable +- Your assessment of readiness and fit within the pack +- Ask: "Ready to commit? (yes/no)" ### Phase 7: Git Workflow (Optional - User Controls) -**User has full control** over git operations. Each step requires explicit confirmation. - -**Workflow**: +**User has full control**. Each step requires explicit confirmation: 1. **Branch**: "Create `feat/`? (yes/no)" -2. **Stage**: Show files, ask confirmation before staging -3. **Commit**: Show proposed message, wait for approval - ``` - feat: add skill to - - - - Tier 1+2 validated - - Co-Authored-By: Claude Sonnet 4.5 - ``` +2. **Stage**: Show files, ask confirmation +3. **Commit**: Propose message (`feat: add skill to `), wait for approval 4. **Push**: "Push changes? (yes/no)" -5. **PR**: Provide instructions (use `gh pr create` if available, or manual steps) +5. **PR**: Use `gh pr create` if available, or provide manual steps -**Note**: User can skip any step. Git operations are suggested, not required. User manages their repository workflow. +User can skip any step. ### Phase 8: Final Summary -```markdown -## 🎉 Complete! - -**Created**: /skills//SKILL.md -**Quality**: Production-ready -**PR**: - -**CI checks will run automatically** - -Thank you for contributing! 🚀 -``` +Report: skill path, quality status, PR URL (if created), and note that CI checks will run automatically. ## Common Issues @@ -463,6 +462,7 @@ Result: Production-ready skill in rh-virt/skills/vm-backup-create/ ``` **More examples**: [docs/examples.md](docs/examples.md) -- Example 1: VM backup skill (complete interaction) +- Example 1: VM backup skill (complete create interaction) - Example 2: Non-representative name correction -- Example 3: Large skill requiring docs/ folder with iteration +- Example 3: Importing an existing skill (analysis, adaptation, validation) +- Example 4: Large skill requiring docs/ folder with iteration diff --git a/.claude/skills/agentic-contribution-skill/docs/examples.md b/.claude/skills/agentic-contribution-skill/docs/examples.md index 3e752b42..45bec2fa 100644 --- a/.claude/skills/agentic-contribution-skill/docs/examples.md +++ b/.claude/skills/agentic-contribution-skill/docs/examples.md @@ -137,7 +137,80 @@ Choose (1-3) or provide your own: ✅ Using "cluster-health-monitor" ``` -## Example 3: Large Skill Requires docs/ Folder +## Example 3: Importing an Existing Skill + +``` +User: "/agentic-contribution-skill" + +Agentic-Contribution-Skill: +Are you creating a new skill from scratch, or importing an existing SKILL.md? +> "Import - I have a skill at ./my-skills/cluster-backup/SKILL.md" + +Reading ./my-skills/cluster-backup/SKILL.md... + +Analyzed: ./my-skills/cluster-backup/SKILL.md +Name: cluster-backup | Lines: 180 | Frontmatter: needs-fixes (missing model, color) +Suggested pack: ocp-admin (keywords: cluster, etcd, backup, restore) + +No color in frontmatter. Analyzing operations... +Inferred color: red — Reason: skill performs etcd backup and restore, which are irreversible operations +Confirm? (yes/override) +> yes + +Reading ocp-admin/mcps.json... +MCP tools referenced: resources_get, pods_exec, pods_log +✅ All 3 tools found in openshift-administration MCP server + +Analyzed: ./my-skills/cluster-backup/SKILL.md +Name: cluster-backup | Lines: 180 | Frontmatter: needs-fixes (missing model, color) +Suggested pack: ocp-admin (keywords: cluster, etcd, backup, restore) +Color: red (inferred) +MCP tools: 3 verified, 0 not found +Missing sections: Prerequisites, Dependencies + +Proceed with adaptation? (yes/no/try another file) +> yes + +Reading SKILL_DESIGN_PRINCIPLES.md for compliant adaptation... +✅ I consulted SKILL_DESIGN_PRINCIPLES.md to ensure compliant adaptation. + +Adapting skill: +✅ Added model: inherit +✅ Added color: red (confirmed by user) +✅ Added Prerequisites section with verification +✅ Added Dependencies section (MCP servers, tools) +✅ Fixed section ordering per DP7 + +Placing file: ocp-admin/skills/cluster-backup/SKILL.md +Updating: ocp-admin/CLAUDE.md (intent routing) + +Summary of changes: +- Frontmatter: added model, color, metadata +- Added 2 missing sections (Prerequisites, Dependencies) +- Reordered sections to match DP7 +- Original content preserved + +Confirm changes? (yes/no) +> yes + +Running validation... + +✅ Tier 1: PASSED +✅ Tier 2: PASSED + +## ✅ Skill Imported + +Files: +✅ ocp-admin/skills/cluster-backup/SKILL.md (210 lines) +✅ ocp-admin/CLAUDE.md (intent routing updated) + +Ready to commit? (yes/no) +> yes + +[Git workflow with confirmations] +``` + +## Example 4: Large Skill Requires docs/ Folder ``` [After Phase 2 - Definition] diff --git a/.claude/skills/federation-review/SKILL.md b/.claude/skills/federation-review/SKILL.md new file mode 100644 index 00000000..3bf7da4e --- /dev/null +++ b/.claude/skills/federation-review/SKILL.md @@ -0,0 +1,178 @@ +--- +name: federation-review +description: | + Review a Federation Request issue and validate the external pack for inclusion in the catalog. + + Use when: + - "Review federation request" + - "Validate external pack" + - User mentions "federation", "federate", or "external pack" + + NOT for direct contributions (use /agentic-contribution-skill instead). +model: inherit +color: yellow +license: Apache-2.0 +metadata: + author: Red Hat Ecosystem Engineering + version: 1.0.0 + category: internal-tooling +--- + +# Federation Review + +Evaluate a Federation Request issue and validate the external agentic pack for inclusion in the Red Hat Agentic Collections catalog. + +## Prerequisites + +- Access to the agentic-collections repository +- The federation request issue number or URL +- `uv` installed for running validation scripts +- Optional: `gitleaks` installed for credential scanning + +## When to Use This Skill + +Use this when a Federation Request issue has been opened and you need to evaluate whether the external pack meets quality standards for inclusion in the catalog. + +## Workflow + +### Phase 1: Read the Federation Request + +1. **Action:** Ask the user for the Federation Request issue number or URL +2. **Action:** Read the issue using `gh issue view ` +3. **Extract** from the issue: + - Repository URL + - Ref (SHA or tag) + - Pack name + - License + - AI agent compatibility + - Skills subset (if specified, empty means full pack) + - Owner/contact info +4. **Output to user:** Summary of the request with all extracted fields + +### Phase 2: Run Automated Validation + +1. **Action:** Run the validation script: + +```bash +# Full pack validation +uv run python scripts/validate_federation.py + +# If the pack is in a subdirectory +uv run python scripts/validate_federation.py --pack-path + +# If only specific skills were requested +uv run python scripts/validate_federation.py --skills +``` + +2. **Output to user:** The full validation report with pass/fail for each check: + - Clone and access + - Lola pack structure + - Tier 1 (agentskills.io spec) + - Tier 2 (design principles) + - MCP version pinning + - Credential scan (gitleaks) + +### Phase 3: Manual Review + +These checks require human judgment. Present each to the user for confirmation: + +1. **License compatibility:** Is the declared license compatible with Apache 2.0? + - Compatible: Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause + - Incompatible: GPL, AGPL, SSPL, proprietary + - Ask user to confirm + +2. **AI agent compatibility:** Do the declared agents match what the pack actually supports? + - Claude Code: CLAUDE.md with intent routing exists? + - Cursor: cursor-compatible config exists? + - ChatGPT: chatgpt-compatible config exists? + +3. **Pack quality:** Based on the README and CLAUDE.md: + - Is the pack description clear and accurate? + - Is the target persona well-defined? + - Are prerequisites documented? + +4. **Output to user:** Summary of manual review findings + +### Phase 4: Decision + +Present the combined results and ask the user for a decision: + +| Decision | Action | +|----------|--------| +| **Approve** | Comment on the issue confirming approval. Create a PR to add the pack to `marketplace/rh-agentic-collection.yml` under `federated_modules`. Add label `federation` to the PR. | +| **Request changes** | Comment on the issue listing specific fixes needed. Add label `federation/changes-requested` to the issue. | +| **Reject** | Comment on the issue with explanation. Add label `federation/rejected` to the issue. Close the issue. | + +### Phase 5: Registration (only if approved) + +If the user approves, create the registration PR: + +1. **Action:** Add the federated module entry to `marketplace/rh-agentic-collection.yml`: + +```yaml +federated_modules: + - name: "" + description: "" + version: "" + license: "" + repository: "" + ref: "" + pack_path: "." + tags: [] +``` + +2. **Action:** Create a branch, commit, and push +3. **Action:** Create a PR with label `federation` linking to the original issue +4. **Output to user:** PR URL + +## Dependencies + +### Required MCP Servers + +None — this skill uses CLI tools (`gh`, `git`, `uv`) and repository scripts only. + +### Required MCP Tools + +None — no MCP tools are invoked. + +### Related Skills + +- `/agentic-contribution-skill` — for direct contributions (create or import skills into this repo) + +### Reference Documentation + +**Internal:** +- [Federation Review Guide](../../../docs/FEDERATION_REVIEW_GUIDE.md) — full evaluation criteria +- [CONTRIBUTING.md](../../../CONTRIBUTING.md) — contribution paths overview +- [SKILL_DESIGN_PRINCIPLES.md](../../../SKILL_DESIGN_PRINCIPLES.md) — Tier 2 design principles + +**Scripts:** +- `scripts/validate_federation.py` — automated validation checks + +## Human-in-the-Loop + +This skill requires human confirmation at two points: +1. **Manual review** (Phase 3): License, AI agent compatibility, and quality are judgment calls +2. **Decision** (Phase 4): The approve/reject decision is always made by the maintainer + +## Example Usage + +``` +User: /federation-review +Skill: What is the Federation Request issue number? +User: 42 +Skill: Reading issue #42... + Repository: https://github.com/partner/network-pack + Ref: v2.1.0 + License: Apache-2.0 + Skills: entire pack (no subset specified) + Running validation... + [shows validation report] + Manual review needed: + - License Apache-2.0: compatible? [Y/n] + - Claude Code support declared, CLAUDE.md found: confirmed + Decision: Approve, Request changes, or Reject? +User: Approve +Skill: Creating registration PR... + PR #87 created: https://github.com/RHEcosystemAppEng/agentic-collections/pull/87 +``` diff --git a/.github/ISSUE_TEMPLATE/federation-request.yml b/.github/ISSUE_TEMPLATE/federation-request.yml new file mode 100644 index 00000000..bb462df7 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/federation-request.yml @@ -0,0 +1,146 @@ +name: Federation Request +description: Request to federate a new external pack or update an existing federation +labels: ["federation"] +body: + - type: markdown + attributes: + value: | + ## Federation Request + + Use this form to: + - **New federation**: Request that your external agentic pack be listed in the catalog + - **Update**: Request a ref update for an already-federated pack + + Your code stays in your repo — we reference it in our marketplace. + + See the [Federation Review Guide](https://github.com/RHEcosystemAppEng/agentic-collections/blob/main/docs/FEDERATION_REVIEW_GUIDE.md) for the full evaluation criteria. + + - type: dropdown + id: request_type + attributes: + label: Request type + description: Are you requesting a new federation or updating an existing one? + options: + - New federation + - Update existing federation + validations: + required: true + + - type: input + id: contact + attributes: + label: Owner / Contact + description: Name and email (or GitHub handle) of the person responsible for this pack + placeholder: "@github-handle or name " + validations: + required: true + + - type: input + id: repository_url + attributes: + label: Repository URL + description: Public GitHub repository containing the agentic pack + placeholder: https://github.com/org/repo + validations: + required: true + + - type: input + id: ref + attributes: + label: Ref (commit SHA or release tag) + description: A stable ref — not a branch name. Use a full commit SHA or a release tag. + placeholder: v1.0.0 or a1b2c3d4e5f6... + validations: + required: true + + - type: input + id: pack_name + attributes: + label: Pack name + description: Name of the agentic pack (as registered or to be registered in our marketplace) + placeholder: my-agentic-pack + validations: + required: true + + - type: textarea + id: description + attributes: + label: Pack description + description: Brief description of what the pack does, who it's for, and what value it provides. For updates, describe what changed in this version. + placeholder: | + New: This pack automates X for Y engineers working with Z... + Update: v2.1.0 adds 3 new skills for network diagnostics and fixes MCP pinning issues. + validations: + required: true + + - type: input + id: license + attributes: + label: Repository license + description: Must be compatible with Apache 2.0 (e.g., Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause) + placeholder: Apache-2.0 + validations: + required: true + + - type: input + id: marketplace_yaml_url + attributes: + label: Marketplace YAML URL (if available) + description: URL to the pack's Lola marketplace YAML file, if the repo has one + placeholder: https://raw.githubusercontent.com/org/repo/main/marketplace/my-pack.yml + validations: + required: false + + - type: checkboxes + id: ai_agents + attributes: + label: Compatible AI agents + description: Which AI agents/assistants are these skills compatible with? + options: + - label: Claude Code + - label: Cursor + - label: ChatGPT + - label: Other (specify in description above) + + - type: input + id: pack_path + attributes: + label: Pack path within repository (optional) + description: If the Lola pack is not at the repository root, specify the subdirectory path. Leave empty if the pack is at the root. + placeholder: "my-pack (or leave empty for repo root)" + validations: + required: false + + - type: textarea + id: skills_subset + attributes: + label: Skills to federate (optional) + description: Leave empty to federate the entire pack. If you only want specific skills included, list their paths (one per line). + placeholder: | + skills/my-skill-1/SKILL.md + skills/my-skill-2/SKILL.md + validations: + required: false + + - type: checkboxes + id: prerequisites + attributes: + label: Pre-submission checklist + description: For updates, confirm that the new version still meets all requirements. + options: + - label: The repository is public and accessible + required: true + - label: The pack has a valid Lola structure (CLAUDE.md, README.md, skills/, mcps.json) + required: true + - label: Skills pass Tier 1 validation (agentskills.io spec) + required: true + - label: Skills pass Tier 2 validation (design principles) + required: true + - label: No hardcoded credentials (uses `${ENV_VAR}` format) + required: true + - label: MCP server images use pinned versions or SHAs (no `:latest`) + required: true + - label: The license is compatible with Apache 2.0 + required: true + - label: The ref provided is stable (commit SHA or release tag, not a branch) + required: true diff --git a/.github/gemini-review-prompt.md b/.github/gemini-review-prompt.md index 0d81f8cd..b0b57bd9 100644 --- a/.github/gemini-review-prompt.md +++ b/.github/gemini-review-prompt.md @@ -17,6 +17,7 @@ Check the diff against the project rules injected in PROJECT RULES REFERENCE (fr - **Security**: no hardcoded credentials, no exposed secret values, `${ENV_VAR}` references only - **Skill invocation**: use `/skill-name` slash format, never call MCP tools directly - **Human-in-the-Loop**: required for create/delete/modify/restore/execute ops, not for read-only +- **Pack-persona alignment**: when reviewing new skills, verify that the skill's purpose aligns with the pack's persona (defined in the opening paragraph of the pack's CLAUDE.md). Flag any skill that appears to belong in a different pack - **New packs**: must add pack name to `PACK_DIRS` in `scripts/validate_structure.py`; `docs/data.json` must NOT be committed - **Build reminder**: if skills, agents, or `mcps.json` changed, remind author to run `make validate` diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 9a075548..e1e3401d 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,14 +1,23 @@ + + ## Summary ## Pack(s) affected -- [ ] `rh-sre` -- [ ] `rh-developer` - [ ] `ocp-admin` -- [ ] `rh-virt` - [ ] `rh-ai-engineer` +- [ ] `rh-automation` +- [ ] `rh-basic` +- [ ] `rh-developer` +- [ ] `rh-sre` +- [ ] `rh-support-engineer` +- [ ] `rh-virt` - [ ] Other / repo-wide ## Change type @@ -20,6 +29,16 @@ - [ ] MCP server config (`mcps.json`) - [ ] Docs / README - [ ] CI / tooling +- [ ] Federation (external pack) + +## Contribution method + +- [ ] Created/imported with `/agentic-contribution-skill` +- [ ] Manual contribution (validated with `make validate` + `make validate-skill-design-changed`) + +## Pack-persona alignment (new skills only) + + ## CLAUDE.md compliance diff --git a/.github/workflows/federation-validation.yml b/.github/workflows/federation-validation.yml new file mode 100644 index 00000000..6f45f091 --- /dev/null +++ b/.github/workflows/federation-validation.yml @@ -0,0 +1,184 @@ +name: Federation Validation + +on: + pull_request: + types: [labeled, opened, synchronize] + +jobs: + validate-federation: + if: contains(github.event.pull_request.labels.*.name, 'federation') + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + - name: Install uv + run: curl -LsSf https://astral.sh/uv/install.sh | sh + + - name: Install gitleaks + run: | + GITLEAKS_VERSION="8.21.2" + curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" | tar xz -C /usr/local/bin gitleaks + + - name: Install dependencies + run: | + source $HOME/.cargo/env + make install + + - name: Detect federated module changes + id: detect + run: | + source $HOME/.cargo/env + # Extract new federated_modules entries from the PR diff + uv run python -c " + import yaml, json, subprocess, sys + + # Get marketplace YAML from PR branch + with open('marketplace/rh-agentic-collection.yml') as f: + data = yaml.safe_load(f) or {} + + modules = data.get('federated_modules') or [] + if not isinstance(modules, list): + modules = [] + + # Filter to actual entries (not empty list) + modules = [m for m in modules if isinstance(m, dict) and m.get('repository')] + + if not modules: + print('No federated modules found') + sys.exit(0) + + print(f'Found {len(modules)} federated module(s)') + for m in modules: + print(f' - {m.get(\"name\", \"unknown\")}: {m.get(\"repository\", \"?\")} @ {m.get(\"ref\", \"?\")}') + + # Write module list for next step + with open('/tmp/federation-modules.json', 'w') as f: + json.dump(modules, f) + " || true + + if [ -f /tmp/federation-modules.json ]; then + echo "has_modules=true" >> "$GITHUB_OUTPUT" + else + echo "has_modules=false" >> "$GITHUB_OUTPUT" + fi + + - name: Validate federated modules + id: validate + if: steps.detect.outputs.has_modules == 'true' + run: | + source $HOME/.cargo/env + uv run python -c " + import json, subprocess, sys + + with open('/tmp/federation-modules.json') as f: + modules = json.load(f) + + all_results = [] + failed = False + for mod in modules: + repo = mod.get('repository', '') + ref = mod.get('ref', '') + pack_path = mod.get('pack_path', '.') + skills = mod.get('skills') + name = mod.get('name', 'unknown') + + print(f'\n--- Validating: {name} ---') + cmd = [sys.executable, 'scripts/validate_federation.py', repo, ref, '--pack-path', pack_path, '--json'] + if skills: + cmd.extend(['--skills'] + skills) + + result = subprocess.run(cmd, capture_output=True, text=True) + print(result.stdout) + if result.stderr: + print(result.stderr, file=sys.stderr) + + try: + report = json.loads(result.stdout) + all_results.append({'name': name, 'report': report}) + except json.JSONDecodeError: + all_results.append({'name': name, 'report': None}) + + if result.returncode != 0: + failed = True + + with open('/tmp/federation-results.json', 'w') as f: + json.dump(all_results, f) + + sys.exit(1 if failed else 0) + " + + - name: Post results as PR comment + if: always() && steps.detect.outputs.has_modules == 'true' + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const success = '${{ steps.validate.outcome }}' === 'success'; + const runUrl = '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'; + const shortSha = '${{ github.event.pull_request.head.sha }}'.substring(0, 7); + const timestamp = new Date().toISOString().replace('T', ' ').substring(0, 19) + ' UTC'; + + const checkLabels = { + clone: { name: 'Clone & access', desc: 'Repository cloned at pinned ref' }, + lola_structure: { name: 'Lola structure', desc: 'CLAUDE.md, README.md, mcps.json, skills/' }, + tier1_agentskills: { name: 'Tier 1', desc: 'agentskills.io spec compliance' }, + tier2_design_principles: { name: 'Tier 2', desc: 'Design principles compliance' }, + mcp_version_pinning: { name: 'MCP pinning', desc: 'No `:latest` tags, no hardcoded credentials' }, + gitleaks: { name: 'Credential scan', desc: 'gitleaks secret detection' }, + }; + + let moduleSections = ''; + try { + const raw = fs.readFileSync('/tmp/federation-results.json', 'utf8'); + const results = JSON.parse(raw); + for (const mod of results) { + const report = mod.report; + moduleSections += `\n### ${mod.name}\n\n`; + if (!report || !report.checks) { + moduleSections += '> Could not parse validation results for this module.\n\n'; + continue; + } + moduleSections += '| Check | Description | Status |\n'; + moduleSections += '|-------|-------------|--------|\n'; + for (const check of report.checks) { + const label = checkLabels[check.name] || { name: check.name, desc: '' }; + const statusIcon = check.skipped ? '⚠️' : (check.passed ? '✅' : '❌'); + moduleSections += `| ${label.name} | ${label.desc} | ${statusIcon} |\n`; + } + moduleSections += '\n'; + } + } catch (e) { + moduleSections = '\n> Could not parse validation results.\n\n'; + } + + const icon = success ? '✅' : '❌'; + const status = success ? 'ALL CHECKS PASSED' : 'VALIDATION FAILED'; + + const body = [ + `## ${icon} Federation Validation: ${status}`, + '', + 'Automated validation ran on federated modules declared in `marketplace/rh-agentic-collection.yml`.', + moduleSections, + `[See workflow run for full details](${runUrl})`, + '', + '---', + `Analyzed commit ${shortSha} at ${timestamp}`, + ].join('\n'); + + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: body + }); + + - name: Skip (no federated modules) + if: steps.detect.outputs.has_modules == 'false' + run: echo "No federated modules found in marketplace YAML, nothing to validate." diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d37081ac..f9b7b0db 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -32,3 +32,9 @@ repos: language: system pass_filenames: false files: ^(rh-[-a-z0-9]+|ocp-admin)/skills/[^/]+/SKILL\.md$ + + - id: skill-spec-linter + name: skill spec linter (Tier-1 agentskills.io) + entry: bash -c 'for f in "$@"; do ./scripts/run-skill-linter.sh "$(dirname "$f")/"; done' -- + language: system + files: ^(rh-[-a-z0-9]+|ocp-admin)/skills/[^/]+/SKILL\.md$ diff --git a/CLAUDE.md b/CLAUDE.md index 0b9b41be..4163b195 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,6 +39,14 @@ Each pack follows this structure: Optional (not required for Lola): `.claude-plugin/plugin.json` — Claude Code plugin metadata if you publish this pack through the Claude Code plugin format in addition to Lola. +## Contribution Paths + +There are two ways to add skills to this project: + +**Direct Contribution** — Skills are added directly to this repository, inside an existing pack. The contributor opens a PR, skills are reviewed and merged, and maintainers own them from that point. Use `/agentic-contribution-skill` in Claude Code or follow [CONTRIBUTING.md](CONTRIBUTING.md). + +**Federation** — An external repository containing a complete, independent Lola pack is referenced in our catalog. The code stays in the external repo; users install it directly via Lola. The external owner maintains their pack. To request federation, open a [Federation Request](https://github.com/RHEcosystemAppEng/agentic-collections/issues/new?template=federation-request.yml) issue. Maintainers evaluate the pack using the [Federation Review Guide](docs/FEDERATION_REVIEW_GUIDE.md). + ## Working with Agentic Collections ### Skills diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0378401d..638eb788 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,453 +1,83 @@ # Contributing to Agentic Collections -Thank you for your interest in contributing to the Red Hat Agentic Collections marketplace! This guide will help you create high-quality skills that meet our standards and integrate seamlessly into the ecosystem. +Add skills to the Red Hat Agentic Collections marketplace. -## Quick Start +## How to Contribute -The **fastest and recommended way** to contribute is using the `/agentic-contribution-skill` tool: +The fastest way to contribute is using `/agentic-contribution-skill` in Claude Code. It guides you through everything: discovery, definition, generation, validation, and git workflow. -```bash -# Run the interactive skill builder -/agentic-contribution-skill - -# Follow the interactive prompts -# The tool will guide you through everything: discovery, definition, generation, validation, and git workflow -``` - -The agentic-contribution-skill ensures your contribution meets all quality standards automatically. - -## What is agentic-contribution-skill? - -`agentic-contribution-skill` is an interactive AI assistant available in this repository's `.claude/skills/` directory that: -- ✅ Guides you through skill creation with targeted questions -- ✅ Generates complete skill structure following best practices -- ✅ Validates against agentskills.io specification (Tier 1) -- ✅ Validates against repository design principles (Tier 2) -- ✅ Automates git workflow (branch, commit, PR) -- ✅ Ensures your skill passes CI checks before submission - -**Why use it?** It eliminates guesswork, prevents common mistakes, and saves hours of manual work. - -**How it works**: The skill uses a human-in-the-loop approach - you maintain full control. It will ask for confirmation before: -- Generating files -- Creating git branches -- Making commits -- Pushing to remote - -## Prerequisites - -### Required Tools - -1. **Git** - Version control - ```bash - git --version # Verify installation - ``` - -2. **uv** - Python environment manager (for validation scripts) - ```bash - # Install uv - curl -LsSf https://astral.sh/uv/install.sh | sh - - # Or on macOS - brew install uv - ``` - -3. **Claude Code** - Latest version - - Desktop app: [claude.ai/code](https://claude.ai/code) - - VS Code/JetBrains extension - - CLI tool - -### Repository Setup - -1. **Fork the repository** to your GitHub account - - Visit: https://github.com/RHEcosystemAppEng/agentic-collections - - Click "Fork" button (top right) - -2. **Clone your fork** locally - ```bash - git clone https://github.com/YOUR_USERNAME/agentic-collections - cd agentic-collections - ``` - -3. **Add upstream remote** (optional but recommended) - ```bash - git remote add upstream https://github.com/RHEcosystemAppEng/agentic-collections - ``` - -4. **Open the repository in Claude Code** - - The `/agentic-contribution-skill` will be automatically available - - Claude Code loads skills from `.claude/skills/` directory on startup - - Verify it's available by typing `/agentic-contribution-skill` in the prompt - -### Knowledge Requirements - -- **Basic understanding** of AI skills and agents -- **Familiarity** with YAML frontmatter syntax -- **Knowledge** of the technology/platform your skill targets -- **Understanding** of Git workflow (fork, branch, commit, PR) - -## Contribution Workflow - -### Using agentic-contribution-skill (Recommended) - -The agentic-contribution-skill guides you through 5 phases: - -#### Phase 1: Discovery -Answer questions about your skill: -- What does it do? -- Who uses it? -- Which pack should it belong to? -- What MCP tools does it need? - -#### Phase 2: Definition -Provide detailed specifications: -- Skill name (unique, kebab-case) -- Use case examples (concrete user phrases) -- Workflow steps with MCP tools -- Common issues users might face - -#### Phase 3: Generation -agentic-contribution-skill creates: -- Complete `SKILL.md` with all mandatory sections -- Updated `CLAUDE.md` intent routing -- MCP configuration (if needed) -- New pack structure (if creating new pack) - -#### Phase 4: Validation -Automated quality checks: -- **Tier 1**: agentskills.io specification compliance -- **Tier 2**: Repository design principles (DP1-11) -- Detailed feedback on any issues - -#### Phase 5: Git Workflow -With your confirmation at each step: -- Create branch (`feat/skill-name`) -- Stage files -- Create commit with co-authorship -- Push to your fork -- Guide PR creation - -### Manual Contribution (Advanced) - -If you prefer manual creation: - -1. **Read the standards** - - [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) - Complete requirements - - [agentskills.io specification](https://agentskills.io/specification) - Base standard - -2. **Create skill structure** - ```bash - mkdir -p /skills// - # Create SKILL.md following SKILL_DESIGN_PRINCIPLES.md template - ``` - -3. **Update pack CLAUDE.md** - - Add entry to intent routing table - -4. **Validate locally** - ```bash - # Tier 1 validation (agentskills.io) - ./scripts/run-skill-linter.sh /skills// - - # Tier 2 validation (design principles) - make validate-skill-design-changed - - # Skill docs link validation (must use skill-local docs/... paths) - uv run python scripts/validate_skill_doc_links.py /skills//SKILL.md - - # Skill docs tree validation (skills docs markdown links) - uv run python scripts/validate_docs_tree_links.py /skills//SKILL.md - - # Both must pass before committing - ``` - -5. **Create pull request** - ```bash - git checkout -b feat/ - git add /skills// /CLAUDE.md - git commit -m "feat: add skill to " - git push origin feat/ - # Create PR on GitHub from your fork to upstream - ``` - -## Quality Standards - -All skills must comply with **two tiers** of standards: - -### Tier 1: agentskills.io Specification (Automated) - -Base standard for skill structure and format: -- Valid YAML frontmatter with required fields -- Proper naming conventions (kebab-case) -- Section structure and ordering -- Parameter documentation - -**Validation**: `./scripts/run-skill-linter.sh` - -### Tier 2: Repository Design Principles (Automated) - -Enhanced requirements specific to this repository: - -| Principle | Requirement | Validated By | -|-----------|-------------|--------------| -| **DP1** | Document Consultation Transparency | Script checks for Read tool + declaration | -| **DP2** | Precise Parameter Specification | Validates parameter examples exist | -| **DP3** | Description Conciseness | Counts tokens, must be <500 | -| **DP4** | Dependencies Declaration | Checks Dependencies section presence | -| **DP5** | Human-in-the-Loop Requirements | Validates confirmation steps for critical ops | -| **DP6** | Mandatory Sections | Verifies all required sections present in order | -| **DP7** | MCP Server Verification | Checks for credential exposure | -| **DP11** | Pack-Level CLAUDE.md | Validates intent routing updated | - -**Validation**: `make validate-skill-design-changed` - -See [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) for complete details and rationale. - -## What Makes a Good Skill? - -### Clear Purpose -- **Do one thing well** - Single responsibility -- **Specific use case** - Not too broad -- **Real user need** - Solves actual problem - -### Quality Documentation -- **Concrete examples** - Real user phrases, not generic descriptions -- **Precise parameters** - Exact formats with examples -- **Common issues** - At least 3 documented with solutions -- **Complete workflow** - All steps with error handling - -### Production Ready -- **Error handling** - Covers failure scenarios -- **Security** - No credential exposure -- **Human-in-the-loop** - Confirmations for critical operations -- **Validated** - Passes Tier 1 + Tier 2 checks - -## Pack Selection Guide +### Two paths -Choose the right pack for your skill: +**Have an idea for a new skill?** -| Pack | Persona | When to Use | -|------|---------|-------------| -| **rh-sre** | Site Reliability Engineers | CVE remediation, system compliance, RHEL automation | -| **rh-developer** | Application Developers | App deployment, S2I builds, Helm charts | -| **openshift-virtualization** | Virtualization Admins | VM lifecycle, snapshots, migrations | -| **ocp-admin** | OpenShift Administrators | Cluster management, health reports, monitoring | -| **rh-ai-engineer** | AI/ML Engineers | Model serving, vLLM, KServe, NVIDIA NIM | -| **rh-automation** | Automation Leads | Ansible AAP governance, safety checks | -| **rh-support-engineer** | Support Engineers | Technical support, troubleshooting | - -**Creating a new pack?** Use agentic-contribution-skill - it will create the complete pack structure for you. - -## Testing Your Skill Locally - -Before submitting a PR, test your skill locally: - -1. **Invoke the skill** in Claude Code: - ``` - /your-skill-name - ``` - -2. **Try different scenarios**: - - Normal use case (happy path) - - Edge cases - - Error conditions - -3. **Verify the output**: - - Check that MCP tools are called correctly - - Ensure error messages are helpful - - Confirm human-in-the-loop prompts appear for critical operations - -4. **Test with real data** (if applicable): - - Use actual VMs, clusters, or systems (in a safe environment) - - Don't test destructive operations on production systems - -## Pull Request Process - -### Before Submitting - -- [ ] Skill created with agentic-contribution-skill or manually validated -- [ ] Tier 1 validation passed: `./scripts/run-skill-linter.sh` -- [ ] Tier 2 validation passed: `make validate-skill-design-changed` -- [ ] Skill docs links validation passed: `uv run python scripts/validate_skill_doc_links.py /skills//SKILL.md` -- [ ] Skill docs tree links validation passed: `uv run python scripts/validate_docs_tree_links.py /skills//SKILL.md` -- [ ] Tested skill locally by invoking it in Claude Code -- [ ] Reviewed generated skill for accuracy -- [ ] No credentials exposed in skill documentation - -### PR Template - -When creating your PR, include: - -```markdown -## Description -[What does the skill do and why is it valuable?] - -## Skill Details -- **Pack**: -- **Operation Type**: -- **MCP Servers**: -- **Validation**: Tier 1 + Tier 2 passed - -## Use Cases -- [List 3-5 concrete examples] - -## Testing -- [x] Generated with agentic-contribution-skill / manually validated -- [x] Tier 1 validation passed -- [x] Tier 2 validation passed -- [ ] Manually tested skill invocation - -## Checklist -- [x] Follows SKILL_DESIGN_PRINCIPLES.md -- [x] CLAUDE.md intent routing updated -- [x] No credentials exposed -- [x] Human-in-the-loop for critical operations +``` +/agentic-contribution-skill +# Choose "Create" → answer discovery questions → skill is generated and validated ``` -### CI Checks - -Your PR will automatically run: -1. **Compliance check** - Validates marketplace structure -2. **Skill spec linter** - Tier 1 validation (agentskills.io) -3. **Design validator** - Tier 2 validation (design principles) - -All checks must pass before merge. - -### Review Process - -1. **Automated validation** - CI checks run -2. **Maintainer review** - Quality and fit assessment -3. **Feedback** - Requested changes (if needed) -4. **Iteration** - Address feedback, re-validate -5. **Approval** - Maintainer approves -6. **Merge** - Skill added to marketplace - -## Common Issues and Solutions - -### Validation Failures - -**Issue**: "Description exceeds 500 tokens" -- **Fix**: Shorten frontmatter description, move details to body sections -- **Tool**: `head -n 20 SKILL.md | wc -w` (rough estimate: ~1.3 words per token) - -**Issue**: "Skill name already exists" -- **Fix**: Choose more specific name, check: `ls /skills/` - -**Issue**: "MCP server not configured" -- **Fix**: Add to `/mcps.json` using `${ENV_VAR}` format for credentials - -### Git Issues - -**Issue**: "Push authentication failed" -- **Fix**: Configure git credentials or SSH key -- **HTTPS**: `git config --global credential.helper store` -- **SSH**: Add SSH key to GitHub settings - -**Issue**: "Merge conflicts" -- **Fix**: Sync with upstream, resolve conflicts - ```bash - git fetch upstream - git merge upstream/main - # Resolve conflicts - git commit - ``` - -### Skill Quality - -**Issue**: "Workflow steps too vague" -- **Fix**: Add precise parameter names and examples -- **Example**: Instead of "Create VM", use "Create VM with `name: `, `namespace: `" - -**Issue**: "Missing common issues" -- **Fix**: Document at least 3 real issues users might face with causes and solutions - -## Getting Help - -### Resources - -- **Design Principles**: [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) -- **agentskills.io Spec**: https://agentskills.io/specification -- **Documentation Site**: https://rhecosystemappeng.github.io/agentic-collections - -### Support Channels - -- **Issues**: [GitHub Issues](https://github.com/RHEcosystemAppEng/agentic-collections/issues) -- **Discussions**: [GitHub Discussions](https://github.com/RHEcosystemAppEng/agentic-collections/discussions) -- **PR Comments**: Ask questions directly in your pull request - -### Reporting Bugs - -Found a bug in agentic-contribution-skill or existing skills? +**Already have a SKILL.md?** -1. Check [existing issues](https://github.com/RHEcosystemAppEng/agentic-collections/issues) -2. Create new issue with: - - Clear description of the problem - - Steps to reproduce - - Expected vs actual behavior - - Environment (Claude Code version, OS) +``` +/agentic-contribution-skill +# Choose "Import" → point to your file → skill is analyzed, adapted, and validated +``` -## Advanced Topics +### Quick start -### Creating a New Pack +1. Fork and clone the repository +2. Open the project in Claude Code +3. Run `/agentic-contribution-skill` and follow the prompts -Use agentic-contribution-skill - it automates the process: -1. Detects when a new pack is needed -2. Creates complete pack structure: - - `README.md` - - `CLAUDE.md` with intent routing - - `skills/` directory - - `mcps.json` (if MCP servers needed) - - Optional: `.claude-plugin/plugin.json` (if publishing via Claude Code plugin format) -3. Updates `marketplace/rh-agentic-collection.yml` +The skill handles pack selection, CLAUDE.md routing, validation (Tier 1 + Tier 2), and git workflow automatically. -Manual creation is possible but not recommended - see [CLAUDE.md](CLAUDE.md) for structure. +## Pack Selection -### Adding MCP Servers +Each pack targets a specific persona. Choose the one that matches your skill: -agentic-contribution-skill guides you through: -1. MCP server selection based on use case -2. `mcps.json` configuration with environment variables -3. Security validation (no hardcoded credentials) +| Pack | Persona | Use when | +|------|---------|----------| +| `rh-sre` | Site Reliability Engineers | CVE remediation, system compliance, RHEL automation | +| `rh-developer` | Application Developers | App deployment, S2I builds, Helm charts | +| `rh-virt` | Virtualization Admins | VM lifecycle, snapshots, migrations | +| `ocp-admin` | OpenShift Administrators | Cluster management, health reports, monitoring | +| `rh-ai-engineer` | AI/ML Engineers | Model serving, vLLM, KServe, NVIDIA NIM | +| `rh-automation` | Automation Leads | Ansible AAP governance, safety checks | +| `rh-basic` | General Red Hat Users | CVE diagnostics, lifecycle, patching | +| `rh-support-engineer` | Support Engineers | Technical support, troubleshooting | -Manual addition: Add to `/mcps.json` following existing patterns. +Not sure which pack? The skill will suggest one based on your skill's content. -### Skill Templates +## Manual Contribution -Reference templates: -- **General**: [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) - Standard template -- **rh-virt Enhanced**: [rh-virt/SKILL_TEMPLATE.md](rh-virt/SKILL_TEMPLATE.md) - With additional quality checks +If you prefer to create skills manually: -agentic-contribution-skill uses these templates automatically. +1. Read the standards: [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) +2. Create `/skills//SKILL.md` following the template +3. Update `/CLAUDE.md` intent routing table +4. Validate Tier 1: `./scripts/run-skill-linter.sh /skills//` +5. Validate Tier 2: `make validate-skill-design-changed` -## Code of Conduct +Both tiers must pass before submitting a PR. -We are committed to providing a welcoming and inclusive experience for all contributors. +## Before You Submit -**Expected Behavior**: -- Be respectful and professional -- Accept constructive feedback gracefully -- Focus on what's best for the community -- Show empathy towards others +- [ ] Tier 1 validation passed (agentskills.io spec) +- [ ] Tier 2 validation passed (design principles) +- [ ] Skill doc links validated: `uv run python scripts/validate_skill_doc_links.py /skills//SKILL.md` +- [ ] Skill doc tree links validated: `uv run python scripts/validate_docs_tree_links.py /skills//SKILL.md` +- [ ] Pack CLAUDE.md intent routing updated +- [ ] Tested skill locally by invoking it in Claude Code +- [ ] No credentials hardcoded (use `${ENV_VAR}` format) +- [ ] Human-in-the-loop confirmation for destructive operations -**Unacceptable Behavior**: -- Harassment or discriminatory language -- Personal attacks or trolling -- Spam or off-topic content +## Resources -Report issues to: eco-engineering@redhat.com +- [SKILL_DESIGN_PRINCIPLES.md](SKILL_DESIGN_PRINCIPLES.md) -- Design principles and templates +- [Federation Review Guide](docs/FEDERATION_REVIEW_GUIDE.md) -- How external packs are evaluated for federation +- [agentskills.io specification](https://agentskills.io/specification) -- Base skill standard +- [Documentation site](https://rhecosystemappeng.github.io/agentic-collections) -- Browse all collections +- [GitHub Issues](https://github.com/RHEcosystemAppEng/agentic-collections/issues) -- Report bugs or ask questions ## License -By contributing to this repository, you agree that your contributions will be licensed under the Apache License 2.0. - -See [LICENSE](LICENSE) for details. - ---- - -## Thank You! - -Your contributions make this marketplace valuable for the entire Red Hat ecosystem. Whether you're fixing a typo or adding a complex skill, every contribution matters. - -**Ready to contribute?** Start with `/agentic-contribution-skill` and let the tool guide you! - -Questions? Open an [issue](https://github.com/RHEcosystemAppEng/agentic-collections/issues) or discussion. +By contributing, you agree your contributions are licensed under [Apache License 2.0](LICENSE). diff --git a/Makefile b/Makefile index 68c91ac4..afb06769 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install validate validate-collection-schema validate-collection-compliance catalog-mirror-json validate-skill-design validate-skill-design-changed validate-mcp-tools generate serve clean test test-full check-uv +.PHONY: help install validate validate-collection-schema validate-collection-compliance catalog-mirror-json validate-skill-design validate-skill-design-changed validate-mcp-tools validate-federated generate serve clean test test-full check-uv help: @echo "agentic-collections Documentation Generator" @@ -11,6 +11,7 @@ help: @echo " catalog-mirror-json - Regenerate all .catalog/collection.json from YAML" @echo " validate-skill-design - Validate all skills (use PACK=rh-sre for a specific pack)" @echo " validate-skill-design-changed - Validate only changed skills (staged + unstaged, for local dev)" + @echo " validate-federated - Validate federated modules from marketplace YAML" @echo " validate-mcp-tools - Validate allowed-tools against live MCP servers (requires podman)" @echo " generate - Generate docs/data.json" @echo " serve - Start local server on http://localhost:8000" @@ -71,6 +72,10 @@ validate-mcp-tools: check-uv @uv run python scripts/validate_mcp_tools.py $(if $(PACK),$(PACK)) @echo "✓ MCP tool validation complete!" +validate-federated: check-uv + @echo "Validating federated modules..." + @uv run python scripts/fetch_federated_skills.py + generate: check-uv @echo "Generating documentation..." @uv run python scripts/build_website.py diff --git a/README.md b/README.md index 196c7fa8..ceae1ea9 100644 --- a/README.md +++ b/README.md @@ -140,13 +140,11 @@ Each plugin has additional requirements: --- -## 🤝 Contributing +## Contributing -**Want to add your own skill to the marketplace?** We've made it easy with our interactive skill builder! - -Use the `/agentic-contribution-skill` skill to create production-ready skills with automated validation, or contribute manually following our design principles. The skill builder guides you through discovery, definition, generation, validation, and git workflow - no prior experience needed! - -📖 **See the complete guide**: [CONTRIBUTING.md](CONTRIBUTING.md) +**New skill idea?** Run `/agentic-contribution-skill` in Claude Code. +**Have an existing skill?** Run `/agentic-contribution-skill` and choose import mode. +**Full guide:** [CONTRIBUTING.md](CONTRIBUTING.md) --- @@ -516,6 +514,7 @@ That CLI checks marketplace/plugin manifests for that workflow, including `plugi - **[Documentation Site](https://rhecosystemappeng.github.io/agentic-collections)**: Browse all collections, skills, and MCP servers - **[CLAUDE.md](CLAUDE.md)**: Repository structure and development workflow - **[Skill Design Principles](SKILL_DESIGN_PRINCIPLES.md)**: Quality guidelines for skills +- **[Federation Review Guide](docs/FEDERATION_REVIEW_GUIDE.md)**: How we evaluate external packs for federation - **[VALIDATION_REPORT.md](VALIDATION_REPORT.md)**: Marketplace compliance verification - **[Security Policy](SECURITY.md)**: Credential handling and vulnerability reporting diff --git a/docs/FEDERATION_REVIEW_GUIDE.md b/docs/FEDERATION_REVIEW_GUIDE.md new file mode 100644 index 00000000..4289c6a2 --- /dev/null +++ b/docs/FEDERATION_REVIEW_GUIDE.md @@ -0,0 +1,170 @@ +# Federation Review Guide + +Step-by-step guide for maintainers evaluating a [Federation Request](https://github.com/RHEcosystemAppEng/agentic-collections/issues/new?template=federation-request.yml). + +Federation means referencing an **external agentic pack** in our catalog. The code stays in the external repo — we don't copy or modify it. Users install federated packs directly from the external repo via Lola. + +## Two ways to review + +| Method | When to use | +|--------|------------| +| **`/federation-review` skill** | Interactive review from Claude Code. Reads the issue, runs validation, guides you through manual checks, and creates the registration PR if approved. | +| **Manual (this guide)** | Step-by-step commands when you prefer to review without Claude Code. | + +CI also runs automated validation on any PR with the `federation` label (see [CI automation](#ci-automation) below). + +--- + +## Automated validation (recommended) + +Steps 1–6 can be run with a single command from the agentic-collections repo root: + +```bash +# Full pack at repo root +uv run python scripts/validate_federation.py + +# Pack in a subdirectory +uv run python scripts/validate_federation.py --pack-path + +# Only specific skills +uv run python scripts/validate_federation.py --skills + +# JSON output (for CI) +uv run python scripts/validate_federation.py --json +``` + +The script checks: clone access, Lola structure, Tier 1, Tier 2, MCP version pinning, and gitleaks. If all pass, the pack is ready for manual review (steps 7–8). + +--- + +## Manual steps + +### Step 1: Verify access and basic info + +- [ ] Repository URL is reachable and public +- [ ] The ref (SHA or tag) exists: `git ls-remote ` +- [ ] Owner/contact information is provided +- [ ] License file exists in the repo and is compatible with Apache 2.0 (e.g., Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause) + +```bash +git clone --no-checkout /tmp/federation-review +cd /tmp/federation-review +git checkout +``` + +### Step 2: Verify Lola pack structure + +The pack must have the minimum structure for Lola installation: + +``` +/ +├── CLAUDE.md # Required: persona, intent routing, rules +├── README.md # Required: description, prerequisites, installation +├── mcps.json # Required: MCP server configs (can be empty: {"mcpServers": {}}) +└── skills/ # Required: at least one skill + └── / + └── SKILL.md # Required: YAML frontmatter + implementation +``` + +```bash +ls CLAUDE.md README.md mcps.json skills/*/SKILL.md +``` + +If the request is for a **subset of skills** (not the full pack), verify only the listed skill paths exist. + +### Step 3: Validate skills — Tier 1 (agentskills.io spec) + +Tier 1 is **mandatory**. Run the linter on each skill: + +```bash +# From the agentic-collections repo root: +for skill_dir in /tmp/federation-review/skills/*/; do + ./scripts/run-skill-linter.sh "$skill_dir" +done +``` + +**Result:** All skills must pass (exit code 0). Warnings are acceptable; errors block federation. + +### Step 4: Validate skills — Tier 2 (design principles) + +Tier 2 is **mandatory**. Run the design principles validator: + +```bash +uv run python scripts/validate_skill_design.py /tmp/federation-review +``` + +See [SKILL_DESIGN_PRINCIPLES.md](../SKILL_DESIGN_PRINCIPLES.md) for the full list of design principles. + +### Step 5: Validate MCP configuration + +If the pack uses MCP servers (`mcps.json` is not empty): + +- [ ] **No `:latest` tags** — All container images must use pinned versions or SHAs +- [ ] **No hardcoded credentials** — All secrets use `${ENV_VAR}` format + +```bash +grep -r ":latest" mcps.json && echo "FAIL: found :latest" || echo "PASS: no :latest" +``` + +### Step 6: Security review + +- [ ] No credentials or secrets in any file +- [ ] Destructive operations have human-in-the-loop confirmation + +```bash +gitleaks detect --source /tmp/federation-review --verbose +``` + +LLM-based security scan is triggered **on-demand** by a maintainer due to cost. + +### Step 7: AI agent compatibility + +Verify the declared AI agent compatibility from the issue: + +- [ ] If **Claude Code** is declared: CLAUDE.md exists with proper intent routing +- [ ] If **Cursor** is declared: check for Cursor-compatible configuration +- [ ] If **ChatGPT** is declared: check for ChatGPT-compatible configuration + +### Step 8: Decision + +| Result | Action | +|--------|--------| +| All checks pass | Approve — create registration PR with label `federation` | +| Minor issues | Comment on issue with specific fixes, label `federation/changes-requested` | +| Major issues | Reject with explanation, label `federation/rejected`, close issue | + +When approving, add the pack to `marketplace/rh-agentic-collection.yml` under `federated_modules` (include the `license` field from the issue) and link back to the issue. + +--- + +## CI automation + +PRs with the `federation` label automatically trigger the **Federation Validation** workflow (`.github/workflows/federation-validation.yml`). It: + +1. Reads `federated_modules` from `marketplace/rh-agentic-collection.yml` +2. Runs `scripts/validate_federation.py` on each entry +3. Posts a summary comment on the PR with pass/fail results + +The label-based trigger ensures validation only runs on federation PRs, not on every marketplace YAML change. + +--- + +## Cleanup + +```bash +rm -rf /tmp/federation-review +``` + +## Quick reference + +| Check | Required | Automated | Tool/Command | +|-------|----------|-----------|--------------| +| Public access | Yes | Yes | `git ls-remote` | +| License compatibility | Yes | No | Manual review | +| Lola pack structure | Yes | Yes | `scripts/validate_federation.py` | +| Tier 1 (agentskills.io) | Yes | Yes | `scripts/validate_federation.py` | +| Tier 2 (design principles) | Yes | Yes | `scripts/validate_federation.py` | +| MCP version pinning | Yes | Yes | `scripts/validate_federation.py` | +| Credential scan | Yes | Yes | `gitleaks` via script | +| AI agent compatibility | Yes | No | Manual review | +| Security scan (LLM) | On-demand | No | Maintainer-triggered | diff --git a/docs/app.js b/docs/app.js index 16f8ac90..98350ed8 100644 --- a/docs/app.js +++ b/docs/app.js @@ -306,6 +306,12 @@ function createPackCard(pack) { } badges.appendChild(badge); } + if (pack.source === 'federated') { + const fedBadge = document.createElement('span'); + fedBadge.className = 'pack-eval-badge is-federated'; + fedBadge.textContent = 'FEDERATED'; + badges.appendChild(fedBadge); + } if (badges.childNodes.length > 0) { headerRow.appendChild(badges); } @@ -313,7 +319,11 @@ function createPackCard(pack) { const meta = document.createElement('p'); meta.className = 'pack-meta'; - meta.textContent = `By Red Hat · v${pack.plugin.version || '0.0.0'}`; + if (pack.source === 'federated') { + meta.textContent = `External · v${pack.plugin.version || '0.0.0'} · ${pack.ref || ''}`; + } else { + meta.textContent = `By Red Hat · v${pack.plugin.version || '0.0.0'}`; + } div.appendChild(meta); // Description (prefer collection catalog metadata over plugin fallback text) diff --git a/docs/styles.css b/docs/styles.css index 26c71934..ed7be592 100644 --- a/docs/styles.css +++ b/docs/styles.css @@ -589,6 +589,12 @@ section.collapsed .grid { border-color: #d1d5db; } +.pack-eval-badge.is-federated { + color: #1e40af; + background: #dbeafe; + border-color: #60a5fa; +} + .pack-eval-divider { width: 100%; height: 1px; diff --git a/marketplace/rh-agentic-collection.yml b/marketplace/rh-agentic-collection.yml index 1c2569ee..895b6b70 100644 --- a/marketplace/rh-agentic-collection.yml +++ b/marketplace/rh-agentic-collection.yml @@ -105,3 +105,24 @@ modules: - "troubleshooting" - "execution" +# Federated modules — complete Lola packs hosted in external repositories. +# Each entry points to a public repo at a pinned ref. The build clones, validates, +# and includes them in the catalog as standalone federated packs. +# To request federation, open a GitHub issue using the "Federation Request" template. +federated_modules: [] + # Example: + # - name: "partner-network-tools" + # description: "Network troubleshooting skills for SDN and OVN diagnostics" + # version: "1.2.0" + # license: "Apache-2.0" # must be compatible with Apache 2.0 + # repository: "https://github.com/partner-org/network-skills" + # ref: "a1b2c3d4e5f6789012345678901234567890abcd" + # pack_path: "." # path within the repo where the Lola pack lives (default: repo root) + # skills: # optional: federate only these skills (omit to include all) + # - "skills/sdn-diagnostics/SKILL.md" + # - "skills/ovn-trace/SKILL.md" + # - "skills/network-policy-audit/SKILL.md" + # tags: + # - "networking" + # - "sdn" + # - "troubleshooting" diff --git a/scripts/fetch_federated_skills.py b/scripts/fetch_federated_skills.py new file mode 100755 index 00000000..b581dbe9 --- /dev/null +++ b/scripts/fetch_federated_skills.py @@ -0,0 +1,244 @@ +#!/usr/bin/env python3 +""" +Fetch and validate skills from federated external repositories. + +Reads federated_modules from marketplace/rh-agentic-collection.yml, clones each +external repo at its pinned ref, runs Tier 1 validation on declared skills, and +reports results as structured JSON. + +Usage: + python scripts/fetch_federated_skills.py # validate all + python scripts/fetch_federated_skills.py --json # JSON output + python scripts/fetch_federated_skills.py --fetch-only # clone without validating + python scripts/fetch_federated_skills.py --output-dir /tmp/fed # custom clone target +""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +import subprocess +import sys +import tempfile +from dataclasses import dataclass, field, asdict +from pathlib import Path +from typing import List, Optional + +import pack_registry + + +@dataclass +class SkillResult: + path: str + passed: bool + warnings: bool = False + output: str = "" + + +@dataclass +class ModuleResult: + name: str + repository: str + ref: str + pack_path: str + clone_ok: bool = False + skills: List[SkillResult] = field(default_factory=list) + error: str = "" + clone_path: str = "" + + +def clone_at_ref(repository: str, ref: str, dest: Path) -> Optional[str]: + """Clone a repository and checkout the pinned ref. Returns error string or None.""" + try: + subprocess.run( + ["git", "clone", "--quiet", "--no-checkout", repository, str(dest)], + check=True, + capture_output=True, + text=True, + timeout=120, + ) + subprocess.run( + ["git", "checkout", "--quiet", ref], + check=True, + capture_output=True, + text=True, + cwd=dest, + timeout=30, + ) + return None + except subprocess.CalledProcessError as exc: + return exc.stderr.strip() or str(exc) + except subprocess.TimeoutExpired: + return "git operation timed out" + + +def validate_skill(skill_dir: Path, repo_root: Path) -> SkillResult: + """Run the Tier 1 linter on a single skill directory.""" + skill_md = skill_dir / "SKILL.md" + rel_path = str(skill_dir.relative_to(repo_root)) + + if not skill_md.exists(): + return SkillResult(path=rel_path, passed=False, output=f"SKILL.md not found at {rel_path}") + + linter = Path(__file__).resolve().parent.parent / ".claude" / "skills" / "skill-linter" / "scripts" / "validate-skill.sh" + if not linter.exists(): + return SkillResult(path=rel_path, passed=False, output="Linter script not found") + + try: + result = subprocess.run( + [str(linter), str(skill_dir)], + capture_output=True, + text=True, + timeout=30, + ) + has_warnings = "[WARN]" in result.stdout + return SkillResult( + path=rel_path, + passed=result.returncode == 0, + warnings=has_warnings, + output=result.stdout.strip(), + ) + except subprocess.TimeoutExpired: + return SkillResult(path=rel_path, passed=False, output="Linter timed out") + + +def process_module( + module: dict, + base_dir: Path, + validate: bool = True, +) -> ModuleResult: + """Clone and optionally validate a single federated module.""" + name = module.get("name", "unknown") + repository = module.get("repository", "") + ref = module.get("ref", "") + pack_path = module.get("pack_path", ".") + skill_paths = module.get("skills", []) + + result = ModuleResult(name=name, repository=repository, ref=ref, pack_path=pack_path) + + if not repository or not ref: + result.error = "Missing repository or ref" + return result + + clone_dest = base_dir / name + err = clone_at_ref(repository, ref, clone_dest) + if err: + result.error = f"Clone failed: {err}" + return result + + result.clone_ok = True + result.clone_path = str(clone_dest) + pack_root = clone_dest / pack_path + + if not validate: + return result + + if skill_paths: + skill_dirs = [] + for sp in skill_paths: + if sp.endswith("/SKILL.md"): + skill_dirs.append(pack_root / Path(sp).parent) + else: + d = pack_root / sp + skill_dirs.append(d.parent if d.is_file() else d) + else: + skills_dir = pack_root / "skills" + skill_dirs = sorted( + d for d in skills_dir.iterdir() + if d.is_dir() and (d / "SKILL.md").exists() + ) if skills_dir.is_dir() else [] + + for skill_dir in skill_dirs: + sr = validate_skill(skill_dir, pack_root) + result.skills.append(sr) + + return result + + +def main() -> int: + parser = argparse.ArgumentParser(description="Fetch and validate federated skills") + parser.add_argument("--json", action="store_true", help="Output results as JSON") + parser.add_argument("--fetch-only", action="store_true", help="Clone repos without validating") + parser.add_argument("--output-dir", type=str, help="Directory to clone repos into (default: temp dir)") + parser.add_argument("--module", type=str, help="Validate only this module name") + args = parser.parse_args() + + modules = pack_registry.load_federated_modules() + if not modules: + if args.json: + print(json.dumps({"modules": [], "summary": "No federated modules configured"})) + else: + print("No federated modules configured in marketplace YAML.") + return 0 + + if args.module: + modules = [m for m in modules if m.get("name") == args.module] + if not modules: + print(f"Module '{args.module}' not found in federated_modules", file=sys.stderr) + return 1 + + use_temp = args.output_dir is None + base_dir = Path(tempfile.mkdtemp(prefix="federated-")) if use_temp else Path(args.output_dir) + base_dir.mkdir(parents=True, exist_ok=True) + + results: List[ModuleResult] = [] + any_failure = False + + for mod in modules: + if not args.json: + print(f"\n{'='*60}") + print(f"Module: {mod.get('name', '?')}") + print(f" Repository: {mod.get('repository', '?')}") + print(f" Ref: {mod.get('ref', '?')}") + print(f" Pack path: {mod.get('pack_path', '.')}") + print(f"{'='*60}") + + mr = process_module(mod, base_dir, validate=not args.fetch_only) + results.append(mr) + + if not args.json: + if not mr.clone_ok: + print(f" CLONE FAILED: {mr.error}") + any_failure = True + continue + + print(f" Cloned to: {mr.clone_path}") + + if args.fetch_only: + print(" (fetch-only mode, skipping validation)") + continue + + for sr in mr.skills: + status = "PASS" if sr.passed else "FAIL" + warn = " (warnings)" if sr.warnings else "" + print(f" [{status}{warn}] {sr.path}") + if not sr.passed: + any_failure = True + + elif mr.error or any(not s.passed for s in mr.skills): + any_failure = True + + if args.json: + output = { + "modules": [asdict(r) for r in results], + "all_passed": not any_failure, + } + print(json.dumps(output, indent=2)) + + if use_temp and not args.fetch_only: + shutil.rmtree(base_dir, ignore_errors=True) + + if not args.json: + print() + total_skills = sum(len(r.skills) for r in results) + passed = sum(1 for r in results for s in r.skills if s.passed) + failed = sum(1 for r in results for s in r.skills if not s.passed) + print(f"Summary: {passed}/{total_skills} skills passed, {failed} failed") + + return 1 if any_failure else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/generate_pack_data.py b/scripts/generate_pack_data.py index 3d42c6b6..167ad8ad 100644 --- a/scripts/generate_pack_data.py +++ b/scripts/generate_pack_data.py @@ -288,6 +288,98 @@ def parse_docs(pack_dir: str) -> List[Dict[str, Any]]: return sorted(docs, key=lambda d: (d['category'], d['title'])) +def load_federated_packs() -> List[Dict[str, Any]]: + """ + Fetch federated modules and return them as standalone pack entries. + + Each federated pack gets ``source: "federated"`` so the docs site can + badge it differently from in-tree packs. + """ + import shutil + import subprocess + import tempfile + + modules = pack_registry.load_federated_modules() + if not modules: + return [] + + packs: List[Dict[str, Any]] = [] + tmp = Path(tempfile.mkdtemp(prefix="federated-build-")) + + try: + for mod in modules: + name = mod.get("name", "unknown") + repository = mod.get("repository", "") + ref = mod.get("ref", "") + description = mod.get("description", "") + version = mod.get("version", "0.0.0") + license_id = mod.get("license", "Unknown") + tags = mod.get("tags", []) + pack_path = mod.get("pack_path", ".") + skill_subset = mod.get("skills") + + if not repository or not ref: + print(f" Warning: federated module '{name}' missing repository or ref, skipping") + continue + + clone_dest = tmp / name + try: + subprocess.run( + ["git", "clone", "--quiet", "--no-checkout", repository, str(clone_dest)], + check=True, capture_output=True, text=True, timeout=120, + ) + subprocess.run( + ["git", "checkout", "--quiet", ref], + check=True, capture_output=True, text=True, cwd=clone_dest, timeout=30, + ) + except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as exc: + print(f" Warning: failed to clone '{name}': {exc}") + continue + + pack_dir = clone_dest / pack_path + + skills = [] + if skill_subset: + for sp in skill_subset: + skill_file = pack_dir / sp if sp.endswith("/SKILL.md") else pack_dir / sp / "SKILL.md" + if skill_file.exists(): + fm = parse_yaml_frontmatter(skill_file) + skill_name = fm.get("name", skill_file.parent.name) + desc = fm.get("description", "") + if isinstance(desc, str): + desc = " ".join(desc.split()) + skills.append({"name": skill_name, "description": desc, "file_path": sp}) + else: + skills = parse_skills(str(pack_dir)) + + pack = { + "name": name, + "path": repository, + "source": "federated", + "repository": repository, + "ref": ref[:12], + "plugin": { + "name": name, + "title": name, + "version": version, + "description": description, + "author": {"name": "External"}, + "license": license_id, + "keywords": tags, + }, + "skills": sorted(skills, key=lambda s: s["name"]), + "agents": [], + "docs": [], + "has_readme": (pack_dir / "README.md").exists(), + } + packs.append(pack) + print(f" ✓ Federated '{name}': {len(skills)} skill(s) from {repository}") + finally: + shutil.rmtree(tmp, ignore_errors=True) + + return packs + + def generate_pack_data() -> List[Dict[str, Any]]: """ Generate pack data for all agentic packs. @@ -296,7 +388,7 @@ def generate_pack_data() -> List[Dict[str, Any]]: List of pack dictionaries """ packs = [] - + # Load plugin title mappings from docs/plugins.json plugin_titles = load_plugin_titles() @@ -328,6 +420,11 @@ def generate_pack_data() -> List[Dict[str, Any]]: plugin_title = pack['plugin'].get('title', pack_dir) print(f"✓ Parsed {plugin_title}: {len(pack['skills'])} skills, {len(pack['agents'])} agents, {len(docs)} docs") + federated = load_federated_packs() + if federated: + packs.extend(federated) + print(f"✓ Added {len(federated)} federated pack(s)") + return packs diff --git a/scripts/pack_registry.py b/scripts/pack_registry.py index 390b2c05..2b2f28f7 100644 --- a/scripts/pack_registry.py +++ b/scripts/pack_registry.py @@ -85,6 +85,21 @@ def load_marketplace_module_by_path( return None +def load_federated_modules( + marketplace_path: Optional[Path] = None, +) -> List[Dict[str, Any]]: + """Return the list of federated module entries from marketplace YAML.""" + path = marketplace_path or (_repo_root() / DEFAULT_MARKETPLACE) + if not path.exists(): + return [] + with open(path, "r", encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + federated = data.get("federated_modules") or [] + if not isinstance(federated, list): + return [] + return [m for m in federated if isinstance(m, dict)] + + def load_plugin_title(pack_dir: str, repo_root: Optional[Path] = None) -> Optional[str]: root = repo_root or _repo_root() p = root / DEFAULT_PLUGINS_JSON diff --git a/scripts/validate_federation.py b/scripts/validate_federation.py new file mode 100755 index 00000000..2c2d69c1 --- /dev/null +++ b/scripts/validate_federation.py @@ -0,0 +1,352 @@ +#!/usr/bin/env python3 +""" +Validate an external repository for federation into Red Hat Agentic Collections. + +Automates the mechanical checks from docs/FEDERATION_REVIEW_GUIDE.md: + 1. Clone at pinned ref + 2. Verify Lola pack structure + 3. Tier 1 validation (agentskills.io spec) + 4. Tier 2 validation (design principles) + 5. MCP version pinning (no :latest) + 6. Credential leak scan (gitleaks) + +Usage: + python scripts/validate_federation.py [--pack-path ] [--skills skill1 skill2] + python scripts/validate_federation.py --json + +Examples: + # Validate entire pack at repo root + python scripts/validate_federation.py https://github.com/org/repo v1.0.0 + + # Pack lives in a subdirectory + python scripts/validate_federation.py https://github.com/org/repo v1.0.0 --pack-path my-pack + + # Validate only specific skills + python scripts/validate_federation.py https://github.com/org/repo abc123 --skills sdn-diagnostics ovn-trace +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +from dataclasses import dataclass, field, asdict +from pathlib import Path + + +REQUIRED_FILES = ["CLAUDE.md", "README.md", "mcps.json"] +REQUIRED_DIRS = ["skills"] +REPO_ROOT = Path(__file__).resolve().parent.parent + + +@dataclass +class CheckResult: + name: str + passed: bool = False + skipped: bool = False + details: list[str] = field(default_factory=list) + + +@dataclass +class ValidationReport: + repository: str + ref: str + checks: list[CheckResult] = field(default_factory=list) + + @property + def all_passed(self) -> bool: + return all(c.passed for c in self.checks) + + +def clone_at_ref(repo_url: str, ref: str, dest: Path) -> CheckResult: + check = CheckResult(name="clone") + try: + subprocess.run( + ["git", "clone", "--quiet", "--no-checkout", repo_url, str(dest)], + check=True, capture_output=True, text=True, timeout=120, + ) + subprocess.run( + ["git", "checkout", "--quiet", ref], + check=True, capture_output=True, text=True, cwd=dest, timeout=30, + ) + check.passed = True + check.details.append(f"Cloned and checked out {ref}") + except subprocess.CalledProcessError as exc: + check.passed = False + check.details.append(exc.stderr.strip() or str(exc)) + except subprocess.TimeoutExpired: + check.passed = False + check.details.append("Git operation timed out") + return check + + +def check_lola_structure(pack_dir: Path) -> CheckResult: + check = CheckResult(name="lola_structure") + missing = [] + for f in REQUIRED_FILES: + if not (pack_dir / f).exists(): + missing.append(f) + for d in REQUIRED_DIRS: + if not (pack_dir / d).is_dir(): + missing.append(f"{d}/") + + skills = list((pack_dir / "skills").glob("*/SKILL.md")) if (pack_dir / "skills").is_dir() else [] + + if missing: + check.passed = False + check.details.append(f"Missing: {', '.join(missing)}") + elif not skills: + check.passed = False + check.details.append("No skills found in skills/") + else: + check.passed = True + check.details.append(f"Structure valid: {len(skills)} skill(s) found") + return check + + +def run_tier1(pack_dir: Path, skill_subset: list[str] | None = None) -> CheckResult: + check = CheckResult(name="tier1_agentskills") + linter = REPO_ROOT / ".claude" / "skills" / "skill-linter" / "scripts" / "validate-skill.sh" + if not linter.exists(): + check.passed = False + check.details.append("Linter script not found in agentic-collections repo") + return check + + skills_dir = pack_dir / "skills" + if skill_subset: + skill_dirs = [skills_dir / s for s in skill_subset] + else: + skill_dirs = sorted(d for d in skills_dir.iterdir() if d.is_dir() and (d / "SKILL.md").exists()) + + if not skill_dirs: + check.passed = False + check.details.append("No skills to validate") + return check + + passed = 0 + failed = 0 + for sd in skill_dirs: + if not (sd / "SKILL.md").exists(): + check.details.append(f"FAIL {sd.name}: SKILL.md not found") + failed += 1 + continue + try: + result = subprocess.run( + [str(linter), str(sd)], + capture_output=True, text=True, timeout=30, + ) + if result.returncode == 0: + passed += 1 + if "[WARN]" in result.stdout: + check.details.append(f"WARN {sd.name}: passed with warnings") + else: + check.details.append(f"PASS {sd.name}") + else: + failed += 1 + check.details.append(f"FAIL {sd.name}: linter errors") + except subprocess.TimeoutExpired: + failed += 1 + check.details.append(f"FAIL {sd.name}: timed out") + + check.passed = failed == 0 + check.details.insert(0, f"{passed}/{passed + failed} skills passed Tier 1") + return check + + +def run_tier2(pack_dir: Path) -> CheckResult: + check = CheckResult(name="tier2_design_principles") + validator = REPO_ROOT / "scripts" / "validate_skill_design.py" + if not validator.exists(): + check.passed = False + check.details.append("Design validator not found") + return check + + try: + result = subprocess.run( + [sys.executable, str(validator), str(pack_dir)], + capture_output=True, text=True, timeout=60, + ) + if result.returncode == 0: + check.passed = True + check.details.append("All skills passed Tier 2") + else: + check.passed = False + for line in result.stdout.strip().splitlines(): + if line.strip() and ("❌" in line or "•" in line): + check.details.append(line.strip()) + if not check.details: + check.details.append("Tier 2 validation failed (see output above)") + except subprocess.TimeoutExpired: + check.passed = False + check.details.append("Validation timed out") + return check + + +def check_mcp_pinning(pack_dir: Path) -> CheckResult: + check = CheckResult(name="mcp_version_pinning") + mcps_file = pack_dir / "mcps.json" + + if not mcps_file.exists(): + check.passed = True + check.details.append("No mcps.json found (no MCP servers)") + return check + + try: + with open(mcps_file, "r", encoding="utf-8") as f: + data = json.load(f) + except (json.JSONDecodeError, OSError) as exc: + check.passed = False + check.details.append(f"Failed to parse mcps.json: {exc}") + return check + + servers = data.get("mcpServers", {}) + if not servers: + check.passed = True + check.details.append("mcps.json has no servers configured") + return check + + latest_found = [] + hardcoded_creds = [] + + for name, srv in servers.items(): + args_str = " ".join(srv.get("args", [])) + if ":latest" in args_str: + latest_found.append(name) + + for k, v in srv.get("env", {}).items(): + if isinstance(v, str) and v.strip() and not v.startswith("${"): + hardcoded_creds.append(f"{name}.env.{k}") + + if latest_found: + check.details.append(f":latest found in: {', '.join(latest_found)}") + if hardcoded_creds: + check.details.append(f"Hardcoded credentials in: {', '.join(hardcoded_creds)}") + + check.passed = not latest_found and not hardcoded_creds + if check.passed: + check.details.append(f"{len(servers)} server(s) checked: all versions pinned, no hardcoded credentials") + return check + + +def run_gitleaks(pack_dir: Path) -> CheckResult: + check = CheckResult(name="gitleaks") + if not shutil.which("gitleaks"): + check.passed = True + check.skipped = True + check.details.append("gitleaks not installed — SKIPPED (install for full scan)") + return check + + try: + result = subprocess.run( + ["gitleaks", "detect", "--source", str(pack_dir), "--no-git", "--quiet"], + capture_output=True, text=True, timeout=60, + ) + if result.returncode == 0: + check.passed = True + check.details.append("No secrets detected") + else: + check.passed = False + check.details.append("Potential secrets detected") + for line in result.stdout.strip().splitlines()[:10]: + check.details.append(f" {line}") + except subprocess.TimeoutExpired: + check.passed = False + check.details.append("gitleaks timed out") + return check + + +def print_report(report: ValidationReport) -> None: + print() + print("=" * 60) + print("Federation Validation Report") + print(f" Repository: {report.repository}") + print(f" Ref: {report.ref}") + print("=" * 60) + + for c in report.checks: + if c.skipped: + status = "SKIP" + icon = "⚠️" + elif c.passed: + status = "PASS" + icon = "✅" + else: + status = "FAIL" + icon = "❌" + print(f"\n{icon} [{status}] {c.name}") + for d in c.details: + print(f" {d}") + + print() + print("=" * 60) + if report.all_passed: + print("✅ ALL CHECKS PASSED — ready for federation approval") + else: + failed = [c.name for c in report.checks if not c.passed] + print(f"❌ FAILED CHECKS: {', '.join(failed)}") + print("=" * 60) + print() + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Validate an external repo for federation" + ) + parser.add_argument("repo_url", help="Public repository URL") + parser.add_argument("ref", help="Commit SHA or release tag") + parser.add_argument("--pack-path", default=".", help="Path to the pack within the repo (default: repo root)") + parser.add_argument("--skills", nargs="*", help="Validate only these skills (by directory name)") + parser.add_argument("--json", action="store_true", help="Output as JSON") + parser.add_argument("--keep-clone", action="store_true", help="Don't delete the cloned repo after validation") + args = parser.parse_args() + + report = ValidationReport(repository=args.repo_url, ref=args.ref) + tmp = Path(tempfile.mkdtemp(prefix="federation-review-")) + + try: + # Step 1: Clone + clone_result = clone_at_ref(args.repo_url, args.ref, tmp / "repo") + report.checks.append(clone_result) + if not clone_result.passed: + if args.json: + print(json.dumps(asdict(report), indent=2)) + else: + print_report(report) + return 1 + + pack_dir = tmp / "repo" / args.pack_path + + # Step 2: Lola structure + report.checks.append(check_lola_structure(pack_dir)) + + # Step 3: Tier 1 + report.checks.append(run_tier1(pack_dir, args.skills)) + + # Step 4: Tier 2 + report.checks.append(run_tier2(pack_dir)) + + # Step 5: MCP pinning + report.checks.append(check_mcp_pinning(pack_dir)) + + # Step 6: Gitleaks + report.checks.append(run_gitleaks(pack_dir)) + + finally: + if not args.keep_clone: + shutil.rmtree(tmp, ignore_errors=True) + else: + print(f"Clone kept at: {tmp / 'repo'}") + + if args.json: + print(json.dumps(asdict(report), indent=2)) + else: + print_report(report) + + return 0 if report.all_passed else 1 + + +if __name__ == "__main__": + sys.exit(main())