Skip to content

Add pluggable verifier interface for skill promotion#1

Closed
nutstrut wants to merge 1 commit into
mainfrom
codex/add-pluggable-verifier-interface-for-skill-promotion
Closed

Add pluggable verifier interface for skill promotion#1
nutstrut wants to merge 1 commit into
mainfrom
codex/add-pluggable-verifier-interface-for-skill-promotion

Conversation

@nutstrut

@nutstrut nutstrut commented May 18, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Introduce a new verified_skills operating mode to gate automatic skill promotion with pluggable verifiers.
  • Provide a safe verification fallback so skill evolution can run even if verifier configuration fails.

Mode comparison

Mode GPU Required Cloud Required Verification Auditability
skills_only No No None Existing skill metadata
verified_skills No No by default Pluggable verifier Verification metadata + optional receipts
rl No Yes Existing RL evaluation Existing training artifacts
madmax No Yes Existing training path Existing training artifacts

Description

  • Add a verification subsystem under metaclaw/verification with a VerifierInterface, VerificationResult, and implementations: LocalVerifier, NullVerifier, RemoteHTTPVerifier, and SettlementWitnessVerifier, plus a factory with build_verifier/build_verifier_safe helpers.
  • Add VerificationConfig and wire a verification field into MetaClawConfig and _DEFAULTS in config_store, and expose config parsing in ConfigStore.to_metaclaw_config and describe.
  • Integrate verifier into the FastAPI proxy (metaclaw/api_server.py) by building a safe verifier, constructing verification packets for evolved skills, attaching verification metadata, auditing rejected/indeterminate promotions, and skipping promotion unless verifier rules permit it.
  • Add verified_skills to the CLI and interactive SetupWizard, treat it similar to skills_only in launcher and environment setup, and update README.md with mode docs and config examples.
  • Extend SkillManager to persist and load optional metadata.json alongside SKILL.md so verifier metadata is stored with skills.
  • Add unit tests in tests/test_verification.py covering NullVerifier, LocalVerifier behavior, and factory/fallback selection.

Testing

  • Ran the new unit tests with pytest -q tests/test_verification.py and all tests passed.

  • Verified the verification factory fallback by creating a MetaClawConfig with a bad verifier and asserting build_verifier_safe returns a NullVerifier (test passed).

  • Exercised LocalVerifier cases in unit tests covering fail/duplicate/indeterminate/pass verdicts (all assertions succeeded).

  • Focused verifier/runtime tests passed:

  • pytest -q tests/test_verification.py tests/test_memory_system.py tests/test_runtime_state.py

  • Result: 537 passed


Codex Task

@nutstrut nutstrut changed the title Add verified_skills mode with verifier pipeline and skill metadata support Add pluggable verifier interface for skill promotion May 18, 2026
@nutstrut nutstrut marked this pull request as ready for review May 18, 2026 20:38
@nutstrut

Copy link
Copy Markdown
Owner Author

Closing this fork-local PR in favor of the upstream PR at aiming-lab#74.

@nutstrut nutstrut closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant