Skip to content

fix(policies): add NOT_IN operator to enum, evaluator, bridge, schema, and CLI#2373

Open
DhineshPonnarasan wants to merge 1 commit into
microsoft:mainfrom
DhineshPonnarasan:fix/not-in-operator-policy-runtime
Open

fix(policies): add NOT_IN operator to enum, evaluator, bridge, schema, and CLI#2373
DhineshPonnarasan wants to merge 1 commit into
microsoft:mainfrom
DhineshPonnarasan:fix/not-in-operator-policy-runtime

Conversation

@DhineshPonnarasan
Copy link
Copy Markdown
Contributor

@DhineshPonnarasan DhineshPonnarasan commented May 18, 2026

Summary

not_in was accepted by the policy linter and shared validator but broke silently at runtime due to two bugs:

  1. PolicyOperator enum missing NOT_INPolicyDocument.from_yaml() raised ValidationError for any policy using operator: not_in.
  2. Bridge corruption_OPERATOR_MAP_TO_POLICY mapped "not_in"PolicyOperator.NE (scalar !=), silently rewriting allowlist-exclusion rules into scalar inequality checks.

This PR is an additive-only fix at all five operator coupling points. No existing operator semantics are changed.

Root Cause

# This policy would raise ValidationError before this fix
rules:
  - name: deny-unapproved-tool
    condition:
      field: tool_name
      operator: not_in        # ← no corresponding enum member
      value: [read_file, search]
    action: deny

Changes

File Change
schema.py Add NOT_IN = "not_in" to PolicyOperator enum
evaluator.py Add NOT_IN branch in _match_condition() with list-exclusion semantics
shared.py Fix bridge map "not_in"PolicyOperator.NOT_IN (was NE); add reverse mapping
cli.py Add not_in branch in _evaluate_condition(); remove stale List import
policy_schema.json Add "not_in" to PolicyOperator.enum

Files unchanged (pre-existing behavior preserved)

  • bridge.pydocument_to_governance() data loss for non-auto-generated rule names is pre-existing architectural debt, not introduced here
  • lint_policy.pyKNOWN_OPERATORS already contained "not_in" before this PR

Tests Added

12 new regression tests across 3 files:

Test File Purpose
test_not_in_operator_loads_from_yaml test_policy_language.py YAML deserialization no longer raises
test_not_in_inside_allowlist_does_not_match test_policy_language.py In-list → condition not satisfied
test_not_in_outside_allowlist_matches test_policy_language.py Out-of-list → condition satisfied
test_not_in_differs_from_ne test_policy_language.py Key: proves NOT_IN ≠ NE (the original bridge bug)
test_not_in_missing_field_no_match test_policy_language.py Missing field → no match (consistent with all operators)
test_not_in_malformed_target_fails_closed test_policy_language.py Non-iterable target → fail-closed
test_not_in_empty_allowlist_matches_everything test_policy_language.py value=[] → deny fires for every present value (intentional Python membership semantics)
test_not_in_end_to_end_from_yaml test_policy_language.py Full YAML → load → evaluate roundtrip
test_not_in_mapping_to_policy_operator test_shared_policy.py Bridge: "not_in"PolicyOperator.NOT_IN
test_not_in_mapping_from_policy_operator test_shared_policy.py Bridge: PolicyOperator.NOT_IN"not_in"
test_not_in_roundtrip_bridge test_shared_policy.py SharedPolicySchemaPolicyDocument → back
test_not_in_scenario_semantics test_policy_cli.py CLI test subcommand: 2/2 scenarios pass

Pre-existing Issues (not introduced, not fixed in this PR)

  • re.match (anchored) in cli.py vs re.search (anywhere) in evaluator.py for MATCHES — semantic drift in the CLI dev tool only
  • bridge.py reverse map: GTE → "gt" and LTE → "lt" data loss
  • Three parallel condition evaluator implementations (architectural debt)

Test Results

78 targeted tests passing, 223 broader policy tests passing, lint clean (ruff)

Prior Art

Standard Python membership semantics (in / not in). No external prior art. This operator was already documented in the policy schema JSON and accepted by the linter — the implementation simply completes the missing runtime and deserialization wiring.


Fixes #2372.

@imran-siddique — would appreciate a review when you get a chance. The change is additive-only (no existing operator semantics touched), and all 78 targeted tests plus the broader 223-test policy suite are green. Happy to address any feedback.

…, and CLI

- Add PolicyOperator.NOT_IN enum member to schema.py
- Implement list-exclusion semantics in _match_condition() (evaluator.py)
- Correct shared policy bridge: not_in now maps to NOT_IN (was NE)
- Add NOT_IN to _OPERATOR_MAP_FROM_POLICY reverse mapping (shared.py)
- Add not_in branch to CLI _evaluate_condition() (cli.py)
- Add 'not_in' to PolicyOperator enum in policy_schema.json

Previously PolicyDocument.from_yaml() raised ValidationError on policies
using operator: not_in, and the bridge silently mapped not_in to scalar NE
semantics, corrupting allowlist-exclusion rules.

Regression tests added for:
  - YAML loading and deserialization
  - evaluator semantics (in-list vs out-of-list, missing field, malformed target)
  - NOT_IN vs NE semantic difference
  - bridge roundtrip (shared <-> PolicyDocument)
  - CLI test-runner scenarios
  - end-to-end from YAML policy file

77 targeted tests passing, 223 broader policy tests passing, lint clean.
@github-actions github-actions Bot added the tests label May 18, 2026
@github-actions
Copy link
Copy Markdown

🤖 AI Agent: breaking-change-detector — API Compatibility

API Compatibility

No breaking changes detected.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: security-scanner — View details

No security issues found.

@github-actions github-actions Bot added the size/L Large PR (< 500 lines) label May 18, 2026
@github-actions
Copy link
Copy Markdown

🤖 AI Agent: docs-sync-checker — Docs Sync

Docs Sync

  • PolicyOperator.NOT_IN in schema.py -- missing docstring
  • README.md -- update required to reflect the addition of the NOT_IN operator
  • CHANGELOG.md -- missing entry for the addition of the NOT_IN operator and its behavioral implications

Please address these documentation issues.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: code-reviewer — View details

TL;DR: 0 blockers, 1 warning. Solid fix with minor follow-up needed.

# Sev Issue Where
1 Warn Pre-existing semantic drift in MATCHES cli.py, evaluator.py

Action items: None.

Warnings (fine as follow-up PRs)
Address semantic drift between re.match in cli.py and re.search in evaluator.py for MATCHES operator.

@github-actions
Copy link
Copy Markdown

🤖 AI Agent: test-generator — View details

Test coverage looks good. No gaps identified.

@github-actions
Copy link
Copy Markdown

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential NONE
Overall MEDIUM

Automated check by AGT Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label May 18, 2026
@github-actions
Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Warning See details
🛡️ Security Scan ✅ Passed No issues found
🔄 Breaking Changes ✅ Passed No issues found
📝 Docs Sync ✅ Completed Analysis complete
🧪 Test Coverage ✅ Completed Analysis complete

Verdict: ⚠️ Ready for human review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-review:MEDIUM Contributor check flagged MEDIUM risk size/L Large PR (< 500 lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: not_in operator passes linting but crashes at runtime and is silently mis-mapped in the SharedPolicySchema bridge

1 participant