[FR] Includes deprecated rule stubs to the package for upstream testing

[dplumlee]

Pull Request

Issue link(s): https://github.com/elastic/security-team/issues/6344

One-pager with additional context: internal link

Summary - What I changed

Includes deprecated rule stub objects with the rules package via the packaging script. Will live under the security_rule/ folder the same as active rules. Uses the existing version lock to align with the rest of the rules in the package and track for any future use. Only includes deprecated assets if building for a ^9.4.0 version to protect against backporting to older kibana versions that don't support this saved object formatting.

The deprecated rule objects currently have the following schema:

{
         "id": asset_id,
         "type": "security-rule",
          "attributes": {
                "rule_id": rule_id,
                 "version": number,
                 "name": string,
                 "deprecated": boolean (true),
                 "deprecated_reason" string (optional) // Only added if deprecated_reason exists in deprecation data
           },
}

How To Test

Manually kicking off the release fleet workflow targeting this branch: elastic/integrations#17866

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[Mikaayenson]

@dplumlee In general this looks good to me. Take a look at this diff which will add deprecated_reason. I'm not sure when/if we actually plan to support in Kibana, but here it is. It also includes a unit test.

Rule Deprecation Reason

diff --git a/detection_rules/packaging.py b/detection_rules/packaging.py
index 7e68442b1..9a0f1c7c3 100644
--- a/detection_rules/packaging.py
+++ b/detection_rules/packaging.py
@@ -29,12 +29,68 @@ from .utils import Ndjson, get_etc_path, get_path
 from .version_lock import loaded_version_lock
 
 RULES_CONFIG = parse_rules_config()
+
+# Minimum stack version that supports deprecated rule stubs in the package (Kibana SO mapping).
+# When this repo is backported to 8.latest, 9.latest-1, 9.latest-2, etc., packages are built
+# for that branch's stack; deprecated_reason is only added when stack_version >= 9.4.
+#
+# Rule deprecation workflows (must support both; 8.19 has no Kibana deprecation feature):
+#
+# - Pre-9.4 (e.g. 8.19), two-stage: (1) Add "Deprecated - " prefix to rule name, set
+#   maturity = "deprecated" and deprecation_date in [metadata], optionally reduce risk/severity
+#   (see in-place deprecation PRs). Ship so customers see the rule is deprecated. (2) Next
+#   release move the rule to rules/_deprecated/ or remove. The package filter (packages.yaml
+#   maturity) controls which rules ship as full rules; to ship stage-1 deprecated rules as
+#   full rules on pre-9.4 branches, include "deprecated" in the filter for that package.
+#
+# - 9.4+, one-step: Move rule to rules/_deprecated/, set maturity = "deprecated" and
+#   deprecation_date; optional deprecated_reason in [metadata]. Package emits stubs only
+#   (no full rule); Kibana shows deprecation UI. deprecated_reason is only added to the
+#   stub when stack_version >= 9.4.
+MIN_STACK_VERSION_DEPRECATED_STUBS = Version.parse("9.4.0")
+
 RELEASE_DIR = get_path(["releases"])
 PACKAGE_FILE = str(RULES_CONFIG.packages_file)
 NOTICE_FILE = get_path(["NOTICE.txt"])
 FLEET_PKG_LOGO = get_etc_path(["security-logo-color-64px.svg"])
 
 
+def build_deprecated_rule_asset(
+    rule_id: str,
+    rule_name: str,
+    deprecated_version: int,
+    deprecated_reason: str | None = None,
+    stack_version: Version | None = None,
+) -> dict[str, Any]:
+    """Build the saved-object dict for a deprecated rule stub (package asset).
+
+    Used when generating the registry package for stack versions that support
+    deprecated rule stubs (9.4.0+). Caller is responsible for version gate.
+
+    The deprecated_reason attribute is a 9.4+ Kibana feature; it is only added
+    when stack_version >= 9.4.0 and a non-empty value is provided. This ensures
+    that when the code is backported to 8.latest / 9.latest-1 / 9.latest-2, the
+    package built for those stacks will not include deprecated_reason.
+    """
+    asset_id = f"{rule_id}_{deprecated_version}"
+    attributes: dict[str, Any] = {
+        "rule_id": rule_id,
+        "version": deprecated_version,
+        "name": rule_name,
+        "deprecated": True,
+    }
+    # deprecated_reason is only available on 9.4+ (Kibana feature)
+    if deprecated_reason and (
+        stack_version is not None and stack_version >= MIN_STACK_VERSION_DEPRECATED_STUBS
+    ):
+        attributes["deprecated_reason"] = deprecated_reason
+    return {
+        "id": asset_id,
+        "type": definitions.SAVED_OBJECT_TYPE,
+        "attributes": attributes,
+    }
+
+
 def filter_rule(rule: TOMLRule, config_filter: dict[str, Any], exclude_fields: dict[str, Any] | None = None) -> bool:
     """Filter a rule based off metadata and a package configuration."""
     flat_rule = rule.contents.flattened_dict()
@@ -285,7 +341,9 @@ class Package:
 
         rules = all_rules.filter(lambda r: filter_rule(r, rule_filter, exclude_fields))
 
-        # add back in deprecated fields
+        # Deprecated rules are always attached; they are written as stubs for 9.4+ only.
+        # For pre-9.4 two-stage workflow, include "deprecated" in filter.maturity to ship
+        # them as full rules (e.g. with "Deprecated - " prefix) so customers see the deprecation.
         rules.deprecated = all_rules.deprecated
 
         if verbose:
@@ -481,7 +539,7 @@ class Package:
 
         # Only generate deprecated rule assets for 9.4.0+, where Kibana has
         # the SO mapping and logic to handle them
-        if stack_version >= Version.parse("9.4.0"):
+        if stack_version >= MIN_STACK_VERSION_DEPRECATED_STUBS:
             deprecated_lock = loaded_version_lock.deprecated_lock
             version_lock = loaded_version_lock.version_lock
 
@@ -491,20 +549,14 @@ class Package:
                     continue
 
                 deprecated_version = lock_entry.version + 1
-                asset_id = f"{rule_id}_{deprecated_version}"
-
-                asset = {
-                    "id": asset_id,
-                    "type": definitions.SAVED_OBJECT_TYPE,
-                    "attributes": {
-                        "rule_id": rule_id,
-                        "version": deprecated_version,
-                        "name": dep_entry.rule_name,
-                        "deprecated": True,
-                    },
-                }
-
-                asset_path = rules_dir / f"{asset_id}.json"
+                asset = build_deprecated_rule_asset(
+                    rule_id=rule_id,
+                    rule_name=dep_entry.rule_name,
+                    deprecated_version=deprecated_version,
+                    deprecated_reason=getattr(dep_entry, "deprecated_reason", None),
+                    stack_version=stack_version,
+                )
+                asset_path = rules_dir / f"{asset['id']}.json"
                 asset_path.write_text(json.dumps(asset, indent=4, sort_keys=True), encoding="utf-8")
 
         notice_contents = NOTICE_FILE.read_text()
diff --git a/detection_rules/version_lock.py b/detection_rules/version_lock.py
index 1c6d893bd..8e898f745 100644
--- a/detection_rules/version_lock.py
+++ b/detection_rules/version_lock.py
@@ -74,6 +74,7 @@ class DeprecatedRulesEntry(MarshmallowDataclassMixin):
     deprecation_date: definitions.Date | definitions.KNOWN_BAD_DEPRECATED_DATES
     rule_name: definitions.RuleName
     stack_version: definitions.SemVer
+    deprecated_reason: str | None = None  # Optional; from rule [metadata] when maturity == "deprecated"
 
 
 @dataclass(frozen=True)
@@ -337,11 +338,15 @@ class VersionLock:
 
         for rule in rules.deprecated:
             if rule.id in newly_deprecated:
-                current_deprecated_lock[rule.id] = {
+                entry: dict[str, Any] = {
                     "rule_name": rule.name,
                     "stack_version": current_stack_version(),
                     "deprecation_date": rule.contents.metadata["deprecation_date"],
                 }
+                # deprecated_reason is set in the rule TOML [metadata]; copy into lock when present.
+                if rule.contents.metadata.get("deprecated_reason"):
+                    entry["deprecated_reason"] = rule.contents.metadata["deprecated_reason"]
+                current_deprecated_lock[rule.id] = entry
 
         if save_changes or verbose:
             click.echo(f" - {len(changed_rules)} changed rules")
diff --git a/tests/test_packages.py b/tests/test_packages.py
index c6ed9e7e8..e22f7c580 100644
--- a/tests/test_packages.py
+++ b/tests/test_packages.py
@@ -7,12 +7,20 @@
 
 import unittest
 import uuid
+from pathlib import Path
 
 from marshmallow import ValidationError
 from semver import Version
 
 from detection_rules import rule_loader
-from detection_rules.packaging import PACKAGE_FILE, Package
+from detection_rules.rule_loader import RuleCollection
+from detection_rules.packaging import (
+    MIN_STACK_VERSION_DEPRECATED_STUBS,
+    PACKAGE_FILE,
+    Package,
+    build_deprecated_rule_asset,
+)
+from detection_rules.schemas import definitions
 from detection_rules.schemas.registry_package import RegistryPackageManifestV1, RegistryPackageManifestV3
 from tests.base import BaseRuleTest
 
@@ -108,3 +116,87 @@ class TestRegistryPackage(unittest.TestCase):
 
         with self.assertRaises(ValidationError):
             RegistryPackageManifestV1.from_dict(registry_config)
+
+
+class TestDeprecatedRuleAsset(unittest.TestCase):
+    """Test deprecated rule stub asset building and version gate."""
+
+    def test_build_deprecated_rule_asset_structure(self):
+        """build_deprecated_rule_asset returns a dict with id, type, and attributes."""
+        asset = build_deprecated_rule_asset(
+            rule_id="aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee",
+            rule_name="Deprecated Rule Name",
+            deprecated_version=3,
+        )
+        self.assertIsInstance(asset, dict)
+        self.assertEqual(asset["id"], "aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee_3")
+        self.assertEqual(asset["type"], definitions.SAVED_OBJECT_TYPE)
+        self.assertIn("attributes", asset)
+        self.assertEqual(asset["attributes"]["rule_id"], "aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee")
+        self.assertEqual(asset["attributes"]["version"], 3)
+        self.assertEqual(asset["attributes"]["name"], "Deprecated Rule Name")
+        self.assertIs(asset["attributes"]["deprecated"], True)
+
+    def test_build_deprecated_rule_asset_serializable(self):
+        """Asset is JSON-serializable and matches expected shape for package."""
+        import json
+
+        asset = build_deprecated_rule_asset(
+            rule_id="00000000-0000-4000-8000-000000000001",
+            rule_name="Test",
+            deprecated_version=1,
+        )
+        # Should not raise
+        json_str = json.dumps(asset, indent=4, sort_keys=True)
+        loaded = json.loads(json_str)
+        self.assertEqual(loaded["attributes"]["deprecated"], True)
+        self.assertEqual(loaded["type"], "security-rule")
+
+    def test_build_deprecated_rule_asset_deprecated_reason_only_on_94(self):
+        """deprecated_reason is only added when stack_version is 9.4+ (Kibana feature)."""
+        reason = "Replaced by rule X"
+        # With stack_version None or < 9.4, deprecated_reason must not appear
+        for stack_ver in (None, Version(9, 3, 0)):
+            with self.subTest(stack_version=stack_ver):
+                asset = build_deprecated_rule_asset(
+                    rule_id="aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee",
+                    rule_name="Deprecated Rule",
+                    deprecated_version=2,
+                    deprecated_reason=reason,
+                    stack_version=stack_ver,
+                )
+                self.assertNotIn("deprecated_reason", asset["attributes"])
+        # With stack_version >= 9.4 it appears
+        asset = build_deprecated_rule_asset(
+            rule_id="aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee",
+            rule_name="Deprecated Rule",
+            deprecated_version=2,
+            deprecated_reason=reason,
+            stack_version=Version(9, 4, 0),
+        )
+        self.assertEqual(asset["attributes"]["deprecated_reason"], reason)
+
+    @unittest.skipIf(rule_loader.RULES_CONFIG.bypass_version_lock, "Version lock bypassed")
+    def test_deprecated_rule_load_dict_preserves_deprecated_reason(self):
+        """Loading a deprecated rule dict with deprecated_reason in [metadata] preserves it."""
+        rule_id = "aaaaaaaa-bbbb-4ccc-dddd-eeeeeeeeeeee"
+        reason = "Replaced by rule X"
+        obj = {
+            "metadata": {
+                "creation_date": "2020/01/01",
+                "deprecation_date": "2024/01/01",
+                "updated_date": "2024/01/01",
+                "maturity": "deprecated",
+                "deprecated_reason": reason,
+            },
+            "rule": {
+                "rule_id": rule_id,
+                "name": "Test Deprecated Rule",
+                "description": "Minimal deprecated rule for test",
+            },
+        }
+        collection = RuleCollection()
+        path = Path("rules/_deprecated/test_deprecated_reason.toml")
+        rule = collection.load_dict(obj, path=path)
+        self.assertIn("deprecated_reason", rule.contents.metadata)
+        self.assertEqual(rule.contents.metadata["deprecated_reason"], reason)

[eric-forte-elastic]

Tested Releases on a 9.4 and 9.5 package release (to test for arbitrary bumps, etc.), with update version lock.

Details

❯ cat releases/9.4/fleet/9.4.0-beta.1/kibana/security_rule/015cca13-8832-49ac-a01b-a396114809f6_211.json 
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: releases/9.4/fleet/9.4.0-beta.1/kibana/security_rule/015cca13-8832-49ac-a01b-a396114809f6_211.json
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "attributes": {
   3   │         "deprecated": true,
   4   │         "name": "Deprecated - AWS Redshift Cluster Creation",
   5   │         "rule_id": "015cca13-8832-49ac-a01b-a396114809f6",
   6   │         "version": 211
   7   │     },
   8   │     "id": "015cca13-8832-49ac-a01b-a396114809f6_211",
   9   │     "type": "security-rule"
  10   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────



❯ cat releases/9.5/fleet/9.5.0-beta.1/kibana/security_rule/015cca13-8832-49ac-a01b-a396114809f6_211.json 
───────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: releases/9.5/fleet/9.5.0-beta.1/kibana/security_rule/015cca13-8832-49ac-a01b-a396114809f6_211.json
───────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │     "attributes": {
   3   │         "deprecated": true,
   4   │         "name": "Deprecated - AWS Redshift Cluster Creation",
   5   │         "rule_id": "015cca13-8832-49ac-a01b-a396114809f6",
   6   │         "version": 211
   7   │     },
   8   │     "id": "015cca13-8832-49ac-a01b-a396114809f6_211",
   9   │     "type": "security-rule"
  10   │ }
───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[eric-forte-elastic]

I think would be great to have a unit test like @Mikaayenson describes in #5813 (review) for when a user may bypass locked versions, but otherwise this patch/fix appears to function as intended based on example run, and seems fine as is too. 👍 Testing Details