Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,7 @@ If AIGuard preserves EdgeEnv/Orchestrator `candidate_context.producer` lineage,

If EdgeEnv preserves an Orchestrator `operation_risk_summary`, Lab shows the compact queue-pressure, max-pressure task, worker-health, and producer/device-local event markers as navigation context in the Runtime Intelligence Risk Summary. These markers help reviewers find the relevant operation evidence, but they do not become EdgeEnv regression deltas, comparability fields, or a deployment decision override.
Lab also renders a separate `Orchestrator queue/deadline/fallback markers` row when those compact counters are present. That row keeps `queue_pressure_reason`, `max_total_queue_depth` when available, deadline miss count, and fallback count together so reviewers can spot operation pressure before reading the detailed task rollup.
When AIGuard raw context preserves `orchestrator_candidate_operation_max_total_queue_depth`, Lab also renders an `AIGuard max queue raw-context traceability` row. This ties the visible `max_total_queue_depth` report marker back to deterministic AIGuard context and Orchestrator operation evidence without changing Lab's deployment decision ownership.

When EdgeEnv/Orchestrator context includes reviewer-facing duration metadata,
Lab renders a `Runtime replay duration scope` row with `duration_label`,
Expand Down Expand Up @@ -539,6 +540,7 @@ Markdown and HTML reports include a Runtime Intelligence Risk Summary that conne
- Runtime history seed and run_config traceability
- Orchestrator operation risk summary markers
- compact queue/deadline/fallback operation markers with `max_total_queue_depth`
- AIGuard raw-context traceability for `max_total_queue_depth`
- Lab EdgeEnv preservation context markers
- device-local producer lineage handoff
- Orchestrator-declared downstream guard alignment
Expand Down
6 changes: 4 additions & 2 deletions docs/ci/runtime_intelligence_gitlab_artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,13 +186,15 @@ EdgeEnv handoff summary and AIGuard deterministic evidence agree on
producer-lineage guard-alignment run IDs. This keeps the cross-repo marker
check file-based and does not make AIGuard a deployment decision owner.

The artifact gate is implemented by `scripts/check_runtime_intelligence_artifact_bundle.py`. It checks the generated Markdown / HTML report for the required Runtime Intelligence rows, including Lab ownership, EdgeEnv comparability, telemetry coverage-gap markers, Runtime replay duration scope with `source=entrypoint_requested_frames` traceability, Orchestrator operation feed context, compact queue/deadline/fallback operation markers with `max_total_queue_depth`, Orchestrator task event rollup, Lab EdgeEnv preservation context, Jetson/device-local preservation identity and detail labels, Orchestrator `operation_risk_summary` navigation context, AIGuard runtime operation anomalies, AIGuard `edgeenv_orchestrator_operation_risk_summary` evidence, AIGuard `edgeenv_orchestrator_task_event_rollup` evidence, remote dispatch starter event summary, `Remote fallback starter evidence`, `edgeenv_orchestrator_producer_lineage`, `runtime_history_seed_run_config_traceability`, `remote_execution_recovered_by_fallback`, and triggered deployment review rules.
The artifact gate is implemented by `scripts/check_runtime_intelligence_artifact_bundle.py`. It checks the generated Markdown / HTML report for the required Runtime Intelligence rows, including Lab ownership, EdgeEnv comparability, telemetry coverage-gap markers, Runtime replay duration scope with `source=entrypoint_requested_frames` traceability, Orchestrator operation feed context, compact queue/deadline/fallback operation markers with `max_total_queue_depth`, AIGuard max queue raw-context traceability, Orchestrator task event rollup, Lab EdgeEnv preservation context, Jetson/device-local preservation identity and detail labels, Orchestrator `operation_risk_summary` navigation context, AIGuard runtime operation anomalies, AIGuard `edgeenv_orchestrator_operation_risk_summary` evidence, AIGuard `edgeenv_orchestrator_task_event_rollup` evidence, remote dispatch starter event summary, `Remote fallback starter evidence`, `edgeenv_orchestrator_producer_lineage`, `runtime_history_seed_run_config_traceability`, `remote_execution_recovered_by_fallback`, and triggered deployment review rules.

The bundle manifest gate also checks the external AIGuard artifact before the
rendered report stage. In particular, `runtime_queue_overload` must preserve
`orchestrator_candidate_operation_max_total_queue_depth=7` in
`raw_context.edgeenv_regression`, so the Lab `max_total_queue_depth` report row
can be traced back to Orchestrator producer-side operation context.
can be traced back to Orchestrator producer-side operation context. The report
now surfaces that check as `AIGuard max queue raw-context traceability`, keeping
the reviewer-facing row easy to identify in Markdown/HTML artifacts.
When that report gate passes, its summary now emits a
`Validated Duration Traceability` section with `duration_handoff_alignment`,
`duration_source: source=entrypoint_requested_frames`,
Expand Down
8 changes: 8 additions & 0 deletions docs/portfolio/edgeenv_runtime_regression_lab_handoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,10 @@ Expected Lab behavior:
`orchestrator_candidate_operation_max_total_queue_depth=7`, keeping the
rendered Lab `max_total_queue_depth` row traceable to Orchestrator's
producer-side operation context.
- Lab now renders that link as `AIGuard max queue raw-context traceability`,
with the visible report value and the AIGuard raw-context value side by side
so reviewers can identify the preservation path without treating it as a
deployment decision override.
- The same gate summary emits `expected_report_markers: remote fallback Lab context row declared`, keeping the remote fallback row-value marker visible in file-based CI artifacts without making CI a runtime control plane.
- The same handoff gate verifies that the referenced `runtime_telemetry_history` artifact exists and preserves EdgeEnv history schema, telemetry coverage, and Runtime history seed ownership markers.
- The same handoff gate verifies that missing telemetry entries remain evidence gaps while preserving Orchestrator producer markers, owner boundary flags, and EdgeEnv mapping hints when Orchestrator context is attached.
Expand All @@ -194,6 +198,10 @@ Expected Lab behavior:
- Markdown/HTML reports include a `Runtime Intelligence Risk Summary` that summarizes EdgeEnv comparability/regression, telemetry replay gaps, Runtime history seed/run_config traceability, AIGuard deterministic evidence, and the Lab-owned deployment decision in one reviewer-facing table.
- When EdgeEnv includes preserved Orchestrator feed context, the `Runtime Intelligence Risk Summary` surfaces queue, thermal, throttling, memory, fallback context, and compact `operation_risk_summary` markers as supplemental runtime evidence.
- Lab now keeps queue pressure, `max_total_queue_depth` when available, deadline miss count, and fallback count together in an `Orchestrator queue/deadline/fallback markers` row when those compact counters are present. This is reviewer navigation context, not a production scheduler state or deployment decision override.
- Lab also surfaces `AIGuard max queue raw-context traceability` when AIGuard
preserves the same `orchestrator_candidate_operation_max_total_queue_depth`
value, making the report row traceable back to deterministic AIGuard raw
context and Orchestrator operation evidence.
- When EdgeEnv includes replay duration metadata, the same Risk Summary surfaces `Runtime replay duration scope` with `duration_label`, `duration_class`, frame count, and optional `duration_source` / `duration_scope_label` traceability as reviewer navigation context. This does not change EdgeEnv comparability or Lab deployment policy.
- The report artifact gate summary also emits a `Validated Duration Traceability`
section so reviewers can see `duration_handoff_alignment`,
Expand Down
151 changes: 151 additions & 0 deletions inferedgelab/report/runtime_intelligence.py
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,12 @@ def build_runtime_intelligence_risk_rows(
warning_items,
evidence_items,
)
_append_aiguard_max_queue_traceability_row(
rows,
edgeenv_regression,
evidence_items,
guard_analysis,
)
_append_aiguard_remote_dispatch_rows(rows, guard_analysis, evidence_items)
_append_aiguard_run_config_traceability_row(rows, evidence_items)

Expand Down Expand Up @@ -915,6 +921,151 @@ def _append_aiguard_runtime_operation_rows(
)


def _append_aiguard_max_queue_traceability_row(
rows: list[tuple[str, str, str]],
edgeenv_regression: dict[str, Any] | None,
evidence_items: list[dict[str, Any]],
guard_analysis: dict[str, Any],
) -> None:
label = _aiguard_max_queue_traceability_label(
edgeenv_regression,
evidence_items,
guard_analysis,
)
if not label:
return

rows.append(
(
"AIGuard max queue raw-context traceability",
label,
(
"The visible Lab max_total_queue_depth marker is traceable "
"through AIGuard raw context to Orchestrator operation evidence; "
"it remains review context, not a decision override."
),
)
)


def _aiguard_max_queue_traceability_label(
edgeenv_regression: dict[str, Any] | None,
evidence_items: list[dict[str, Any]],
guard_analysis: dict[str, Any],
) -> str:
report_values: dict[str, Any] = {}
if isinstance(edgeenv_regression, dict):
telemetry_context = edgeenv_regression.get("runtime_telemetry_context")
if isinstance(telemetry_context, dict):
report_values = _orchestrator_max_queue_depth_by_run(telemetry_context)

raw_values = _aiguard_max_queue_raw_context_by_run(
evidence_items,
guard_analysis,
)
if not report_values or not raw_values:
return ""

labels: list[str] = []
for run_label in ("baseline", "candidate"):
report_value = report_values.get(run_label)
raw_value = raw_values.get(run_label)
if report_value is None or raw_value is None:
continue
labels.append(
(
f"{run_label}: report=max_total_queue_depth="
f"{_format_compact_value(report_value)}, "
f"raw_context=orchestrator_{run_label}_operation_"
"max_total_queue_depth="
f"{_format_compact_value(raw_value)}, "
f"match={_compact_values_match(report_value, raw_value)}"
)
)
return "; ".join(labels)


def _orchestrator_max_queue_depth_by_run(
context: dict[str, Any],
) -> dict[str, Any]:
values: dict[str, Any] = {}
for run_label in ("baseline", "candidate"):
run_context = context.get(run_label)
if not isinstance(run_context, dict):
continue
operation_context = run_context.get("orchestrator_operation_context")
if not isinstance(operation_context, dict):
continue

candidate_context = operation_context.get("candidate_context")
if not isinstance(candidate_context, dict):
candidate_context = {}
operation = candidate_context.get("operation")
if not isinstance(operation, dict):
operation = {}
operation_summary = operation_context.get("operation_risk_summary")
if not isinstance(operation_summary, dict):
operation_summary = {}
queue_state = operation_context.get("queue_state_summary")
if not isinstance(queue_state, dict):
queue_state = {}

value = _first_present(
operation_summary.get("max_total_queue_depth"),
queue_state.get("max_total_queue_depth"),
operation.get("max_total_queue_depth"),
candidate_context.get("max_total_queue_depth"),
)
if value is not None:
values[run_label] = value
return values


def _aiguard_max_queue_raw_context_by_run(
evidence_items: list[dict[str, Any]],
guard_analysis: dict[str, Any],
) -> dict[str, Any]:
values: dict[str, Any] = {}
for edgeenv_metrics in _aiguard_edgeenv_metric_sources(
evidence_items,
guard_analysis,
):
for run_label in ("baseline", "candidate"):
field = f"orchestrator_{run_label}_operation_max_total_queue_depth"
value = edgeenv_metrics.get(field)
if value is not None and run_label not in values:
values[run_label] = value
return values


def _aiguard_edgeenv_metric_sources(
evidence_items: list[dict[str, Any]],
guard_analysis: dict[str, Any],
) -> list[dict[str, Any]]:
sources: list[dict[str, Any]] = []
for item in evidence_items:
raw_context = item.get("raw_context")
if not isinstance(raw_context, dict):
continue
edgeenv_metrics = raw_context.get("edgeenv_regression")
if isinstance(edgeenv_metrics, dict):
sources.append(edgeenv_metrics)

candidate_summary = guard_analysis.get("candidate_summary")
if isinstance(candidate_summary, dict):
edgeenv_metrics = candidate_summary.get("edgeenv_regression")
if isinstance(edgeenv_metrics, dict):
sources.append(edgeenv_metrics)
return sources


def _compact_values_match(left: Any, right: Any) -> bool:
try:
return float(left) == float(right)
except (TypeError, ValueError):
return str(left) == str(right)


def _append_aiguard_run_config_traceability_row(
rows: list[tuple[str, str, str]],
evidence_items: list[dict[str, Any]],
Expand Down
12 changes: 12 additions & 0 deletions scripts/check_runtime_intelligence_artifact_bundle.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,12 @@
"aiguard_producer_lineage_handoff": (
"| AIGuard producer lineage handoff | sources=device_local_cli_override"
),
"aiguard_max_queue_traceability": (
"| AIGuard max queue raw-context traceability | candidate: "
"report=max_total_queue_depth=7, "
"raw_context=orchestrator_candidate_operation_max_total_queue_depth=7, "
"match=True |"
),
"aiguard_producer_lineage_evidence": "edgeenv_orchestrator_producer_lineage",
"aiguard_producer_lineage_recommendation": (
"Device-local Orchestrator producer lineage is preserved"
Expand Down Expand Up @@ -175,6 +181,12 @@
"lab_edgeenv_registry_marker": "lab_preservation=present",
"orchestrator_operation_risk_queue": "queue=queue_backlog_threshold_exceeded",
"aiguard_producer_lineage_handoff": "AIGuard producer lineage handoff",
"aiguard_max_queue_traceability": (
"AIGuard max queue raw-context traceability"
),
"aiguard_max_queue_traceability_value": (
"raw_context=orchestrator_candidate_operation_max_total_queue_depth=7"
),
"aiguard_producer_lineage_evidence": "edgeenv_orchestrator_producer_lineage",
"aiguard_producer_lineage_recommendation": (
"Device-local Orchestrator producer lineage is preserved"
Expand Down
3 changes: 3 additions & 0 deletions scripts/check_runtime_intelligence_ci_artifacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
"aiguard_raw_context: downstream_guard_alignment preserved",
"aiguard_raw_context: producer_lineage_guard_alignment preserved",
"aiguard_raw_context: missing_telemetry_orchestrator_context preserved",
"aiguard_raw_context: max_total_queue_depth traceability preserved",
"aiguard_handoff_alignment: external required evidence types satisfied",
"expected_report_markers: Runtime Intelligence report markers declared",
"expected_report_markers: remote fallback Lab context row declared",
Expand Down Expand Up @@ -184,6 +185,8 @@ def _validate_runtime_report(path: Path, errors: list[str]) -> None:
"Orchestrator queue/deadline/fallback markers",
"queue_pressure_reason=queue_backlog_threshold_exceeded",
"max_total_queue_depth=7",
"AIGuard max queue raw-context traceability",
"raw_context=orchestrator_candidate_operation_max_total_queue_depth=7",
"AIGuard producer-lineage guard alignment",
"edgeenv_orchestrator_producer_lineage",
"AIGuard run_config traceability evidence",
Expand Down
13 changes: 13 additions & 0 deletions tests/test_compare_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,11 @@ def make_edgeenv_orchestrator_context_guard_analysis() -> dict:
"why_it_matters": "Queue backlog can inflate runtime latency.",
"suspected_causes": ["queue_overload", "scheduler_contention"],
"recommendation": "Review Orchestrator queue policy.",
"raw_context": {
"edgeenv_regression": {
"orchestrator_candidate_operation_max_total_queue_depth": 7.0,
},
},
},
],
"suspected_causes": ["thermal_pressure", "queue_overload"],
Expand All @@ -563,6 +568,7 @@ def make_edgeenv_orchestrator_context_guard_analysis() -> dict:
"history_orchestrator_feed_runs": 1.0,
"candidate_orchestrator_context_present": True,
"candidate_queue_depth": 7.0,
"orchestrator_candidate_operation_max_total_queue_depth": 7.0,
"candidate_max_temperature_c": 78.5,
"candidate_throttling_detected": True,
}
Expand Down Expand Up @@ -810,6 +816,12 @@ def fake_analyze_edgeenv_regression_report(report):
"| Orchestrator queue/deadline/fallback markers | candidate: "
"max_total_queue_depth=7, deadline_missed_count=2, fallback_count=1 |"
) in bundle["markdown"]
assert (
"| AIGuard max queue raw-context traceability | candidate: "
"report=max_total_queue_depth=7, "
"raw_context=orchestrator_candidate_operation_max_total_queue_depth=7, "
"match=True |"
) in bundle["markdown"]
assert "| Orchestrator task event rollup | candidate: " in bundle["markdown"]
assert "vision_agent(delay=1,miss=1,max_delay_cycles=3,max_wait_ms=15)" in bundle[
"markdown"
Expand All @@ -824,6 +836,7 @@ def fake_analyze_edgeenv_regression_report(report):
]
assert "AIGuard does not own the final decision" in bundle["markdown"]
assert "AIGuard Orchestrator context handoff" in bundle["html"]
assert "AIGuard max queue raw-context traceability" in bundle["html"]
assert "supplemental operation evidence" in bundle["html"]


Expand Down
Loading
Loading