Skip to content

plugins.mdx

Latest commit

 

History

History
123 lines (86 loc) · 4 KB

plugins.mdx

File metadata and controls

123 lines (86 loc) · 4 KB
title Plugin System
description Lifecycle hooks for metrics, audit trails, and custom processing.

Sentinel uses an opt-in plugin system where extensions subscribe to lifecycle events by implementing specific interfaces. The base interface requires only a Name() method — all hooks are optional.

Base interface

type Extension interface {
    Name() string
}

Lifecycle hooks

There are 16 hook interfaces organized by category. Extensions implement only the hooks they need.

Eval lifecycle

Interface Method When it fires
EvalRunStarted OnEvalRunStarted(ctx, suiteID, runID, model) Evaluation run begins
EvalRunCompleted OnEvalRunCompleted(ctx, suiteID, runID, passRate, elapsed) Run finishes successfully
EvalRunFailed OnEvalRunFailed(ctx, suiteID, runID, err) Run fails with error

Case lifecycle

Interface Method When it fires
CaseStarted OnCaseStarted(ctx, runID, caseID) Case evaluation begins
CaseCompleted OnCaseCompleted(ctx, runID, caseID, score, elapsed) Case evaluation finishes
CaseFailed OnCaseFailed(ctx, runID, caseID, err) Case evaluation fails

Regression

Interface Method When it fires
RegressionDetected OnRegressionDetected(ctx, suiteID, baselineID, delta) Performance regression found

Baseline

Interface Method When it fires
BaselineSaved OnBaselineSaved(ctx, suiteID, baselineID) Baseline saved

Red team

Interface Method When it fires
RedTeamStarted OnRedTeamStarted(ctx, suiteID, attackCount) Red team evaluation begins
RedTeamCompleted OnRedTeamCompleted(ctx, suiteID, bypassCount, elapsed) Red team evaluation finishes

Persona evaluation

Interface Method When it fires
PersonaEvalStarted OnPersonaEvalStarted(ctx, runID, personaName) Persona-aware evaluation begins
PersonaEvalCompleted OnPersonaEvalCompleted(ctx, runID, personaName, dimensions) Persona-aware evaluation finishes

Prompt versioning

Interface Method When it fires
PromptVersionCreated OnPromptVersionCreated(ctx, suiteID, pvID, version) Prompt version created

Comparison

Interface Method When it fires
ComparisonCompleted OnComparisonCompleted(ctx, suiteID, models, elapsed) Multi-model comparison finishes

Shutdown

Interface Method When it fires
Shutdown OnShutdown(ctx) Graceful shutdown

Registry

The plugin.Registry type-caches extensions at registration time:

registry := plugin.NewRegistry(logger)
registry.Register(metricsExt)
registry.Register(auditExt)

When events are emitted, only extensions implementing the relevant hook are called:

registry.EmitEvalRunStarted(ctx, suiteID, runID, model)
// Only calls extensions that implement EvalRunStarted

Error handling

Hook errors are logged but never propagated. Hooks must not block the evaluation pipeline.

Built-in extensions

  • observability.MetricsExtension — Counters for all lifecycle events
  • audithook.Extension — Bridges events to an audit trail backend

Writing a custom extension

type SlackNotifier struct {
    webhookURL string
}

func (s *SlackNotifier) Name() string { return "slack-notifier" }

func (s *SlackNotifier) OnRegressionDetected(ctx context.Context, suiteID id.SuiteID, baselineID id.BaselineID, delta float64) error {
    return postToSlack(s.webhookURL, fmt.Sprintf("Regression detected in suite %s: %.1f%% drop", suiteID, delta*100))
}

// Register:
eng, _ := engine.New(
    engine.WithExtension(&SlackNotifier{webhookURL: "https://hooks.slack.com/..."}),
)