Webhook Improvements - Templateable Payloads

[JacobsonMT]

Overview

This PR introduces enhancements to the webhook notifier, focusing on three main areas:

1. Customizable webhook payloads using templates
2. Support for custom HTTP headers
3. New template functions to provide support for creating well-formatted, valid JSON in payloads.

Key Changes

1. Custom Payload Templates

• Added support for custom JSON payload templates in webhook notifications
• Users can now define their own payload structure using templates
• Includes a new CustomPayload field with:
  • Template field for defining the payload structure
  • Vars field for providing additional template variables accessible in context via .Vars.<variable_name>
• Provides an example payload template which is equivalent to the default payload format for demonstration and regression testing

2. Custom HTTP Headers

• Added support for custom HTTP headers in webhook requests, example usage would be to set the Content-Type header for your custom payload.
• Implemented security measures to prevent misuse:
  • Restricted headers list to prevent security issues
  • Headers like Authorization, Cookie, Host are protected

3. New Template Functions

• Introduced new template functions inspired by gomplate:
  • coll namespace for collection operations (Dict, Slice, Append)
  • data namespace for JSON operations (JSON, ToJSON, ToJSONPretty)
  • tmpl namespace for template operations compatible with pipelines (Inline, Exec)
  • time namespace for time operations (Now)
• These functions enable more complex and flexible payload formatting, especially when creating JSON.
• Currently gomplate does not allow modularized use of select functions or namespaces, so importing it directly would require a significant increase in dependencies. In addition, certain necessary functions have extended functionality in gomplate that makes them risky to use as-is such as the eJSON decryption capability of data.JSON which reads from environment variables. That being said, none of this is insurmountable, and the plan should be to eventually incorporate the gomplate library directly and more broadly.

4. Related Improvements

• Added new fields to ExtendedData and ExtendedAlert to make sure custom payloads have access to all of the information the default webhook payload has:
  • ExtendedAlert.OrgId
  • ExtendedData.GroupKey
  • ExtendedData.TruncatedAlerts

Related Requests

Issues for supporting new notifiers in Grafana are fairly common and are generally reasonable requests. However, they can come with significant burden of maintenance. Some examples:

• Telegram Local Bot API support
• Alerting: Extend Pagerduty notification for Openduty
• Feature request: New contact point type: Amazon EventBridge
• FR: Alerting Contact Point Matrix
• Add web push notifications to Grafana Alerting

Requests for related features are some of the most popular in Alerting:

• [Feature request] Custom json passed as a webhook alert
• Support Backend plugins for Alert notifiers
• Custom Headers on HTTP webhook
• Support custom headers on the webhook contact point
• Webhook Receiver should support custom headers

Upstream Prometheus Alertmanager also has significant community requests, although consensus so far has been to keep the webhook notifier as-is with recommendations to set up a small web server to act as a webhook bridge:

• it's more useful for webhook_config support template and custom the header
• Added functionality to alter Webhook message
• Reintroduce AWS SNS notifications
• Add templated text support for webhook receivers
• Templating support for body of webhook_config
• webhook custom style

Future Work

• Import gomplate and expand template support to more namespaces.

Custom Payload Examples

Could be as simple as writing JSON directly:

{
  "alert_name": "{{ .CommonLabels.alertname }}",
  "status": "{{ .Status }}",
  "environment": "{{ .Vars.environment }}",
  "custom_field": "{{ .Vars.custom_field }}"
}

Or using new functions to help craft valid JSON:

  {{ coll.Dict 
  "receiver" .Receiver
  "status" .Status
  "alerts" .Alerts
  "groupLabels" .GroupLabels
  "commonLabels" .CommonLabels
  "commonAnnotations" .CommonAnnotations
  "externalURL" .ExternalURL
  "version" "1"
  "orgId"  (index .Alerts 0).OrgId
  "truncatedAlerts"  .TruncatedAlerts
  "groupKey" .GroupKey
  "state"  (tmpl.Inline "{{ if eq .Status \"resolved\" }}ok{{ else }}alerting{{ end }}" . )
  "title" (tmpl.Exec "default.title" . )
  "message" (tmpl.Exec "default.message" . )
  | data.ToJSONPretty " "}}

Review Tips

• New gomplate-based functions are nearly 1:1 with the library, so testing was skipped given we're going to eventually remove these in favour of and import. Any differences are pointed out in comments.
• Grafana-side PR will be easier to test with as it will include UI support and enhancements to the template builder to help when creating JSON.

[moustafab]

Looks great! Nice work!

My only thoughts on the implementation are around further improvements in the supported functions. If we anticipate significant churn here it may be good to establish an experimental namespace which we document to allow for a bit of safer additions. I'm hesitant to support to much without clear needs and I think this approach you've taken so far is far preferable to pulling in an entire library until clear need is well documented/established.

Also, templating header values might come in handy in the future but maybe we can wait for a request from the community before adding that.

[titolins]

weekly board maintenance ✅ 😂
great work 👏