Outgoing webhook sender by rbnswartz · Pull Request #186 · WycliffeAssociates/ScriptureRenderingPipeline

rbnswartz · 2026-02-03T15:41:06Z

This pull request introduces a robust outgoing webhook infrastructure for the pipeline, enabling registration, deletion, listing, and resilient dispatching of webhook notifications. The main changes include implementing Azure Table Storage-backed webhook management, providing HTTP endpoints for webhook registration and deletion, and adding a resilient dispatcher for outgoing webhook messages.

Webhook Infrastructure Implementation

Added IWebhookService interface and AzureStorageWebhookStorage implementation for managing webhook registration, deletion, existence checks, and listing, backed by Azure Table Storage (PipelineCommon/IWebhookService.cs, PipelineCommon/Helpers/AzureStorageWebhookStorage.cs). [1] [2]
Introduced WebhookDefinition and response models for registration and deletion operations (PipelineCommon/Models/WebhookDefinition.cs, PipelineCommon/Models/Webhook/WebhookResponses.cs). [1] [2]

HTTP Endpoint Integration

Implemented Azure Functions HTTP endpoints for webhook registration (RegisterWebhook) and deletion (UnregisterWebhook), including validation and error handling (ScriptureRenderingPipeline/OutgoingWebhook.cs).

Webhook Dispatching and Resilience

Added WebhookDispatcher class for sending messages to registered webhooks, using Polly-based retry and resilience strategies to handle transient failures (ScriptureRenderingPipelineWorker/WebhookDispatcher.cs).
Registered webhook services and dispatcher in worker and pipeline startup, and configured Azure Table client for webhook storage (ScriptureRenderingPipeline/Program.cs, ScriptureRenderingPipelineWorker/Program.cs). [1] [2]

Dependency and Project Configuration

Added necessary NuGet packages for Azure Table Storage and resilience, and updated project files to support new functionality (PipelineCommon/PipelineCommon.csproj, ScriptureRenderingPipeline/ScriptureRenderingPipeline.csproj, ScriptureRenderingPipelineWorker/ScriptureRenderingPipelineWorker.csproj). [1] [2] [3]- WIP

Copilot

Pull request overview

Adds an outgoing webhook system so pipeline events (e.g., WACS events and repo analysis results) can be delivered to registered external endpoints.

Changes:

Added HTTP endpoints to register/unregister outgoing webhooks backed by Azure Table Storage.
Added worker Service Bus triggers + dispatcher to POST matching messages to registered webhook URLs.
Introduced shared webhook models/storage service in PipelineCommon, plus resilience/retry support for dispatching.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
ScriptureRenderingPipelineWorker/WebhookDispatcherTrigger.cs	New Service Bus triggers to invoke webhook dispatch for WACS + repo analysis result messages.
ScriptureRenderingPipelineWorker/WebhookDispatcher.cs	New dispatcher that filters registered webhooks and POSTs payloads with retry/resilience.
ScriptureRenderingPipelineWorker/ScriptureRenderingPipelineWorker.csproj	Adds dependency needed for the resilience pipeline.
ScriptureRenderingPipelineWorker/Program.cs	Registers Table client, HttpClient, and webhook services for the worker.
ScriptureRenderingPipeline/ScriptureRenderingPipeline.csproj	Adds Azure Tables package for webhook storage support.
ScriptureRenderingPipeline/Program.cs	Registers Table client and webhook storage service for the API app.
ScriptureRenderingPipeline/OutgoingWebhook.cs	New HTTP functions to register and unregister webhook definitions.
PipelineCommon/PipelineCommon.csproj	Adds Azure Tables + Azure client factory dependencies for shared webhook storage implementation.
PipelineCommon/Models/WebhookDefinition.cs	New model representing a webhook subscription (URL + message/event type).
PipelineCommon/Models/Webhook/WebhookResponses.cs	New response DTOs for webhook registration/deletion endpoints.
PipelineCommon/IWebhookService.cs	New interface for webhook registration, deletion, existence checks, and listing.
PipelineCommon/Helpers/AzureStorageWebhookStorage.cs	New Azure Table Storage implementation of `IWebhookService`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-03T15:52:50Z

+    [Function("RegisterWebhook")]
+    public async Task<HttpResponseData> Register([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestData req)
+    {
+        try
+        {
+            var requestBody = await new StreamReader(req.Body).ReadToEndAsync();
+            var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);
+
+            if (webhookDefinition == null || string.IsNullOrWhiteSpace(webhookDefinition.Url))
+            {
+                var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);
+                await badResponse.WriteStringAsync("Invalid webhook definition: URL is required");


RegisterWebhook is AuthorizationLevel.Anonymous and persists arbitrary URLs provided by the caller. This enables unauthenticated modification of webhook configuration and can create an SSRF vector when the worker later POSTs to those URLs. Consider requiring at least Function auth (or other authn/z), and/or validating/allow-listing destinations (e.g., allowed hostnames, require HTTPS).

Suggested change

[Function("RegisterWebhook")]

public async Task<HttpResponseData> Register([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestData req)

{

try

{

var requestBody = await new StreamReader(req.Body).ReadToEndAsync();

var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);

if (webhookDefinition == null || string.IsNullOrWhiteSpace(webhookDefinition.Url))

{

var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);

await badResponse.WriteStringAsync("Invalid webhook definition: URL is required");

private static bool IsValidWebhookUrl(string url)

{

if (string.IsNullOrWhiteSpace(url))

{

return false;

}

if (!Uri.TryCreate(url, UriKind.Absolute, out var uri))

{

return false;

}

// Only allow HTTPS webhooks to reduce risk of SSRF and ensure transport security.

return string.Equals(uri.Scheme, Uri.UriSchemeHttps, StringComparison.OrdinalIgnoreCase);

}

[Function("RegisterWebhook")]

public async Task<HttpResponseData> Register([HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req)

{

try

{

var requestBody = await new StreamReader(req.Body).ReadToEndAsync();

var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);

if (webhookDefinition == null || !IsValidWebhookUrl(webhookDefinition.Url))

{

var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);

await badResponse.WriteStringAsync("Invalid webhook definition: a valid HTTPS URL is required");

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T14:58:12Z

+    [Function("RegisterWebhook")]
+    public async Task<HttpResponseData> Register([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestData req)
+    {
+        try
+        {
+            using var reader = new StreamReader(req.Body);
+            var requestBody = await reader.ReadToEndAsync();
+            var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);
+
+            if (webhookDefinition == null || string.IsNullOrWhiteSpace(webhookDefinition.Url))
+            {
+                var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);
+                await badResponse.WriteStringAsync("Invalid webhook definition: URL is required");
+                return badResponse;
+            }


This endpoint is AuthorizationLevel.Anonymous and accepts an arbitrary Url that will later be POSTed to by the worker. That’s a direct SSRF/abuse vector (e.g., registering internal IPs/metadata endpoints) and also allows anyone to register/delete webhooks. Require authentication/authorization and add strict URL validation (https-only, host allowlist, block private/link-local IP ranges, etc.).

Copilot · 2026-02-04T14:58:12Z

+            using var reader = new StreamReader(req.Body);
+            var requestBody = await reader.ReadToEndAsync();
+            var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);
+


JsonSerializer.Deserialize<WebhookDefinition>(requestBody) uses default options (case-sensitive by default). Clients sending conventional camelCase JSON (url, messageType, eventType) will fail to bind, and MessageType may silently fall back to the model default. Use a configured JsonSerializerOptions (e.g., case-insensitive) or add WebhookDefinition to a source-generated JsonSerializerContext and deserialize with that for consistent API behavior.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T15:06:30Z

+    public async Task DispatchGenericMessageAsync<T>(string messageType, string eventType, T message)
+    {
+        try
+        {
+            var webhooks = await _webhookService.GetWebhooksAsync();
+
+            // Filter webhooks matching the event type
+            var matchingWebhooks = webhooks
+                .Where(w => string.Equals(w.EventType, eventType, StringComparison.OrdinalIgnoreCase) &&
+                            string.Equals(w.MessageType, messageType, StringComparison.OrdinalIgnoreCase))
+                .ToList();
+


New webhook dispatch logic is core behavior and would benefit from automated tests (e.g., filtering by MessageType+EventType, retry behavior, and that failed webhook calls don’t block others). There are existing NUnit tests under SRPTests; consider adding unit tests for this dispatcher and webhook storage integration points.

Copilot · 2026-02-04T15:06:31Z

+    [Function("RegisterWebhook")]
+    public async Task<HttpResponseData> Register([HttpTrigger(AuthorizationLevel.Anonymous, "post")] HttpRequestData req)
+    {
+        try
+        {
+            using var reader = new StreamReader(req.Body);
+            var requestBody = await reader.ReadToEndAsync();
+            var webhookDefinition = JsonSerializer.Deserialize<WebhookDefinition>(requestBody);
+
+            if (webhookDefinition == null || string.IsNullOrWhiteSpace(webhookDefinition.Url))
+            {
+                var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);
+                await badResponse.WriteStringAsync("Invalid webhook definition: URL is required");
+                return badResponse;
+            }
+
+            if (!AllowedMessageTypes.Any(i => i == webhookDefinition.MessageType))
+            {
+                var badResponse = req.CreateResponse(HttpStatusCode.BadRequest);
+                await badResponse.WriteStringAsync($"Invalid webhook definition: MessageType '{webhookDefinition.MessageType}' is not allowed");
+                return badResponse;
+            }


This endpoint is AuthorizationLevel.Anonymous and allows registering arbitrary webhook URLs. That enables SSRF and data exfiltration (the worker will POST internal data to attacker-controlled URLs). Require authentication (function key/AAD) and add URL validation/allowlisting (e.g., only https, block private IP ranges, optional domain allowlist) before persisting the webhook.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rbnswartz · 2026-02-04T16:56:01Z

@PurpleGuitar What do you think about the needing or not needing a secret function key to create and or delete webhooks?

rbnswartz · 2026-03-23T21:04:17Z

Ended up making the register webhook require a function key. We can remove that later if we want.

rbnswartz added 6 commits January 26, 2026 16:48

WIP

58f936a

Finishing up

6b27004

Fixing some of the naming of topics

05b24fc

Added Message Type to webhook filters

4cab123

Fixed some issues with webhook creation

9b57e6b

Fixed one last issue

068386b

rbnswartz requested review from PurpleGuitar and Copilot February 3, 2026 15:41

rbnswartz changed the title ~~rs/webhook sender~~ Outgoing webhook sender Feb 3, 2026

Copilot started reviewing on behalf of rbnswartz February 3, 2026 15:42 View session

Copilot AI reviewed Feb 3, 2026

View reviewed changes

PurpleGuitar requested changes Feb 3, 2026

View reviewed changes

Comment thread ScriptureRenderingPipelineWorker/WebhookDispatcher.cs Outdated

Comment thread ScriptureRenderingPipelineWorker/WebhookDispatcher.cs Outdated

Apply suggestions from code review

5f1f29d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rbnswartz requested a review from Copilot February 4, 2026 14:50

Copilot started reviewing on behalf of rbnswartz February 4, 2026 14:50 View session

Copilot AI reviewed Feb 4, 2026

View reviewed changes

rbnswartz requested a review from Copilot February 4, 2026 14:58

Copilot started reviewing on behalf of rbnswartz February 4, 2026 14:58 View session

Copilot AI reviewed Feb 4, 2026

View reviewed changes

rbnswartz and others added 7 commits February 4, 2026 10:39

More changes from review

3fe88b0

Fixed a potential issue with reusing http request message

1b16e8f

Fixed an issue with Webhook exists

4485d4c

Lets do case insensitive in webhook validation

5453412

Fixed a comment

a6a7e13

Use string.empty for message type

10e6f6d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Separate logic for validation out

b0409af

rbnswartz added 2 commits March 19, 2026 10:34

Added topics that we didn't have

0b70529

Use Parallel for each because it should scale better

43d9a71

PurpleGuitar approved these changes Mar 20, 2026

View reviewed changes

rbnswartz added 2 commits March 23, 2026 16:55

Require function key for registering webhook

1cc2b28

Merge branch 'develop' into rs/webhook-sender

07d0f24

rbnswartz merged commit 13aef04 into develop Mar 23, 2026
6 checks passed

Conversation

rbnswartz commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rbnswartz commented Feb 4, 2026

Uh oh!

Uh oh!

rbnswartz commented Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rbnswartz commented Feb 3, 2026 •

edited

Loading