Skip to content

feat(logging): add PII scrubbing middleware with custom Winston trans…#242

Open
habnark wants to merge 1 commit into
StellarFlow-Network:mainfrom
habnark:main
Open

feat(logging): add PII scrubbing middleware with custom Winston trans…#242
habnark wants to merge 1 commit into
StellarFlow-Network:mainfrom
habnark:main

Conversation

@habnark
Copy link
Copy Markdown

@habnark habnark commented Apr 27, 2026

Closes #210


Log Redaction & PII Scrubbing Middleware — Full Description

Problem Being Solved

The existing logMasker.ts only patched console.* methods. The Winston logger itself — which writes to rotating log files (external storage) — had no scrubbing at all. Any secret, private IP, JWT, or webhook URL passed to logger.info() / logger.error() was written verbatim to disk and any downstream log aggregator. This is a data exposure risk for admin credentials, internal network topology, and third-party API keys.


Architecture: Two Independent Scrubbing Layers

Application code calls logger.info("...", meta)
        │
        ▼
┌─────────────────────────────────────────────────┐
│  Layer 1: redactFormat  (Winston format)        │
│  scrubLogInfo() runs on every entry             │
│  before any transport sees it                   │
└──────────────────┬──────────────────────────────┘
                   │
        ┌──────────┴──────────┐
        ▼                     ▼
┌───────────────┐    ┌──────────────────────────────┐
│ Console       │    │ Layer 2: RedactingTransport  │
│ transport     │    │ scrubLogInfo() runs again     │
│ (+ console.*  │    │ (defense-in-depth)            │
│ monkey-patch  │    │         │                    │
│ from index.ts)│    │         ▼                    │
└───────────────┘    │  DailyRotateFile             │
                     │  → logs/application-DATE.log │
                     └──────────────────────────────┘

File 1: src/utils/logMasker.ts (expanded)

New regex patterns added:

Pattern | What it catches | Replacement -- | -- | -- 10.x.x.x | RFC 1918 class-A private IPs | [INTERNAL_IP] 172.16-31.x.x | RFC 1918 class-B private IPs | [INTERNAL_IP] 192.168.x.x | RFC 1918 class-C private IPs | [INTERNAL_IP] 127.x.x.x | Loopback addresses | [INTERNAL_IP] 169.254.x.x | Link-local addresses | [INTERNAL_IP] ::1 | IPv6 loopback | [INTERNAL_IP] fd00::/8 | IPv6 Unique Local Addresses | [INTERNAL_IP] eyXXX.XXX.XXX | JWT tokens (3 base64url segments) | [REDACTED] discord.com/api/webhooks/… | Discord webhook URLs | [REDACTED_URL] hooks.slack.com/services/… | Slack webhook URLs | [REDACTED_URL] api_key=xxxxx | Generic key=value assignments | [REDACTED] Object key admin, cluster, node_ip, server_ip | Admin/internal fields in objects | [REDACTED]

Fixed bugs in the existing code:

  • All global regexes now reset lastIndex before each replace() call — previously, stateful regexes silently skipped every second match when called on different strings
  • Stellar secret key pattern tightened from 48–56 chars to exactly 55 trailing chars (total 56)

New export — scrubLogInfo(info):

  • Takes a raw Winston log info object
  • Scrubs message and all string/object metadata fields
  • Leaves level, timestamp, and all Symbol-keyed Winston internals (Symbol(level), Symbol(splat)) completely untouched — altering these breaks Winston's internal pipeline

File 2: src/utils/redactingTransport.ts (new file)

A custom Winston Transport class — the "transport layer for scrubbing" the spec requires.

RedactingTransport extends Transport (winston-transport)
  │
  ├─ constructor({ inner, label, ...opts })
  │    inner = the real transport to delegate to (DailyRotateFile)
  │
  ├─ log(info, callback)
  │    1. scrubLogInfo(info)       ← PII scrub
  │    2. inner.log(scrubbed, cb)  ← delegate to file transport
  │    3. emit('logged', scrubbed) ← notify Winston backpressure system
  │    4. callback()               ← signal ready for next entry
  │
  └─ close()
       propagates close() to the inner transport so file handles
       are released cleanly on SIGINT/SIGTERM shutdown

Key design decisions:

  • Does not subclass DailyRotateFile — stays generic so any transport (HTTP, S3, Datadog, etc.) can be wrapped by passing it as inner
  • Handles the case where inner.log doesn't exist (graceful fallback)
  • Emits 'logged' after the inner transport confirms the write — correct backpressure handling so Winston doesn't fill its internal buffer under high log volume

File 3: src/utils/winstonLogger.ts (updated)

Changes:

  • Added redactFormat — a format(info => scrubLogInfo(info))() custom Winston format applied at the logger level, meaning it runs before the entry reaches either transport
  • DailyRotateFile is now instantiated privately and wrapped in new RedactingTransport({ inner: dailyRotateFileTransport }) — callers only ever interact with the scrubbing wrapper
  • LOG_LEVEL environment variable now respected (process.env.LOG_LEVEL ?? 'info') so log verbosity can be adjusted per environment without code changes
  • fetcherError convenience method now has correct TypeScript typing (was previously typed as any)
  • File and console transports retain their own handleExceptions: true / handleRejections: true flags so unhandled errors and promise rejections continue to be captured

…port

Implement a two-layer log redaction system to ensure sensitive data is
stripped from all log output before reaching external storage or the
console.

Layer 1 — redactFormat (logMasker.ts + winstonLogger.ts):
  A custom Winston format applied globally at the logger level so every
  entry is scrubbed before any transport receives it.

Layer 2 — RedactingTransport (redactingTransport.ts):
  A new custom Winston Transport class that wraps DailyRotateFile (the
  external-storage transport). It applies scrubLogInfo() a second time
  as a defense-in-depth measure, catching any data injected after format
  processing.

New patterns added to logMasker.ts:
  - Private / internal IPv4 ranges (RFC 1918: 10.x, 172.16-31.x,
    192.168.x; loopback: 127.x; link-local: 169.254.x) → [INTERNAL_IP]
  - IPv6 loopback (::1) and ULA (fd00::/8) → [INTERNAL_IP]
  - JWT tokens (three base64url segments) → [REDACTED]
  - Discord and Slack webhook URLs → [REDACTED_URL]
  - Generic key=value API key assignments → [REDACTED]
  - Object keys matching admin/cluster/node_ip/server_ip → [REDACTED]
  - Improved Stellar secret key pattern (exactly 56 chars)
  - Regex lastIndex reset so global patterns work on repeated calls

New exports from logMasker.ts:
  - scrubLogInfo(info): sanitises a Winston log info object, preserving
    level/timestamp and all Symbol-keyed Winston internals unchanged.

winstonLogger.ts:
  - Imports redactFormat and RedactingTransport
  - Wraps DailyRotateFile with RedactingTransport
  - Respects LOG_LEVEL env var (falls back to 'info')
  - Keeps fetcherError convenience method with proper TypeScript typing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@habnark habnark closed this Apr 27, 2026
@habnark habnark reopened this Apr 27, 2026
@drips-wave
Copy link
Copy Markdown

drips-wave Bot commented Apr 27, 2026

@habnark Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Logging | Log Redaction & PII Scrubbing Middleware

1 participant