Phase 2: Priority-based throttling, mutex guard, admin UI, and WP-CLI

[mrtwebdesign]

Context

Issue #3 defined a full implementation framework for priority-based concurrency throttling. PR #4 delivered the load detection and global throttle foundation as "Phase 1." This issue covers the remaining work needed to complete that vision.

What Phase 1 delivered (PR #4)

• Load probing (Threads_running + AS queue depth) with normal/elevated/critical classification
• Hysteresis with separate enter/exit thresholds and minimum dwell time
• Global AS queue-runner throttling (batch size, time limit, concurrent batches)
• Four-stage rollout (off → test_observe → observe → enforce)
• Cross-request state persistence (object cache → APCu → option fallback)
• Structured logging for all throttle events and load transitions

What Phase 1 cannot do

Distinguish between action types. Under load, all AS work slows equally — a NoFraud payment webhook gets the same treatment as a Facebook catalog sync. The May 1 incident needed the opposite: keep payments flowing, pause deferrable syncs.

---

Progressive implementation plan

Each wave can be deployed independently and provides value on its own. Later waves build on earlier ones but each is a shippable increment.

Wave A — Refactor & registry (no behavior change, safe to deploy anytime)

Structural prep work. No runtime behavior changes, so this can ship without a rollout plan.

• Extract Load Monitor to class-hcqg-load-monitor.php — refactor only, per #12. Moves existing probe and level-evaluation logic into its own class. Zero behavior change. #20
• Priority Registry (class-hcqg-priority-registry.php) — map AS hook names to priority tiers (critical/high/normal/deferrable) with wildcard/prefix matching and a filterable default registry. Ships inert until Wave B wires it into throttle decisions. #21

Wave B — Per-action throttling (core feature, needs rollout plan)

The primary deliverable. Requires Wave A. Should follow the same observe → enforce rollout pattern as Phase 1.

• Throttle Engine — per-action decisions — hook action_scheduler_before_execute to skip/defer individual actions based on the priority × load level decision matrix. Deferred actions are rescheduled, never deleted. Includes cooldown ramp after load normalizes. #22
• Per-action logging — extend existing structured logging to include each deferred action with hook name, priority tier, load level, and reschedule delay. #23

Wave C — Thundering herd prevention (independent, can ship in any order)

Completely standalone — no dependency on Waves A or B. Addresses a different failure mode (N concurrent requests triggering the same expensive operation).

• Mutex Guard (class-hcqg-mutex-guard.php) — lock mechanism so only one request runs an expensive operation at a time. acquire_lock() / release_lock() / force_release() API. Uses wp_options with autoload = no. #27

Wave D — Operator tooling (additive, ship as components land)

Each item here can ship as soon as the component it manages exists. No need to wait for everything.

• WP-CLI — throttle commands — wp queryguard throttle status/pause/resume/history. Can ship after Wave B.
• WP-CLI — mutex commands — wp queryguard mutex list/release. Can ship after Wave C.
• Admin Settings page — UI for enable/disable toggles, load thresholds, priority registry editor, reschedule delays, current load status indicator, and last 20 throttled actions. Can ship incrementally as sections become relevant.

---

Component details

Priority Registry (Wave A)

Map Action Scheduler hook names to priority tiers. Stored as a WordPress option, editable in admin (Wave D), with a filter for code-based overrides.

Default registry:

Tier	Hook patterns	Behavior under load
Critical	nofraud_*, woocommerce_payment_*, wc_payment_*	Always runs
High	woocommerce_scheduled_subscription_*, wcs_*, woocommerce_deliver_webhook_*	Delayed 5-10 min during critical load
Normal	woocommerce_run_*, action_scheduler_*	Deferred +15 min during critical, +5 min during elevated
Deferrable	facebook_for_woocommerce_*, wc_facebook_*, shipstation_*, klaviyo_*, woocommerce_flush_*	Paused until load normalizes

Public API: get_priority( $hook_name ) returns tier string. Supports prefix/wildcard matching. Unregistered hooks default to normal.

Throttle Engine — per-action decisions (Wave B)

Hook action_scheduler_before_execute to apply the priority × load decision matrix:

Load Level	Critical	High	Normal	Deferrable
Normal	Run	Run	Run	Run
Elevated	Run	Run	Defer +5 min	Defer +15 min
Critical	Run	Defer +5 min	Defer +15 min	Defer +60 min

Deferred actions are rescheduled, never deleted. After load returns to normal, a cooldown ramp gradually releases deferred actions over 2-3 cycles to prevent a burst.

This builds on Phase 1's existing load detection and batch-size reduction — Phase 1 slows the whole queue, Phase 2 adds selective filtering within it.

Mutex Guard (Wave C)

Prevents thundering herd patterns (e.g., 150 concurrent NoFraud admin_init scans).

API:

• acquire_lock( $operation_key, $ttl = 60 ) — returns true if acquired, false if held
• release_lock( $operation_key ) — early release
• is_locked( $operation_key ) — check without acquiring
• force_release( $operation_key ) — admin/CLI recovery

Storage: wp_options with autoload = no (not transients — transients may use object cache that isn't shared across PHP processes on some hosts). Lock expired when current_time > stored_timestamp + TTL.

Admin Settings (Wave D)

A settings page (or tab within an existing Hypercart admin page) with:

• Enable/disable toggles for throttle system and mutex guard
• Load threshold inputs (threads_running and queue depth for elevated/critical)
• Priority registry editor (hook patterns per tier)
• Reschedule delay configuration per load level
• Current load status indicator (normal/elevated/critical)
• Last 20 throttled actions with timestamps

WP-CLI Commands (Wave D)

wp queryguard throttle status     # Current load level, batch size, active deferrals
wp queryguard throttle pause      # Force critical mode (maintenance window)
wp queryguard throttle resume     # Return to auto-detection
wp queryguard throttle history    # Last 50 throttle events
wp queryguard mutex list          # Show active locks
wp queryguard mutex release <key> # Force-release a stuck lock

---

Deployment summary

Wave	What ships	Value on its own	Depends on
A	Load monitor extraction + priority registry	Clean architecture, registry available for filters — no behavior change	Nothing (safe anytime)
B	Per-action throttle + per-action logging	Selective throttling: payments keep flowing, deferrable syncs pause under load	Wave A
C	Mutex guard	Prevents thundering herd (N concurrent expensive operations)	Nothing (independent)
D	Admin UI + WP-CLI	Operator visibility and control without code deploys	Respective waves

---

Success criteria

From the original spec in #3:

• During May 1-like conditions: deferrable syncs auto-pause, critical payment processing continues, subscription renewals delay but complete
• Zero false positives during normal operation
• Admin visibility into throttle decisions in real-time
• No action is ever lost — only rescheduled

Dependencies

• #12 — Extract throttle engine into separate classes (covered by Wave A)
• Phase 1 runtime validation via test_observe / observe on staging should complete first to confirm load detection baselines

References

• Issue #3 — Original implementation framework
• PR #4 — Phase 1 AS throttle
• May 1 incident — BinoidCBD #818

[noelsaw1]

Codex's feedback:

Issue #19 is a mixed bag. The real product improvement is concentrated in one part: per-action throttling. Most of the rest is scaffolding or operator tooling.

My read:

• Wave B is the meaningful improvement. It solves the gap Phase 1 still has: today Phase 1 only slows the whole Action Scheduler runner; it does not distinguish payments from Facebook/Klaviyo/ShipStation work. If the goal is “keep revenue-critical work flowing while deferrable syncs back off,” this is the part that actually changes behavior in the right direction. That matches the original goal in #3 and builds on PR #4.

• Wave C is meaningful only if you have concrete integration points. A mutex is useful for thundering-herd patterns, but a generic Mutex Guard class does nothing by itself. It only helps if the expensive operations you care about are code paths you can actually wrap. So this is potentially high-value, but not automatically high-value.

• Wave A is mostly maintainability. The extraction from a ~760 line file into classes, as suggested in #12, is sane engineering, but it is not a user-facing improvement. The priority registry is only meaningful as an enabler for Wave B; standalone it is inert.

• Wave D is lowest leverage right now. WP-CLI commands are useful once the throttle exists. The admin UI is probably premature. It adds complexity, settings surface area, and support burden before the core behavior has proven itself.

So if you want a blunt percentage: roughly 25-35% of #19 is “real behavior improvement,” and that is mostly Wave B, plus some of Wave C if you can wire it into real hotspots. The other 65-75% is architecture, observability, and control-plane work.

Two important caveats:

• #19 assumes the Phase 1 throttle from PR #4 is your base. Your current main branch in this workspace does not have that code; it’s on development. So as of May 6, 2026, this issue is not an incremental improvement on current main, it’s an incremental improvement on development.

• The proposed Wave B hook, action_scheduler_before_execute, is technically plausible, because Action Scheduler checks status immediately after that hook in its queue runner source: ActionScheduler_Abstract_QueueRunner.php. But it is not perfect. It runs after the action has already been claimed, so it won’t prevent all queue overhead or claim churn. Real improvement, yes; complete prevention, no.

If you want the highest-signal version of #19, I’d cut it down to:

1. Ship Wave B.
2. Only do the minimum registry/refactor needed to support it.
3. Add CLI status/history, not a full admin UI.
4. Treat mutex work as a separate incident-driven track, only where you control the call sites.

[noelsaw1]

My question: Is Mutex Guard | Thundering herd lock so N concurrent requests don't all trigger the same expensive operation have a dependency or sequencing issue if we do that first?

Codex answer:
Not really a hard dependency issue. Wave C can ship before Wave A or B.

The real sequencing concern is not technical dependency, it’s usefulness:

• A generic Mutex Guard class is independent. It can be built first without needing the load monitor, priority registry, or per-action throttle.
• But it has an integration dependency: you need a specific expensive code path to wrap. Until you apply it to a real hotspot, it is just infrastructure.
• That makes it safer than Wave B, but also easier to overbuild with little immediate payoff.

So the honest answer is:

• No hard sequencing blocker.
• Yes practical sequencing risk if you do it “abstractly” before picking one concrete herd pattern.

If you do it first, the right order is:

1. Pick one real herd target, like a known NoFraud scan or another duplicated expensive operation.
2. Build the smallest lock API around that use case.
3. Prove lock acquisition, TTL expiry, and recovery behavior under concurrent requests.
4. Generalize only after that works.

The bigger thing to watch is failure semantics:

• Lock TTL must prevent deadlocks if PHP dies mid-request.
• Lock storage must be cross-process, so request-local memory is useless.
• “Fail open” vs “fail closed” needs to be explicit. For payment-critical flows, you probably want fail open.
• You need a stable operation key. Bad key design either over-serializes unrelated work or fails to stop the herd.

So yes, you can do Mutex Guard first. I would only do it first if you already know the first call site you want to protect. Without that, Wave B is the more meaningful first feature.