Skip to content

Distributed Locks

Jump to bottom
Nick edited this page Mar 10, 2026 · 2 revisions

Distributed Locks for Multi-Instance Deployment

PATAS supports horizontal scaling across multiple instances using distributed locks to coordinate operations and prevent duplicate work.

Overview

When running PATAS on multiple instances (e.g., for high availability or load distribution), distributed locks ensure that:

  • Pattern mining operations don't run simultaneously on different instances
  • Rule promotion/deprecation operations are coordinated
  • No duplicate work is performed

Architecture

PATAS uses a two-tier locking strategy:

  1. Redis (Preferred): Fast, distributed locks with automatic expiration
  2. PostgreSQL Advisory Locks (Fallback): Database-level locks when Redis is unavailable

Lock Types

  • Pattern Mining Locks: Prevent concurrent pattern mining operations
  • Rule Promotion Locks: Coordinate rule promotion/deprecation across instances

Configuration

Redis Setup

  1. Install and start Redis:
# Using Docker
docker run -d -p 6379:6379 redis:latest

# Or using system package manager
sudo apt-get install redis-server
  1. Configure PATAS to use Redis:
# In .env or environment variables
REDIS_URL=redis://localhost:6379/0
ENABLE_DISTRIBUTED_LOCKS=true
LOCK_TIMEOUT_SECONDS=3600

PostgreSQL Fallback

If Redis is not configured, PATAS automatically falls back to PostgreSQL advisory locks. No additional configuration is needed.

Disabling Distributed Locks

For single-instance deployments, you can disable distributed locks:

ENABLE_DISTRIBUTED_LOCKS=false

Lock Behavior

Pattern Mining Locks

  • Lock Key: pattern_mining:{days}:{min_spam_count}
  • Timeout: Configurable (default: 3600 seconds)
  • Behavior: If another instance is already mining patterns with the same parameters, the operation returns immediately with already_in_progress status

Rule Promotion Locks

  • Lock Keys:
    • rule_promotion:shadow - For shadow rule promotion
    • rule_promotion:monitor - For active rule monitoring
  • Timeout: Configurable (default: 3600 seconds)
  • Behavior: Only one instance can promote or monitor rules at a time

Heartbeat Mechanism

When using Redis, locks are automatically refreshed (heartbeat) to prevent expiration during long-running operations:

  • Heartbeat Interval: 60 seconds (configurable)
  • Automatic Extension: Lock TTL is refreshed while the operation is running

Error Handling

  • Redis Connection Failure: Automatically falls back to PostgreSQL advisory locks
  • Lock Acquisition Failure: Operation returns with appropriate error message
  • Lock Release Failure: Logged as warning, doesn't affect operation result

Best Practices

  1. Use Redis for Production: Provides better performance and automatic expiration
  2. Set Appropriate Timeouts: Ensure lock timeout exceeds expected operation duration
  3. Monitor Lock Contention: Watch for frequent already_in_progress responses
  4. Single Instance: Disable distributed locks for single-instance deployments to reduce overhead

Troubleshooting

Lock Not Acquired

If you see already_in_progress errors:

  1. Check if another instance is running the same operation
  2. Verify lock timeout is sufficient for your operation duration
  3. Check Redis/PostgreSQL connectivity

Redis Connection Issues

If Redis is unavailable:

  1. PATAS automatically falls back to PostgreSQL locks
  2. Check Redis connectivity: redis-cli ping
  3. Verify REDIS_URL configuration

Lock Expiration

If locks expire during long operations:

  1. Increase LOCK_TIMEOUT_SECONDS configuration
  2. Check Redis memory and connection pool settings
  3. Monitor operation duration and optimize if needed

Related Documentation

Clone this wiki locally