init

Author: markwallsgrove

EXERCISE_SUMMARY.md

@@ -0,0 +1,71 @@
+# Lab Automation Interview Exercise - Quick Reference
+
+## For the Interviewer
+
+This is a 60-90 minute paired programming exercise for a Staff Full-Stack Engineer role.
+
+### Setup (Before Interview)
+```bash
+docker-compose up --build
+```
+
+Verify all bugs are present by checking that Bug 2 prevents workflows from starting.
+
+### Exercise Flow
+
+**Phase 1: Debugging (30-40 min)**
+Candidate fixes 4 bugs:
+1. Missing `DEVICE_API_URL` env var → service crashes
+2. Wrong API endpoint path → workflows fail to start
+3. Race condition in device booking → run `./test-race-condition.sh` to demonstrate
+4. Frontend status cache → device status doesn't update in UI
+
+**Phase 2: Feature Implementation (30-50 min)**
+Implement pause/resume workflow feature (backend + frontend)
+
+### Quick Debugging
+
+If candidate gets stuck:
+
+**Bug 1**: Check `docker-compose logs workflow-service`
+**Bug 2**: Look at the endpoint path in `workflow-service/app.py` line ~127
+**Bug 3**: Run `./test-race-condition.sh` and check logs for multiple "successfully booked" messages
+**Bug 4**: Check `DeviceList.js` - notice the status caching logic
+
+### Evaluation Focus
+
+✅ Systematic debugging approach (logs, tools)
+✅ Full-stack competency (React + Python)
+✅ Understanding of distributed systems
+✅ Clean code and testing
+✅ Communication and problem-solving
+
+### Files to Review
+- `README.md` - Candidate instructions
+- `SOLUTION_GUIDE.md` - Complete solutions and fixes
+- `test-race-condition.sh` - Script to reproduce Bug 3
+
+### Key Artifacts
+- Candidate should fix bugs in code
+- Candidate should implement pause/resume endpoints + UI
+- Check git commits for quality
+
+---
+
+## For the Candidate
+
+See [README.md](README.md) for full instructions.
+
+### Quick Start
+```bash
+docker-compose up --build
+# Open http://localhost:3000
+# Fix bugs, then implement pause/resume feature
+```
+
+### Key Commands
+```bash
+docker-compose logs -f workflow-service    # View service logs
+./test-race-condition.sh                   # Test concurrency bug
+docker-compose restart workflow-service    # Restart after changes
+```

README.md

@@ -61,11 +61,14 @@ The system currently has several bugs that prevent it from working correctly. Yo
 - Enable workflows to be created and started successfully
 - Ensure device status updates correctly in the UI
 - Identify and fix the race condition in device booking
+  - Run `./test-race-condition.sh` to reproduce the concurrency bug
+  - Check logs to see if multiple workflows book the same device
 
 **Tips**:
 - Check service logs: `docker-compose logs [service-name]`
 - Use browser DevTools to inspect API calls
 - Test the system end-to-end after each fix
+- For the race condition, look for multiple "successfully booked" messages in device-service logs
 
 ### Phase 2: Feature Implementation (30-50 minutes)
 
@@ -115,8 +118,19 @@ We're looking for:
 # View logs for all services
 docker-compose logs -f
 
-# View logs for specific service
+# View logs for specific service (recommended for debugging)
 docker-compose logs -f workflow-service
+docker-compose logs -f device-service
+docker-compose logs -f sample-service
+
+# View recent logs without following
+docker-compose logs --tail=50 workflow-service
+
+# Test for race condition in device booking
+./test-race-condition.sh
+
+# Reset all data (clear workflows, device statuses, samples)
+docker-compose restart redis
 
 # Restart a specific service
 docker-compose restart workflow-service
@@ -127,6 +141,9 @@ docker-compose up --build
 # Stop all services
 docker-compose down
 
+# Stop all services and remove volumes (complete clean slate)
+docker-compose down -v
+
 # Check if services are healthy
 curl http://localhost:5001/health
 curl http://localhost:5002/health

SOLUTION_GUIDE.md

@@ -61,13 +61,65 @@ response = requests.post(
 
 ### Bug 3: Race Condition in Device Booking
 
-**Location**: `services/device-service/app.py:70-91`
+**Location**: `services/device-service/app.py:93-125`
 
 **Symptom**: Multiple workflows can book the same device simultaneously when requests arrive concurrently
 
 **Root Cause**: No atomic check-and-set operation. The code reads status, waits (simulating processing), then sets status - allowing race conditions.
 
-**Fix**: Use Redis atomic operations. Replace the book_device function:
+**How to Reproduce**: Run `./test-race-condition.sh` and check the logs. You'll see multiple "successfully booked" messages for the same device.
+
+**Fix Option 1 - Python Threading Lock** (Simpler, works for single instance):
+
+```python
+from threading import Lock
+
+# Add at module level
+device_locks = {}
+
+@app.route('/devices/<device_id>/book', methods=['POST'])
+def book_device(device_id):
+    """Book a device for a workflow"""
+    if device_id not in DEVICES:
+        logger.warning(f"Device not found: {device_id}")
+        return jsonify({'error': 'Device not found'}), 404
+
+    data = request.json
+    workflow_id = data.get('workflow_id')
+
+    if not workflow_id:
+        logger.error("Booking request missing workflow_id")
+        return jsonify({'error': 'workflow_id required'}), 400
+
+    # Create lock for this device if it doesn't exist
+    if device_id not in device_locks:
+        device_locks[device_id] = Lock()
+
+    logger.info(f"Attempting to book device {device_id} for workflow {workflow_id}")
+
+    # Use lock to ensure atomic check-and-set
+    with device_locks[device_id]:
+        current_status = get_device_status(device_id)
+
+        if current_status != 'available':
+            logger.warning(f"Device {device_id} is not available (status: {current_status})")
+            return jsonify({'error': 'Device is not available'}), 409
+
+        time.sleep(0.1)
+        set_device_status(device_id, 'busy', workflow_id)
+
+    logger.info(f"Device {device_id} successfully booked by workflow {workflow_id}")
+    return jsonify({
+        'device_id': device_id,
+        'status': 'busy',
+        'workflow_id': workflow_id,
+        'booked_at': datetime.utcnow().isoformat()
+    })
+```
+
+**Fix Option 2 - Redis Atomic Operations** (Better for distributed systems):
+
+Replace the book_device function:
 
 ```python
 @app.route('/devices/<device_id>/book', methods=['POST'])
@@ -124,15 +176,17 @@ def release_device(device_id):
 ```
 
 **Evaluation Points**:
-- ✅ Identifies the race condition (may require testing or code review)
-- ✅ Understands distributed systems challenges
-- ✅ Knows about atomic operations (SETNX, compare-and-swap, etc.)
-- ✅ Implements proper locking mechanism
-
-**Alternative Solutions** (also acceptable):
-- Database transactions with SELECT FOR UPDATE
-- Distributed locks (Redis SETNX, Redlock)
-- Optimistic locking with version numbers
+- ✅ Runs the test script to reproduce the issue
+- ✅ Identifies the race condition from logs or code review
+- ✅ Understands the need for atomic operations
+- ✅ Implements proper locking mechanism (either threading.Lock or Redis)
+- ✅ Verifies fix by running test script again
+
+**Acceptable Solutions**:
+- Threading Lock (shown above) - simple, works for single instance
+- Redis SETNX (shown above) - better for multiple instances
+- Remove the `time.sleep(0.1)` - makes race condition nearly impossible (acceptable but doesn't address root cause)
+- Database transactions with SELECT FOR UPDATE (if they add a database)
 
 ---
 
@@ -492,7 +546,7 @@ function WorkflowList({ workflows, onStart, onComplete, onPause, onResume }) {
 - [ ] Workflows can be created
 - [ ] Workflows fail to start (Bug 2 present)
 - [ ] Device status doesn't update in UI (Bug 4 present)
-- [ ] Race condition can be triggered with concurrent requests (Bug 3)
+- [ ] Race condition can be triggered: `./test-race-condition.sh` shows multiple bookings (Bug 3)
 
 ### Post-Exercise Validation
 
@@ -504,7 +558,7 @@ Run through this flow to verify all bugs are fixed:
 4. Check device status in UI → should show "busy"
 5. Complete the workflow → should succeed
 6. Check device status → should show "available"
-7. Test concurrent bookings (optional, for Bug 3)
+7. Test concurrent bookings: `./test-race-condition.sh` → should show only ONE successful booking
 
 ---
 
@@ -514,21 +568,24 @@ Run through this flow to verify all bugs are fixed:
 
 **Excellent Candidate**:
 - Systematically checks logs and identifies issues
+- Runs the race condition test script proactively
 - Fixes bugs in logical order (environment → API → race condition)
 - Implements clean, well-tested pause/resume feature
 - Asks clarifying questions about requirements
 - Explains their thought process clearly
-- Shows understanding of distributed systems
+- Shows understanding of distributed systems and concurrency
 
 **Good Candidate**:
 - Finds and fixes most bugs
+- Uses test script when prompted
 - Implements working pause/resume feature
-- May need hints for race condition
+- May need hints for race condition fix approach
 - Code works but could be cleaner
 
 **Needs Improvement**:
 - Struggles to identify bugs without significant help
 - Random debugging approach (trial and error)
+- Doesn't use provided tools (test script, logs)
 - Incomplete feature implementation
 - Doesn't test their changes
 
@@ -538,13 +595,14 @@ Typical breakdown for a strong candidate:
 - Bug 1 (env var): 5 minutes
 - Bug 2 (wrong endpoint): 5-10 minutes
 - Bug 4 (frontend cache): 10-15 minutes
-- Bug 3 (race condition): 10-20 minutes
+- Bug 3 (race condition): 15-20 minutes (including running test script and implementing fix)
 - Pause/Resume feature: 25-40 minutes
 
 If time is running short, you can:
-- Skip Bug 3 (race condition) - it's the most complex
+- Skip Bug 3 (race condition) or show them the test script output
 - Reduce scope of pause/resume (backend only)
 - Provide hints more liberally
+- Accept simpler fix for Bug 3 (just remove the `time.sleep(0.1)`)
 
 ### Discussion Questions
 

services/device-service/app.py

@@ -5,10 +5,21 @@
 import os
 import time
 from datetime import datetime
+import logging
+import sys
 
 app = Flask(__name__)
 CORS(app)
 
+logging.basicConfig(
+    level=logging.INFO,
+    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+    handlers=[logging.StreamHandler(sys.stdout)]
+)
+logger = logging.getLogger(__name__)
+
+app.logger.setLevel(logging.INFO)
+
 # Redis connection
 redis_client = redis.from_url(os.getenv('REDIS_URL', 'redis://localhost:6379'))
 
@@ -87,24 +98,29 @@ def get_device(device_id):
 def book_device(device_id):
     """Book a device for a workflow"""
     if device_id not in DEVICES:
+        logger.warning(f"Device not found: {device_id}")
         return jsonify({'error': 'Device not found'}), 404
 
     data = request.json
     workflow_id = data.get('workflow_id')
 
     if not workflow_id:
+        logger.error("Booking request missing workflow_id")
         return jsonify({'error': 'workflow_id required'}), 400
 
+    logger.info(f"Attempting to book device {device_id} for workflow {workflow_id}")
+
     current_status = get_device_status(device_id)
 
     if current_status != 'available':
+        logger.warning(f"Device {device_id} is not available (status: {current_status})")
         return jsonify({'error': 'Device is not available'}), 409
 
-    # Simulate some processing time that makes race condition more likely
     time.sleep(0.1)
 
     set_device_status(device_id, 'busy', workflow_id)
 
+    logger.info(f"Device {device_id} successfully booked by workflow {workflow_id}")
     return jsonify({
         'device_id': device_id,
         'status': 'busy',
@@ -116,18 +132,22 @@ def book_device(device_id):
 def release_device(device_id):
     """Release a device from a workflow"""
     if device_id not in DEVICES:
+        logger.warning(f"Device not found: {device_id}")
         return jsonify({'error': 'Device not found'}), 404
 
     data = request.json
     workflow_id = data.get('workflow_id')
 
-    # Verify the workflow owns this device
+    logger.info(f"Attempting to release device {device_id} from workflow {workflow_id}")
+
     current_workflow = redis_client.get(f'device:{device_id}:workflow')
     if current_workflow and current_workflow.decode('utf-8') != workflow_id:
+        logger.warning(f"Device {device_id} is booked by another workflow")
         return jsonify({'error': 'Device is booked by another workflow'}), 403
 
     set_device_status(device_id, 'available')
 
+    logger.info(f"Device {device_id} released successfully")
     return jsonify({
         'device_id': device_id,
         'status': 'available',
@@ -138,20 +158,23 @@ def release_device(device_id):
 def execute_operation(device_id):
     """Execute an operation on a device"""
     if device_id not in DEVICES:
+        logger.warning(f"Device not found: {device_id}")
         return jsonify({'error': 'Device not found'}), 404
 
     data = request.json
     operation = data.get('operation')
     workflow_id = data.get('workflow_id')
 
-    # Verify the workflow owns this device
+    logger.info(f"Executing operation '{operation}' on device {device_id} for workflow {workflow_id}")
+
     current_workflow = redis_client.get(f'device:{device_id}:workflow')
     if not current_workflow or current_workflow.decode('utf-8') != workflow_id:
+        logger.warning(f"Device {device_id} not booked by workflow {workflow_id}")
         return jsonify({'error': 'Device not booked by this workflow'}), 403
 
-    # Simulate operation execution
     time.sleep(0.5)
 
+    logger.info(f"Operation '{operation}' completed on device {device_id}")
     return jsonify({
         'device_id': device_id,
         'operation': operation,
@@ -165,4 +188,4 @@ def execute_operation(device_id):
         if not redis_client.exists(f'device:{device_id}:status'):
             set_device_status(device_id, 'available')
 
-    app.run(host='0.0.0.0', port=5001, debug=True)
+    app.run(host='0.0.0.0', port=5001, debug=False)