atlantis/docs/enhanced-plan-queue-locks.md

Enhanced Plan Queue and Locking System

This document describes the enhanced plan queue and locking system that addresses the long-standing issues with Atlantis locking and workspace management.

Overview

The enhanced system provides:

1. Plan Queue Functionality - Queues plan requests when locks are unavailable
2. Lock Retry Logic - Automatically retries lock acquisition with configurable delays
3. Race Condition Prevention - In-memory locks prevent concurrent operations on the same project/workspace
4. Working Directory Protection - Prevents premature deletion of working directories
5. Automatic Lock Transfer - Transfers locks to the next person in queue when available

Features

1. Plan Queue System

When a plan request comes in and the project is already locked by another PR, instead of immediately failing, the PR gets queued. The system will:

• Add the request to a queue for that specific project/workspace
• Notify the user about their position in the queue
• Automatically transfer the lock to the next person when the current lock is released
• Notify users when it's their turn to plan

Configuration:

--enable-plan-queue=true

2. Lock Retry Logic

Instead of immediately failing when a lock is busy, the system can automatically retry lock acquisition with configurable delays.

Configuration:

--enable-lock-retry=true
--lock-retry-max-attempts=3
--lock-retry-delay=5

3. Race Condition Prevention

The system uses in-memory locks to prevent race conditions between concurrent operations on the same project/workspace. This addresses issues like:

• Multiple plan requests for the same project/workspace
• Race conditions between post-workflow hooks and automerge pull cleanup
• Lock creation for no apparent reason

4. Working Directory Protection

The system protects working directories from premature deletion by:

• Tracking which working directories are in use
• Preventing deletion while operations are in progress
• Automatic cleanup when operations complete

5. Automatic Lock Transfer

When a lock is released, the system automatically:

• Checks if there are queued requests for that project/workspace
• Transfers the lock to the next person in the queue
• Notifies the user that they now have the lock

Configuration

Command Line Flags

# Enable plan queue functionality
--enable-plan-queue=true

# Enable lock retry functionality
--enable-lock-retry=true

# Maximum number of retry attempts (default: 3)
--lock-retry-max-attempts=3

# Delay between retry attempts in seconds (default: 5)
--lock-retry-delay=5

Environment Variables

# Enable plan queue functionality
ATLANTIS_ENABLE_PLAN_QUEUE=true

# Enable lock retry functionality
ATLANTIS_ENABLE_LOCK_RETRY=true

# Maximum number of retry attempts
ATLANTIS_LOCK_RETRY_MAX_ATTEMPTS=3

# Delay between retry attempts in seconds
ATLANTIS_LOCK_RETRY_DELAY=5

Architecture

Components

1. EnhancedLockingSystem - Core locking system with retry and queue support
2. PlanQueueManager - Manages plan queues for projects/workspaces
3. EnhancedProjectLocker - Enhanced project locker with queue integration
4. Memory Locks - In-memory locks to prevent race conditions
5. Working Directory Protection - Protects working directories from deletion

Data Flow

1. Plan Request → EnhancedProjectLocker
2. Lock Check → EnhancedLockingSystem
3. If Lock Available → Acquire lock and proceed
4. If Lock Busy → Add to queue or retry (based on configuration)
5. Lock Release → Transfer to next person in queue
6. Working Directory → Protected during operations

Benefits

For Users

• No More Manual Retries - System automatically retries lock acquisition
• Queue Awareness - Users know their position in the queue
• Automatic Notifications - Users are notified when it's their turn
• Reduced Interruptions - Fewer failed plan requests due to busy locks

For Operators

• Reduced Support Load - Fewer issues with locks and workspaces
• Better Resource Utilization - Queues ensure efficient use of resources
• Improved Reliability - Race conditions and workspace issues are prevented
• Better Monitoring - Queue status and lock transfers are logged

For the System

• Improved Stability - Race conditions are eliminated
• Better Resource Management - Working directories are properly protected
• Scalability - Queue system handles high concurrency better
• Maintainability - Cleaner separation of concerns

Migration

From Default Locking

The enhanced system is backward compatible. To migrate:

1. Enable features gradually - Start with lock retry, then add queue functionality
2. Monitor logs - Watch for any issues during migration
3. Adjust configuration - Tune retry attempts and delays based on your environment

Configuration Examples

Conservative Migration:

--enable-lock-retry=true
--lock-retry-max-attempts=2
--lock-retry-delay=10
--enable-plan-queue=false

Full Feature Set:

--enable-lock-retry=true
--lock-retry-max-attempts=3
--lock-retry-delay=5
--enable-plan-queue=true

Troubleshooting

Common Issues

1. Queue Not Working

   • Check if --enable-plan-queue=true is set
   • Verify queue manager is properly initialized
   • Check logs for queue-related errors

2. Retry Not Working

   • Check if --enable-lock-retry=true is set
   • Verify retry configuration values
   • Check logs for retry attempts

3. Working Directory Issues

   • Check if working directory protection is enabled
   • Verify cleanup is happening properly
   • Check logs for protection-related messages

Log Messages

The system provides detailed logging for:

• Queue operations (add, remove, transfer)
• Retry attempts and results
• Working directory protection
• Lock transfers
• Memory lock operations

Monitoring

Key metrics to monitor:

• Queue length per project/workspace
• Retry success/failure rates
• Lock transfer success rates
• Working directory protection status

Future Enhancements

Potential future improvements:

1. Priority Queues - Allow users to set priority for their requests
2. Queue Timeouts - Automatically remove stale queue entries
3. Queue Persistence - Store queues in backend for persistence across restarts
4. Advanced Notifications - Slack/email notifications for queue updates
5. Queue Analytics - Metrics and dashboards for queue performance

References

• ADR #3345 - Project Locks
• PR #4997 - Lock Retry Logic
• Issue #1914 - Workspace lock creation
• Issue #2200 - Lock creation for no apparent reason
• Issue #3336 - Race condition between hooks and cleanup