ThordekkCore/THREADPOOL_DEADLOCK_ANALYSIS.md

ThreadPool Deadlock Analysis - Critical Concurrency Issue

Executive Summary

CRITICAL FINDING: The ThreadPool has a fundamental design flaw in its wait/wake mechanism that causes ALL 33 worker threads to enter permanent _Primitive_wait_for state despite having timeout protection.

Deadlock Pattern Identified

Root Cause: Spurious Wakeup + Immediate Re-Sleep Race Condition

The deadlock occurs through this sequence:

1. Initial State: All workers actively processing tasks
2. Task Completion: Workers complete their tasks and check for more work
3. Sleep Decision: Workers find no local work and attempt to steal
4. Steal Failure: All workers fail to steal (queues empty or contention)
5. Mass Sleep Entry: Workers enter Sleep() nearly simultaneously
6. Critical Race Window: Between _sleeping.store(true) and cv.wait_for()
7. Wake Signal Lost: New task arrives, Wake() called, but signal lost in race
8. Permanent Sleep: All workers now waiting with no one to wake them

Detailed Sequence Diagram

Time    Worker1         Worker2         Worker3...33    Main Thread     Issue
----    --------        --------        ------------    -----------     -----
T0      Execute()       Execute()       Execute()       Submit(task1)   Normal operation
T1      Complete        Complete        Complete        Submit(task2)
T2      TrySteal()     TrySteal()      TrySteal()                     All stealing
T3      Fail-Yield     Fail-Yield      Fail-Yield                     Yield backoff working
T4      Sleep()        Sleep()         Sleep()                         Mass sleep entry
T5      Lock mutex     Lock mutex      Lock mutex                      Contention begins
T6      Set sleeping   Wait...         Wait...                         Sequential entry
T7      HasWork=false  Set sleeping    Wait...         Submit(taskN)   New work arrives!
T8      wait_for()     HasWork=false   Set sleeping    Wake(W1)        Wake sent to sleeping W1
T9      [WAITING]      wait_for()      HasWork=false   Wake(W2,W3)     Multi-wake attempted
T10     [timeout]      [WAITING]       wait_for()                      W1 times out BUT...
T11     sleeping=false [timeout]       [WAITING]       Submit(more)    W1 checks HasWork()
T12     HasWork=false  sleeping=false  [timeout]                       Still no work visible!
T13     Sleep() AGAIN  HasWork=false   sleeping=false                  RE-ENTERS SLEEP
T14     [WAITING]      Sleep() AGAIN   HasWork=false                   Vicious cycle
T15     [ALL WAITING]  [ALL WAITING]   [ALL WAITING]                   DEADLOCK

Critical Code Flaws Identified

1. HasWorkAvailable() Race Condition (Line 478-509)

bool WorkerThread::HasWorkAvailable() const
{
    // FLAW: This checks CURRENT state, not FUTURE state
    // Work may arrive AFTER this check but BEFORE wait_for()
    for (size_t i = 0; i < static_cast<size_t>(TaskPriority::COUNT); ++i)
    {
        if (!_localQueues[i].Empty())  // <-- Snapshot in time
            return true;
    }
    // ... steal checks ...
    return false;  // <-- May become stale immediately
}

2. Sleep() Double-Check Pattern Failure (Line 511-548)

void WorkerThread::Sleep()
{
    std::unique_lock<std::mutex> lock(_wakeMutex);
    _sleeping.store(true, std::memory_order_relaxed);

    // CRITICAL FLAW: Gap between check and wait
    bool hasWork = HasWorkAvailable();  // <-- Check at T1

    if (hasWork)
    {
        _sleeping.store(false, std::memory_order_relaxed);
        return;
    }

    // RACE WINDOW: Work can arrive HERE between check and wait
    // Wake() can fire HERE and be lost

    _wakeCv.wait_for(lock, _pool->GetConfiguration().workerSleepTime, [this]() {
        return !_sleeping.load(std::memory_order_relaxed) ||
               !_running.load(std::memory_order_relaxed) ||
               _pool->IsShuttingDown();
    });  // <-- Wait at T2, may miss wake between T1 and T2
}

3. Wake() Notification Timing Issue (Line 443-455)

void WorkerThread::Wake()
{
    std::lock_guard<std::mutex> lock(_wakeMutex);
    _sleeping.store(false, std::memory_order_relaxed);
    _wakeCv.notify_one();  // <-- May fire BEFORE thread enters wait_for()
}

4. Submit() Insufficient Wake Strategy (Line 552-589)

// Wake worker if sleeping
worker->Wake();

// Wake additional workers - BUT WHAT IF THEY'RE ALL MID-SLEEP-ENTRY?
if (_config.enableWorkStealing && !_workers.empty())
{
    uint32 workersToWake = std::min(4u, std::max(2u, ...));
    for (uint32 i = 0; i < workersToWake; ++i)
    {
        // FLAW: Only wakes if CURRENTLY sleeping
        // Misses threads in the PROCESS of going to sleep
        if (helperWorker->_sleeping.load(std::memory_order_relaxed))
        {
            helperWorker->Wake();
        }
    }
}

Why 10ms Timeout Doesn't Prevent Deadlock

The timeout DOES fire, but workers immediately re-enter sleep because:

1. Spurious Wakeup Handling: Worker wakes from timeout
2. Re-checks Conditions: Still no work visible (work is in another worker's queue)
3. Work Stealing Fails: Can't steal because victims are also sleeping/waking
4. Re-enters Sleep: Goes back to sleep for another 10ms
5. Infinite Loop: All workers in this timeout->check->sleep cycle

Proof of Deadlock Scenario

When ALL workers simultaneously:

1. Complete their tasks
2. Find no local work
3. Attempt to steal (all fail - nothing to steal)
4. Enter Sleep() within same ~1ms window
5. New tasks arrive AFTER HasWorkAvailable() but BEFORE wait_for()
6. Wake signals are sent but lost (threads not yet waiting)
7. All threads enter wait_for() AFTER wake signals sent
8. Timeout fires, but HasWorkAvailable() still returns false
9. Re-enter Sleep() immediately
10. System enters stable deadlock state

Contributing Factors

1. Work-Stealing Yield Backoff: Reduces contention but increases sleep entry probability
2. Multi-Worker Wake: Good idea but insufficient when all are mid-transition
3. Lock-Free Queues: Fast but make work visibility non-atomic across workers
4. High Worker Count (33): Increases probability of simultaneous sleep entry

Recommended Fix

Solution: Epoch-Based Wake Guarantee System

class WorkerThread {
    // Add wake epoch counter
    std::atomic<uint64_t> _wakeEpoch{0};
    std::atomic<uint64_t> _lastProcessedEpoch{0};

    void Sleep() {
        std::unique_lock<std::mutex> lock(_wakeMutex);

        // Capture current epoch BEFORE setting sleeping flag
        uint64_t currentEpoch = _wakeEpoch.load(std::memory_order_acquire);

        // Set sleeping flag
        _sleeping.store(true, std::memory_order_release);

        // Check for work with epoch validation
        bool hasWork = HasWorkAvailable();
        uint64_t epochAfterCheck = _wakeEpoch.load(std::memory_order_acquire);

        // If epoch changed, we MUST have been woken (even if we missed the signal)
        if (hasWork || epochAfterCheck != currentEpoch) {
            _sleeping.store(false, std::memory_order_release);
            return;
        }

        // Wait with epoch-based predicate
        _wakeCv.wait_for(lock, workerSleepTime, [this, currentEpoch]() {
            // Wake if epoch changed (guaranteed wake) OR other conditions
            return _wakeEpoch.load(std::memory_order_acquire) != currentEpoch ||
                   !_sleeping.load(std::memory_order_relaxed) ||
                   !_running.load(std::memory_order_relaxed);
        });

        // Record that we processed this epoch
        _lastProcessedEpoch.store(_wakeEpoch.load(std::memory_order_acquire));
        _sleeping.store(false, std::memory_order_release);
    }

    void Wake() {
        std::lock_guard<std::mutex> lock(_wakeMutex);

        // Increment epoch BEFORE clearing sleeping flag
        // This GUARANTEES the sleeping thread will see the change
        _wakeEpoch.fetch_add(1, std::memory_order_release);

        _sleeping.store(false, std::memory_order_release);
        _wakeCv.notify_one();
    }
};

Additional Safety Measures

1. Periodic Broadcast Wake: Every 100ms, wake ALL workers to prevent stable deadlock
2. Work Generation Detection: Track task submissions and force-wake on new work
3. Adaptive Sleep Delay: Stagger sleep entry based on worker ID
4. Emergency Recovery: Detect all-sleeping state and force recovery

Testing Strategy

Deadlock Reproduction Test

TEST(ThreadPoolDeadlock, SimultaneousSleepEntry) {
    ThreadPool pool(33);  // Match production config

    // Phase 1: Saturate pool with quick tasks
    std::vector<std::future<void>> futures;
    for (int i = 0; i < 33; ++i) {
        futures.push_back(pool.Submit(TaskPriority::NORMAL, []() {
            std::this_thread::sleep_for(std::chrono::microseconds(10));
        }));
    }

    // Wait for completion
    for (auto& f : futures) f.get();
    futures.clear();

    // Phase 2: All workers now idle, about to sleep
    // Small delay to let them enter sleep
    std::this_thread::sleep_for(std::chrono::milliseconds(5));

    // Phase 3: Submit burst of new work
    for (int i = 0; i < 100; ++i) {
        futures.push_back(pool.Submit(TaskPriority::HIGH, []() {
            std::this_thread::sleep_for(std::chrono::milliseconds(1));
        }));
    }

    // Phase 4: Check if work completes (should not deadlock)
    auto start = std::chrono::steady_clock::now();
    for (auto& f : futures) {
        auto status = f.wait_for(std::chrono::seconds(5));
        ASSERT_EQ(status, std::future_status::ready) << "Deadlock detected!";
    }
}

Conclusion

The ThreadPool deadlock is caused by a fundamental race condition in the sleep/wake mechanism where wake signals can be lost when all workers simultaneously transition to sleep state. The 10ms timeout doesn't prevent the deadlock because workers immediately re-enter sleep when they timeout, creating a stable deadlock state.

The recommended epoch-based wake guarantee system ensures that no wake signal can ever be lost, even in the presence of race conditions, while maintaining the performance benefits of the work-stealing architecture.