ThordekkCore/THREADPOOL_DEADLOCK_FIXES_SUMMARY.md

ThreadPool Deadlock Fixes - Complete Summary

Date: 2025-10-20 Version: Final (Fix D Applied) Status: Build Successful

Deadlock Timeline & Fixes Applied

Deadlock #1: SpatialGrid Contention (10/19)

Symptom: 10-second hangs during bot movement updates Root Cause: All threads trying to update same grid cell simultaneously Fix A: Implemented SpatialGridScheduler for centralized, sequential grid updates Status: ✅ Applied and tested

Deadlock #2: Sleep/Wake Race Condition (10/19)

Symptom: 24 workers stuck in condition variable wait (1ms timeout) Root Cause: Wake() checked _sleeping flag before Sleep() set it Fix B: Lock before flag operations in both Sleep() and Wake() Status: ✅ Applied and tested

Deadlock #3: Work Distribution Imbalance (10/19)

Symptom: Work piling up on single worker while others sleep Root Cause: Only waking one worker on task submission Fix C: Wake multiple workers (25%, min 2, max 4) + increase sleep timeout to 10ms Status: ✅ Applied and tested

Deadlock #4: Steal Backoff Contention (10/20)

Symptom: Identical to #2 but with 1ms timeout (regression appearance) Root Cause: TryStealTask() exponential backoff using same CV as Sleep(), capping at 1ms Fix D: Replace CV-based backoff with yield-based strategy Status: ✅ Applied - BUILD SUCCESSFUL

Complete Fix Implementation

Fix A: SpatialGridScheduler

// Centralized grid update system
class SpatialGridScheduler {
    void ProcessPendingUpdates() {
        // Sequential processing prevents contention
        for (auto& update : pendingUpdates) {
            grid->UpdateCell(update);
        }
    }
};

Fix B: Sleep/Wake Race Fix

void WorkerThread::Wake() {
    // Lock BEFORE checking flag
    std::lock_guard<std::mutex> lock(_wakeMutex);
    _sleeping.store(false);
    _wakeCv.notify_one();
}

void WorkerThread::Sleep() {
    // Lock BEFORE setting flag
    std::unique_lock<std::mutex> lock(_wakeMutex);
    _sleeping.store(true);
    // ... wait logic ...
}

Fix C: Multi-Worker Wake & Timeout

// In ThreadPool::Submit()
// Wake multiple workers for better distribution
uint32 workersToWake = std::min(4u, std::max(2u, workers.size() / 4));
for (uint32 i = 0; i < workersToWake; ++i) {
    workers[i]->Wake();
}

// Configuration change
std::chrono::milliseconds workerSleepTime{10};  // Was 1ms

Fix D: Yield-Based Steal Backoff (NEW)

bool WorkerThread::TryStealTask() {
    uint32 yieldsPerAttempt = 1;

    while (attempts < maxAttempts) {
        // ... steal attempt logic ...

        if (attempts < maxAttempts) {
            // Yield-based backoff instead of CV wait
            for (uint32 y = 0; y < yieldsPerAttempt; ++y) {
                if (shutdown || urgent_work) return false;
                std::this_thread::yield();
            }
            yieldsPerAttempt = std::min(yieldsPerAttempt * 2, 8u);
        }
    }
}

Key Insights

1. Multiple Wait Points Create Confusion

• Sleep() uses _wakeCv with 10ms timeout
• TryStealTask() ALSO used _wakeCv with 1ms cap
• Debugger showed 1ms, leading to confusion about regression

2. Shared Condition Variables Are Dangerous

• Using same CV for different purposes creates ambiguity
• Hard to debug which wait is blocking
• Solution: Separate CVs or eliminate unnecessary waits

3. Exponential Backoff Can Cause Deadlock

• When ALL threads enter backoff simultaneously
• They all wait on same CV
• No thread available to produce work

4. Yield > Wait for Work Stealing

• std::this_thread::yield() doesn't block
• Allows immediate response to new work
• Prevents condition variable contention

Performance Impact

Before Fixes

• Deadlocks every 10-30 minutes under load
• 24+ threads blocked in wait states
• Server hangs requiring restart

After Fixes

• No condition variable waits in steal path
• Sleep/Wake properly synchronized
• Multiple workers wake for distribution
• Yield-based backoff prevents blocking

Testing Requirements

Stress Test Scenarios

1. 100+ Concurrent Bots: Verify no deadlocks
2. Rapid Task Submission: Test wake distribution
3. Empty Queue Scenario: Test steal backoff behavior
4. Mixed Priority Tasks: Verify priority handling

Monitoring Points

• Thread state distribution (active/sleeping/stealing)
• Steal success rate
• Average queue depth per worker
• Task completion latency

Configuration Recommendations

# Optimal settings for production
Playerbot.Performance.ThreadPool.WorkerCount = 0  # Auto-detect
Playerbot.Performance.ThreadPool.MaxQueueSize = 10000
Playerbot.Performance.ThreadPool.EnableWorkStealing = 1
Playerbot.Performance.ThreadPool.EnableCpuAffinity = 0  # Keep disabled

# Internal (not configurable):
# workerSleepTime = 10ms
# maxStealAttempts = 3
# yieldsPerAttempt = 1-8 (exponential)

Lessons Learned

1. Always separate concerns: Different wait purposes need different primitives
2. Avoid shared condition variables: Creates debugging nightmares
3. Prefer lock-free patterns: yield() > wait() for work stealing
4. Document wait timeouts: Makes debugging much easier
5. Test under full load: Deadlocks often only appear at scale

Next Steps

1. ✅ Fix D implemented and compiled
2. ⚠️ Deploy to test environment
3. ⚠️ Run stress test with 100+ bots
4. ⚠️ Monitor for 24 hours
5. ⚠️ Deploy to production if stable

Files Modified

• src/modules/Playerbot/Performance/ThreadPool/ThreadPool.h

  • Removed _stealBackoff atomic flag
  • Updated comments

• src/modules/Playerbot/Performance/ThreadPool/ThreadPool.cpp

  • TryStealTask(): Replaced CV wait with yield strategy
  • Wake(): Removed stealBackoff flag clear
  • Added progressive yield backoff

Build Status

Build Time: 2025-10-20 (current)
Configuration: Debug
Compiler: MSVC
Status: SUCCESS
Warnings: 15 (unrelated to ThreadPool)

---

CRITICAL: All 4 fixes are now applied. The deadlock pattern should be completely eliminated. The key insight was recognizing that the 1ms timeout came from steal backoff, not the main sleep function, requiring a different fix than initially suspected.