fix(threading): Add counter mismatch auto-correction in ThreadPool
Root cause identified: The "1 in-flight, 0 active workers" timeout was caused by a counter mismatch where totalSubmitted > totalCompleted even though all tasks had actually finished. This can happen if an exception occurs during task submission that doesn't properly update the completed counter. Solution: In WaitForCompletion(), detect the mismatch condition: - All queues are empty - All workers are sleeping (0 active) - But counters show in-flight tasks When detected, log a warning and correct the counters to prevent the permanent 10-second timeout on every update cycle. This is a workaround - the real fix would be to ensure all code paths properly maintain counter balance. But this prevents the timeout from blocking bot updates indefinitely. Co-Authored-By: Claude Opus 4.5 <[email protected]> Signed-off-by: luis <[email protected]>
This commit is contained in:
committed by
luis
co-authored by
Claude Opus 4.5
parent
3013dfd1cf
commit
cb683ef67a
@@ -799,6 +799,23 @@ bool ThreadPool::WaitForCompletion(::std::chrono::milliseconds timeout)
|
||||
if (allEmpty && allTasksFinished)
|
||||
return true;
|
||||
|
||||
// COUNTER MISMATCH DETECTION: If all queues are empty and no workers are active,
|
||||
// but counters show in-flight tasks, we have a "ghost counter" issue.
|
||||
// This can happen if an exception occurs during task submission or execution
|
||||
// that doesn't properly update the completed counter.
|
||||
if (allEmpty && GetActiveThreads() == 0 && submitted > completed)
|
||||
{
|
||||
uint64 inFlight = submitted - completed;
|
||||
TC_LOG_WARN("module.playerbot.threadpool",
|
||||
"COUNTER MISMATCH DETECTED: {} in-flight tasks but all queues empty and 0 active workers. "
|
||||
"Correcting counters to prevent permanent timeout.",
|
||||
inFlight);
|
||||
|
||||
// Correct the mismatch by advancing totalCompleted
|
||||
_metrics.totalCompleted.fetch_add(inFlight, ::std::memory_order_relaxed);
|
||||
return true; // All work is actually done
|
||||
}
|
||||
|
||||
// Check timeout - use effective timeout with hard cap
|
||||
auto now = ::std::chrono::steady_clock::now();
|
||||
if (::std::chrono::duration_cast<::std::chrono::milliseconds>(now - start) >= effectiveTimeout)
|
||||
|
||||
Reference in New Issue
Block a user