OpenAI's engineers have recently tackled a complex bug in their C++ data infrastructure service, Rockset, which powers ChatGPT's search and data plugins. The issue was a mysterious crash that seemed impossible to explain, with functions returning to bogus memory addresses and stack pointers shifting unexpectedly. The team's initial approach was to inspect individual crashes, but this led to a dead end. They then adopted an epidemiological debugging strategy, analyzing production core dumps to identify population-level patterns.
By using ChatGPT to write a script that automatically analyzed core dumps, they discovered two unrelated bugs. One cluster of crashes was traced to a single physical host in an Azure region, where the CPU was silently producing incorrect results. Removing this host resolved the issue. The other cluster was related to a race condition in GNU libunwind's Ux8664_setcontext function, which had been present for 18 years. This function updates the stack pointer before reading the instruction pointer, creating a window where a signal could corrupt the instruction pointer.
The fix involved reordering instructions to ensure the instruction pointer is read before the stack pointer is updated, eliminating the race window. This approach highlights the importance of building a high-quality dataset for debugging production crashes. By analyzing the entire population of failures, the team was able to distinguish between two distinct crash populations and identify the root cause of the issue.
This incident underscores the value of epidemiological debugging and the importance of comprehensive data analysis in resolving complex software bugs. It also emphasizes the need for a systematic approach to debugging, where a thorough examination of the entire dataset can lead to a clearer understanding of the problem and more effective solutions.
