One CPU Atomic Instruction, One Packaging Infinite Loop: The Story of the Lost Update on LA664 - 杰哥的{教学，运维，编程，调板子}小笔记

cpu erratum loongarch loongson 中文版本 TL;DR¶ In February 2026, Wang Miao ran into something strange while packaging normaliz for Debian on a LoongArch server: the math software's built-in test kept timing out, stuck in an infinite loop that it could not escape. Following the code, the problem pointed to a very ordinary operation: OpenMP's #pragma omp atomic accumulating into a shared variable. The loop's exit condition required the accumulated value to equal a certain number, but the accumulated result was always less than that number, causing the infinite loop. Because the program was large and the code complex, we never managed to reduce it to a minimal example a human could understand, so the matter was shelved. Half a year later, in August, Wang Miao came to me again, wanting to pick it back up. This time we took a different approach: instead of having a human locate the problem, we let AI find a minimal reproduction, with the human directing the AI's investigation. About two days later, we had a stable reproducer, and only then discovered the root cause: the CPU's atomic add instruction occasionally fails to be atomic. This meant we had found a new CPU erratum, and after Loongson learned of it, only two weeks passed before they found a fix with almost no performance loss and provided us with test firmware. We confirmed that the test firmware resolves the issue, and Loongson told us the firmware is expected to be released before National Day (October 1), at which point readers will be able to upgrade their firmware to fix the problem. Now let us tell the whole story from beginning to end. Origins¶ loong13 is a community-maintained port of Debian 13 stable to LoongArch, and Wang Miao is one of its maintainers. During the build and packaging process, normaliz's built-in test was found to get stuck in a loop that it could not exit, causing the packaging to time out. At the time we did not immediately find the root cause, so we had no choice but to skip this package. But since several other packages depend on normaliz, we could not keep skipping it forever, so in February we began to focus on investigating the problem. Previously, while building other packages, we had found hidden race conditions or memory-ordering issues in the code, and such problems are more likely to surface on LoongArch, which uses a weak memory model. So at first we guessed the cause might be a similar issue in this software. But once the investigation began, surprise, surprise, there was a surprise. The First Round of Investigation¶ The first round started from normaliz's source code. normaliz uses OpenMP to process data points in parallel. The problematic code snippet can be summarized as follows: func (std::list<std::vector<int>> LatticePoints) { size_t nr_to_match = LatticePoints.size(); // input size size_t nr_points_matched = 0; // number of points already processed while (true) { size_t nr_points_done_in_this_round = 0; // number of points processed this round #pragma omp parallel { auto P = LatticePoints.begin(); // thread-private List pointer size_t ppos = 0; // thread-private List pointer position #pragma omp for for (ppp = 0...nr_to_match){ if (skip_remaining) { // in certain cases skip_remaining is set, skipping unprocessed points continue; } // Based on the difference between ppos and ppp, move P to the position // pointed to by ppp and maintain ppos if ((*P)[0] == 0) { // means it has been processed continue; } #pragma omp atomic nr_points_matched++; #pragma omp atomic nr_points_done_in_this_round++; // process the object pointed to by P (*P)[0] = 0; } } // this break never gets executed if (nr_points_matched == nr_to_match) break; } } The gist of this code is: for a given LatticePoints list, the program processes each point in parallel. While processing each data point, some points may be temporarily skipped, requiring repeated passes until all data points have been processed. In this code, nr_to_match is the total number of data points, nr_points_matched is the number of points already processed, and nr_points_done_in_this_round is the number of points processed this round. The loop's termination condition is nr_points_matched equaling nr_to_match, i.e. all data points processed. The direct cause of the infinite loop is that nr_points_matched never reaches nr_to_match, so the loop cannot terminate. Using gdb, one can find that when this happens, every point in the entire LatticePoints list has been marked as processed, so nr_points_matched stops increasing, yet the loop's exit condition is never satisfied, so it just keeps looping. The question then becomes: why does the value of the counter nr_points_matched not match the actual number of processed data points. According to the code, the per-round increment of nr_points_matched should equal that of nr_points_done_in_this_round, because they are always atomically incremented together. But the actual output was not so: the two counters' values differ slightly, and the gap is unstable, with the result varying from run to run. The first thing ruled out was a memory-ordering issue: this code does not rely on atomic variables to synchronize other variables; in other words, it operates on and reads only the atomic variables themselves the whole time, so from the code's perspective it is logically correct. The next suspicion was whether the OpenMP implementation was at fault: whether the atomic operations annotated with #pragma omp atomic really guarantee atomicity. From the disassembly, one can see the compiler generated the LoongArch64 amadd.d instruction for these atomic operations, as expected. To investigate this, we set up two additional std::atomic counters as controls, used alongside the original two, to see whether the results agreed. It turned out that the four counters' values (computed from the per-round increments) should have agreed, but in fact they showed random discrepancies. This hinted that the atomic add instruction loses updates under certain conditions. However, testing the atomicity of the atomic add instruction with a simple atomic add program could not reproduce the lost update. To find a minimal reproducer, we simplified the aforementioned normaliz processing logic into a similar test program, which also could not reproduce the problem. So we had to keep commenting out computation steps in normaliz's actually-running code, trying to find the minimal condition that triggers the lost atomic add. One bizarre phenomenon was that even after commenting out most of the computation steps, the problem persisted. Because the program was too complex, in the end we still could not find a minimal code snippet that reliably reproduced the lost atomic add. The Second Round of Investigation¶ Six months later, the problem remained unsolved. With the disclosure of the LoongLeak/LoongBleed vulnerabilities, the lost atomic add in normaliz came back into our view. This time, we tried to use AI to assist the investigation. The method was: first point out to the AI that the above normaliz code has an infinite-loop problem, ask the AI to confirm and reproduce it, and then find the possible cause. In the first round of conversation, the AI noticed the problematic loop but did not conclude that the atomic add instruction was at fault. After that, we hinted to the AI that the problem exists only on LoongArch and not on other architectures, but the AI still could not give a definite conclusion. Finally, we directly told the AI the fact that we had already localized the problem to the atomic add, and asked it to reproduce it and provide a minimal reproducer. In that round of conversation, the AI eventually turned its attention to a memcpy call in the processing function, which was exactly the part overlooked in the first round: memcpy's implementation lives in glibc, and glibc chooses the optimal implementation based on currently available hardware features; if the hardware supports a vector instruction set (LSX/LASX on LoongArch), glibc's memcpy will use the corresponding vector instructions to accelerate memory copying. And it was precisely these vectorized memory copies that triggered the lost atomic add on LoongArch64. Two days later, the AI produced a minimal program that reliably reproduces the problem. Expanding the Scope¶ After discovering that the atomic add instruction can lose updates, we had new questions: first, is only atomic add affected, or do other atomic instructions have the same problem; second, do other memory operations also trigger similar problems. For the first question, we first investigated the CAS instruction, because it can be used to implement atomic addition, making it easy to tell whether something goes wrong. It turned out that CAS also has the problem under the same conditions. For other atomic instructions, such as atomic swap, atomic max, atomic min, atomic bitwise AND, atomic bitwise OR, and so on, since even a lost update is not easy to detect from the result, we did not verify them at first. For example, if a lost update happens during an atomic max, then as long as the operation that updated the maximum was not lost, the result is correct. After much thought, we finally found a verification scheme: to check whether such instructions lose updates, we recorded the result of every operation and verified afterward. Take atomic max as an example: if you atomically take the max over the numbers 1 to n in parallel, the final result should be n. Each atomic max modifies the memory and also returns the old maximum. For an operation with input k, if the returned old value is less than k, then this operation updated the maximum. Atomicity guarantees that the return values of all operations that updated the maximum will not repeat. If a repeat occurs, then a lost update of the atomic instruction occurred. Verification showed that these atomic instructions all lose updates under the same conditions.