The fragile foundations of CoT monitoring

Credit: Art Potts Last week, I attended a workshop on Chain of Thought (i.e., reasoning chain) monitorability. Many of the most influential researchers and “members of technical staff” working on this topic were present. I felt lucky to be there, since pretty much everything I know about CoT monitoring I learned from Peter Hase. I have, however, worked extensively on interpretability, and there seems to be a general hope that interpretability will improve or complement CoT monitoring, though this relationship is complicated, as I discuss below. I learned a lot at the workshop and came away with many things to think about. This post seeks to summarize and systematize the main takeaways for me as a sort of outsider to the community: CoT and monitoring: An uneasy and uneven alliance Could we ensure that all computation is verbalized? On the threat of deceptive CoT Denial of knowledge is not a path to safer models Could CoT monitoring have detected the OpenAI attack on Hugging Face? Conclusions CoT and monitoring: An uneasy and uneven alliance The general consensus of the workshop group seemed to be that it is a stroke of luck that CoT provides a basis for monitoring, because it wasn’t invented for that purpose. CoT reasoning was developed primarily to make models better at hard tasks. This hypothesis traces to the classic paper introducing CoT as a prompting technique: Wei et al. 2022. The usual story is that present-day LLMs have a fixed depth, and so their capacity to perform computations before producing an output is highly limited. By allowing models to produce tokens before their final answer, we enrich them with a more open-ended form of computation. For Wei et al., CoT is purely a prompting technique, but it was elevated to a post-training process with the release of OpenAI’s o1 model in September 2024. Though Wei et al. focus on performance improvements, they note that CoT “provides an interpretable window into the behavior of the model”, with the added caveat that “fully characterizing a model’s computations that support an answer remains an open question”. This became a full program for monitoring in the agenda paper Korbak, Balesni, et al. 2025, Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. (For discussion of this paper’s origins and predecessors, see this post by Peter Hase and me.) The phrase “fragile opportunity” from Korbak et al.’s title seems to capture the spirit of their proposal. CoT has stood the test of time because it leads to capability improvements, not because it increases transparency. Furthermore, the two goals are not necessarily aligned. Some recent papers (Kirchner, Chen, et al. 2024; Guo et al. 2025) have even suggested that transparency can be a tax on the effectiveness of CoTs for models. One does not have to be especially cynical about the business of AI to conclude that, if push comes to shove, progress on capabilities will win out over increased transparency. On balance, I agree that it is useful to be able to monitor CoT. However, the fragility of this alliance makes me feel that we should reduce our dependence on CoT for safety, to plan for a future in which CoT isn’t useful. The next two sections of this post provide additional motivation for this conclusion. Could we ensure that all computation is verbalized? Another prominent position at the workshop is related to the above but interestingly distinct from it. This is the idea that it would be useful to prevent models from doing any computation that is not reflected in their CoTs. No one expects this ideal to be completely achievable, but some people seem to like the idea of striving towards it in order to improve safety. As I understand it, the argument goes like this: suppose a model receives an input X, performs an internal computation C, and arrives at its final answer A before generating any tokens, so that its first generated tokens express A. The computation C is hidden from us, and this might make us feel uncertain. For example, even if A is correct, it could be based on faulty reasoning that would make us nervous about how the model will perform on other related inputs X'. One response would be to have the model produce a CoT anyway, as a kind of just-so story about its final answer. I received diverse reactions to this idea from participants. In general, people seemed to think that it would be incorrect or misleading: CoT is supposed to be causally involved in the model producing the final answer. In my scenario, it is not, by stipulation, and so CoT should be null. To my mind, this reflects an allegiance to the original motivation for CoT as capabilities-oriented. I also found this objection hard to square with the fact that the group seemed more interested in benchmark performance than in working out the details of a causal theory of how reasoning affects outputs. The existing literature suggests that this theory will be non-trivial to work out (see especially Yu et al. 2026; Gur-Arieh et al. 2026), and so we absolutely can’t take it for granted. What about a variant of the above idea: have the model produce an explanation after answering (Chen et al. 2023)? Here, the explanation is not presented as a reasoning chain or plan, and so it does not imply a causal story about its role in the model’s response. There seemed to be relatively little enthusiasm for this idea, but I am not sure precisely why. I myself would be concerned about the veracity of these post-hoc rationalizations (a deep concern given what we know about humans; Nisbett and Wilson 1977). However, others seemed more concerned about how they would adapt their existing monitors for this new scenario than about any foundational issues with the idea. So: if we require causal links between the monitored text and the answer, and we think text is the only basis for monitoring, then we are pushed towards the proposal that models should do all their thinking in CoT. All computations verbalized! Some of the participants even advocated for a ban on recurrent architectures, since those threaten to eliminate the need for reasoning by creating an open-ended form of computation that generates no tokens. My vote would be to develop better tools for monitoring internal states instead. Interpretability research provides a starting point. Moving in this direction seems unavoidable. We agree that the ideal of having all computation reflected in tokens is not achievable: the forward pass that processes the input is a rich computation, and reasoning itself is a bunch of forward passes each performing rich internal computations that could far surpass what we can read out of the tokens. Thus, we will have to figure out how to deeply understand the internal representations and computations of these models, else we will inevitably miss something. (I should say that I feel a bit conflicted about this conclusion. It is perhaps unsurprising that an interp researcher has reached the conclusion that only interp can save the day. In my defense: I don’t think it will be easy, and I don’t even have much confidence that the existing interp toolkit will suffice. It just seems clear that internal states will provide a more reliable and comprehensive picture.) On the threat of deceptive CoT This push to have models verbalize everything depends on an assumption that the texts produced during reasoning are an accurate and informative accounting of what the model did. This is an extremely shaky assumption. We know from recent work (e.g., Turpin et al. 2023; Baker, Huizinga, et al. 2025; Chen et al. 2025) that models can produce reasoning that does not reflect the true causes of their outputs. Everyone at the workshop was keenly aware of these results. However, strikingly, they seemed to be not especially worried because, in practice, present-day models aren’t good at being deceitful in this way. This reply is very surprising to me; we all agree that progress in AI has been fast and disruptive, and that it will continue to be so. If anything, the workshop participants were, on average, more optimistic about (and hence more worried about) progress than I am. So their appeal to current conveniences and prevailing good fortunes seems inconsistent to me. Perhaps it is a consequence of the fact that many of them are on the front-lines of dealing with the frontier models deployed right now, which must be stressful and uncertain enough without worrying about future threats as well. Just to crystalize the precise issue for CoT monitoring: the most concerning situation is one in which a model performs a computation C that we regard as problematic, but C is not reflected at all in the reasoning. This could be because (1) it performed C before reasoning (see the verbalization section above), or (2) because it is performing C during reasoning but not producing tokens that transparently describe C. For scenario (1), we need methods that can assess the internal states, as I discussed above. For scenario (2), it seems like token-level analyses could be illuminating. The relevant evidence might be very subtle, though, akin to the kind of signals that identify a model’s provenance (Kuditipudi, Huang, Zhu et al. 2025) or drive effects like subliminal learning (Cloud, Le, et al. 2025).