A Couple Million Lines of Haskell: Production Engineering at Mercury | The Haskell Programming Language's blog
We believe that this document is a mix of AI-generated, and human-written content.
Hacker News Article AI Analysis
Content Label
Mixed
AI Generated
38%
Human
62%
Window 1 - Human
A Couple Million Lines of Haskell: Production Engineering at Mercury Ian Duncan March 30, 2026 [Haskellers from the trenches] #Production #Mercury The editors of the Haskell Blog are happy to announce a new series of articles called "Haskellers from the trenches", where we invite experienced engineers to talk about their subjects of expertise, best practices, and production tales. Engineering rigour and artistic creativity are a fantastic combination, and this series aims to be the synthesis of these two aspects within the Haskell world. I first heard about Haskell when I was sixteen, sitting in a high school computer science class where we were writing Java and learning, among other things, that NullPointerException was apparently a lifestyle choice if you decided to go into software development. While looking at the /r/programming subreddit after school, I stumbled across a reference to a language where null pointer exceptions simply could not happen, where the type system could prevent an entire category of bugs that I had been fighting with every week. Haskell. I was immediately, hopelessly enamored with the idea. I have been writing Haskell for nearly two decades now, and I still think the value proposition I fell in love with at sixteen was basically right. What took me longer to learn is what that promise looks like after a codebase gets large, the company grows faster than its documentation, and the system is allowed to touch money. Haskell earns its keep there in numerous, sturdy ways. It lets you pack operational knowledge into APIs, put dangerous machinery behind tight boundaries, and make the safe path the easy one. At a growing company, those aren't just matters of taste; they are how you keep a system understandable after the people who first understood it have moved on. Fast forward to today: I work at Mercury, a fintech company that provides banking services.* We serve over 300,000 businesses. We processed $248 billion in transaction volume in 2025 on $650 million in annualized revenue, and are, at the time of writing, in the process of obtaining a national bank charter in the USA from the OCC. We have around 1,500 employees. Our engineering organization largely hires generalists, and most of them have never written a line of Haskell before joining. My time working at Mercury has changed how I think about the language more than any sermon about purity ever did.
Window 2 - Human
Elegance is pleasant, but keeping your business alive is compulsory. Our codebase is roughly 2 million lines of Haskell, once you strip out comments and such. This is the part where you are supposed to recoil in horror. A couple million lines of Haskell, maintained by people who learned the language on the job, at a company that moves huge amounts of money? The conventional wisdom says this should be a disaster, but surprisingly, it isn't. The system we've built has worked well for years, through hypergrowth, through the SVB crisis that sent $2 billion in new deposits our way in five days,1 through regulatory examinations, through all the ordinary and extraordinary things that happen to a financial system at scale. This article is about why it works. Not in the "Haskell is beautiful" sense, though it is. Not in the "the compiler will save us from ourselves" sense, though I frequently feel gratitude in that direction. I mean in the much less romantic and much more useful sense that we run this language in production, at scale, with a rapidly changing team, and have learned some hard lessons about what it takes to keep the whole enterprise afloat. The beauty of Haskell is charming enough, but there is a whole swath of operational and organizational reality beyond it, and if you ignore that reality for too long, your company will likely fire the whole Haskell team2 and start writing PHP or something instead. How We Think About Reliability Before diving into practical advice, a note on philosophy. There is a traditional way of thinking about system reliability that focuses on preventing failures. You enumerate the things that can go wrong. You add checks. You write tests for each bad case. You hunt for bugs. This is, of course, necessary work, and we do it. But it is not sufficient, and if you orient entirely around it you develop a specific blind spot: you get very good at cataloguing the ways things break and very bad at understanding why they ordinarily work.3 We try to think about it differently. A system operates reliably because it can absorb variation: it degrades gracefully, its operators can understand and adjust it, and the architecture makes the right thing easy and the wrong thing difficult.4 Reliability is not just the absence of failure. It is the presence of adaptive capacity. It is a system's ability to keep functioning while reality continues its longstanding and regrettable habit of refusing to hold still.
Window 3 - Human
When you have hundreds of engineers working in a multi-million-line codebase, many of whom are six months into their Haskell careers, "adaptive capacity" stops being a nifty phrase from a resilience engineering paper and starts being a daily concern. Patrick McKenzie has observed that in a company growing at 2x per year, half of your coworkers will always have less than a year of experience. A year later, half of your coworkers will still have less than a year of experience. For very successful companies, this never stops being true.5 You become organizationally ancient very quickly, whether you like it or not, and the things you know become institutional dark matter: load-bearing, but invisible to most of the people around you. So the questions we ask are operational. Can the new hire on your team read this module and understand what it does? If the database is slow, does this service degrade or does it fall over and take its neighbors with it? If someone misuses an interface, does the compiler tell them, or do we find out when the on-call gets paged? If you don't have answers to those questions, you have a future incident quietly unfolding. This is why I increasingly think of the type system as an operational aid more than a correctness proof. Its value is not merely that it rules out certain classes of errors, though it does. Its value is that it encodes institutional knowledge in a form that survives the departure of the person who wrote it. In a fast-growing company, people leave, people transfer teams, people go on vacation or parental leave, people join, and the churn means that things people knew walk out the door with them unless you have written them down somewhere. Ideally, you have written them down in a form that the compiler can read, because the compiler is much more disciplined than the average wiki page. This extends beyond code. As a member of our stability engineering team, we constantly investigate the prospective production behavior of features and products. We do not do this to slow down product development, but in partnership with the team shipping the feature, to make sure we are prepared to deal with the fallout when it breaks and, if possible, to make that fallout boring rather than exciting. We ask things like: what is the blast radius if this fails? Which operations must be idempotent, and how? What does the rollback look like? What happens to in-flight work?
Window 4 - Human
Which systems will absorb the failure, and which ones will amplify it? The point is to have the conversation early enough that it shapes the design rather than merely auditing the launch after all the important decisions have already become expensive to revisit.6 Our philosophy, stated plainly: we are not the quality police. We are the people who would like to help you avoid being woken up at 4 AM to deal with the fallout of a broken feature. Rather than a deeply ideological stance, we simply desire to help people. So, in light of that, how do we make Haskell work in production? Purity Is a Boundary, Not a Property My hot take: the first and most consequential misunderstanding about Haskell is that purity is not something the language is, so much as that it is something your interfaces enforce. Under the hood, Haskell is not a magical machine that performs side effects despite being pure. Behind every "pure" function in bytestring, text, and vector lies a cheerful little hellscape of mutable allocation, buffer writes, unsafe coercions, and other behavior that would alarm you if you discovered it in a junior engineer's side project. Behind the ST monad lies in-place mutation and side effects, observable within the computation. What makes it acceptable is that the side effects are encapsulated such that the boundary cannot be violated. runST :: (forall s. ST s a) -> a The rank-2 type (that is, the type s is scoped within the parenthesis and can't escape) of runST ensures that the mutable references created inside the computation cannot escape due to being tagged with the type s. Internally, all sorts of imperative nonsense may occur. Externally, the function is pure. The world outside the boundary gets none of the mutation, only the result. This is, I think, a wonderful design principle in the larger scheme of things when writ large: you can permit arbitrarily dangerous operations within a scope, provided the scope's exit is typed narrowly enough that the danger cannot leak. That principle applies everywhere in production. Your database layer uses connection pooling, retry logic, and mutable state internally. Your cache uses concurrent mutable maps. Your HTTP client probably has circuit breakers, pooled connections, and a small municipal government's worth of bookkeeping. None of this is a problem if the interface is tight enough to prevent misuse and the boundary holds.
Window 5 - 100% AI-Generated
In production, the goal is often not to avoid mutation entirely, because that is not a serious proposition for most real systems. The goal is to contain mutation, make the containment legible, and verify that it stays contained. Often the right question is not "is this pure?" but "where is the impurity, and how much of the codebase is allowed to know about it?" For a new engineer who learned Haskell three months ago, "purity is a boundary you try to maintain" is much more useful than "Haskell is pure." One tells them what to do when they sit down to design a module. The other mostly sits there looking profound. This boundary-oriented view of purity sets up a more general pattern that recurs throughout production engineering in Haskell: dangerous things are tolerable when they are fenced in, carefully exposed, and hard to misuse. That is true of mutation. It is true of retries, transactions, state machines, distributed workflows, and type-level machinery. Much of what follows is really just this same idea, wearing different hats. Make the Right Thing Easy There is a pattern in large codebases where correctness depends on performing operations in a particular order, or including a particular step that has no visible connection to the main work. "Remember to flush the audit log after every transaction." "Always check the feature flag before calling this endpoint." "Make sure to enqueue the notification inside the database transaction, not after it." These are the incantations of operational lore. They live in wiki pages, onboarding documents, half-forgotten design reviews, and the memories of senior engineers who are now three teams away and booked solid until Thursday. In a company that is hiring aggressively, the half-life of tribal knowledge is alarmingly short. When an engineer leaves, the incantations fade. When a deadline approaches, they are the first thing skipped. When a new engineer joins, they often have no way to know the incantation exists at all. Nothing says "robust system design" quite like a critical invariant living in a Slack thread from nine months ago7. Haskell gives you tools to encode these incantations in types so they cannot be forgotten. This is, for my money, the single most valuable thing the language offers a production engineering organization.