Tolerating full cloud outages with Monzo Stand-in

Our customers reasonably expect to be able to spend on their card, make bank transfers and pay their bills 24 hours a day, 365 days a year. Their lives don’t have downtime for maintenance so nor should we. We dedicate a lot of our engineering effort to minimise the risk of downtime during technical migrations and other day-to-day operations, but unforeseen incidents that cause unexpected outages are impossible to eliminate entirely.We take reliability seriously at Monzo so we built a completely separate backup banking infrastructure called Monzo Stand-in to add another layer of defence so customers can continue to use important services provided by us. We consider Monzo Stand-in to be a backup of last resort, not our primary mechanism of providing a reliable service to our customers, by providing us with an extra line of defence.Monzo Stand-in ArchitectureMonzo Stand-in is an independent set of systems that run on Google Cloud Platform (GCP) and is able to take over from our Primary Platform, which runs in Amazon Web Services (AWS), in the event of a major incident. It supports the most important features of Monzo like spending on cards, withdrawing cash, sending and receiving bank transfers, checking account balances and transactions, and freezing or unfreezing cards.Our Primary and Stand-in Platforms run independently to one another, each consisting of Kubernetes clusters running a unique set of services on top of typical platform components such as a database, queueing systems and locking mechanisms. The services running in each platform are unique in the sense that services in the Stand-in Platform don’t ever run in the Primary Platform, or vice-versa, even for behaviours that are common across both like processing a card payment.Each platform is able to make their own decisions about approving or declining transactions and can establish their own connections to payment networks via multiple physical data centres.Monzo Stand-in also runs a limited set of API endpoints tailored to serve limited functionality while we’re using Stand-in. The Monzo App checks whether Monzo Stand-in is enabled periodically in the background, and if it is, it flips to a simplified UI that supports our most important features.Different systems help to mitigate riskIt might seem strange to build brand new services from the ground up for Monzo Stand-in rather than deploy the same services we run on the Primary Platform, but there are a number of motivations for us taking this approach.If we tried to run the exact same set of services we would need to replicate all of our data between the two platforms. To do this well we’d have to maintain strong consistency of our data. This would mean that writes to our database would be considered successful only if the data is written to both platforms. If either platform became unavailable we’d be unable to write anything without sacrificing consistency, reducing our overall availability rather than improving it.Instead of maintaining strong data consistency we accept that the replication is non-blocking and that it is eventually consistent, but systems like our ledger wouldn’t be able to tolerate eventually consistent data.Different software reduces the chance of suffering same failureOur Primary Platform operates across AWS availability zones and it runs multiple replicas of all our services. We design systems to be scalable and we gracefully degrade when non-critical dependencies error. Even with resiliency baked into our Primary Platform’s design, complex systems such as these can fail in surprising ways.There are a large number of possible reasons our Primary Platform could fail. While it’s easy to consider that the risk we want to mitigate is a cloud provider outage, it’s at least as likely that a bug in our code or processes is the cause for an outage.Traditional Disaster Recovery systems predominantly consider hardware failure, assuming the most likely risk to their platform is a network outage or a disk failure. Cloud platform providers like AWS, GCP, Azure and others have for the most part solved for outages caused by hardware failure, but disaster recovery hasn’t really evolved. Today it doesn’t matter how many data centres you have if you run the same software in them all.The greater the independence between the Stand-in environment and the Primary Platform, the smaller the risk that the same issue will impact the Stand-in Platform. In our case, our Primary and Stand-in Platforms run their own independent card issuer processing code, each capable of authorizing transactions. Whilst the platforms are expected to behave in similar ways we implement them separately and aim to minimise the reliance on shared code as much as possible.A minimal system doesn’t cost the worldWe monitor the costs of running our platforms very closely. Monzo Stand-in costs around 1% of the cost of our Primary Platform to keep running in the background, and we would only expect this to increase marginally if we enabled it during a major incident. If instead we wanted to run all of the same systems and replicate all the data then the cost in terms of compute capacity and people needed to maintain it would be much larger, potentially doubling our total platform costs.Syncing data between platformsMonzo Stand-in only contains the minimal state required to support the small number of features it supports. This includes things like balance and limited historic transaction information, enough details about cards and accounts to process payments, and a list of other things like pots and payees. Whenever any of this data changes in the Primary Platform our Stand-in Data Syncer writes those updates to the Stand-in Platform. All processing outcomes, state transitions, and other effects created in the Primary Platform are published into our event system, and the Stand-in Data Syncer consumes a subset of these events to trigger the process of updating state in the Stand-in Platform.All data we sync from the Primary to the Stand-in Platform is treated as immutable. We don’t expect the Stand-in Platform to be running on a perfectly consistent view of the world but in practice the view is extremely close to perfect consistency due to the real-time nature of the syncing. We monitor the lag of this eventually consistent syncing process very closely and alert in rare cases the lag exceeds our appetite.Our tokenized data (e.g. card PANs that we encrypt) follows a similar but slightly different flow for syncing, where we exchange data encrypted for different keysets (labelled A and B in the diagram) between tokenization systems in each platform.Syncing state from Monzo Stand-inWhen Monzo Stand-in is enabled it produces similar processing decisions and state transitions as we do in the Primary Platform, but since this isn’t our Primary Platform, those outcomes are only authoritative for the duration that we’re operating in Stand-in.We store new state in Monzo Stand-in separately to immutable data synced from the Primary Platform, and we record a log of all effects in a durable queue for the Primary Platform to consume when it’s able to do so. If the Primary Platform is only partially unavailable it may be able to consume this queue immediately or it may only be able to do so at a future date if we’re suffering a complete outage.The records of effects in this log are referred to as Monzo Advices, as they each advise our Primary Platform of an effect that was created, for example a card payment that was approved, and we expect Primary Platform to apply the effects of these Advices verbatim.The Primary Platform is our system of record, maintaining our records of truth, and at no point while Monzo Stand-in is enabled does it assume the role of our system of record. This implies that by applying Monzo Advices verbatim from a potentially inconsistent view of a customer’s balance in Monzo Stand-in we may have approved a transaction where the Primary Platform believes the customer had insufficient funds, and in this case would take that customer into an unapproved overdraft. We operate a number of controls in our Primary Platform to help with this possibility but in practice it’s extremely unlikely to occur.Correlating data held by both platformsWhen we apply a Monzo Advice from the Stand-in Platform to the Primary Platform we expect downstream systems to create an equivalent state. For example an Advice for a card payment in Stand-in will move money in the Ledger in the Primary Platform.This creates a small problem for us. We now have state in both the Stand-in and Primary Platforms that represents the same payment, and when money is moved in the Ledger in the Primary Platform a Transaction is also synced to Stand-in by our Data Syncer as we described earlier.To avoid double-counting Transactions and other effects in Stand-in we generate a Correlation ID (labelled A in the diagram) for StandinTransactions and SyncedTransactions, and merge our view correlated transactions when we use them in the Stand-in Platform.Enabling Monzo Stand-inWe’ve mentioned our ability to enable Monzo Stand-in so we want to explain what we mean. We run a Stand-in Configuration service in the Primary Platform and another in the Stand-in Platform, that for the most part are API-compatible that coordinate which platform is enabled, which components of the Stand-in Platform are enabled, and which users they’re enabled for. When we detect an outage in a service we deem critically important to our customers, we can enable the corresponding parts of Monzo Stand-in with this configuration system.Currently we update the configuration using our Stand-in Platform CLI tooling, but you can see that with further work we can fully automate this process by triggering the configuration system on the same heuristics our engineers use manually.This system works even if the Primary Platform is completely unavailable, and as mentioned before, the Monzo App checks both platforms