301 - Has anybody seen my keys?: A key-hierarchy strategy for rack-level security | RFD | Oxide

Background and PurposeThere are many different types of secrets inside an oxide rack. At the base of the system we have the DeviceId and Alias keys stored on the RoT and used for platform identity and measurement signing for attestation respectively [RFD 36]. These keys along with a 3rd RoT hosted keypair used for authenticating ephemeral Diffie-Hellman agreement provide the ability for sleds to form secure sprockets sessions for application layer messages [RFD 238]. These sprockets sessions provide each sled the ability to confidentially share information in a point- to-point fashion, where the integrity of messages is protected, the authenticity of the endpoint and the attestation of its running software is guaranteed.In order to provide the rack-level security guarantees such that an attacker cannot walk off with a subset of sleds or drives and recover any useful information, we have designed a Trust Quorum [RFD 238].The primary protection mechanism behind the trust quorum is shamir secret sharing. A shared rack-level secret which is used as a key-derivation source for other keys used to protect storage at rest. The rack secret is split into N unique key shares by a dealer process and distributed over sprockets sessions to each bootstrap agent along with the unique platform identities of the N trust quorum members. These platform identities are baked into the public key certificates on the RoT to allow verification that an entity is who they say they are. After distribution of this information, bootstrap agents can establish sprockets connections to other bootstrap agents, verify membership in the group (via the exchanged certs), and retrieve K-1 shares from other agents such that they can reconstruct the rack secret from K shares. Importantly, without obtaining K shares, no information about the rack secret can be learned.As described in [RFD 238], individual shares are store unencrypted on each sled’s M.2 drives. An attacker would have to steal at least K of these drives to reconstruct the rack secret, which is infeasble without significant time and disruption during physical access. In the future we plan to "seal" these secrets with the help of our root of trust (RoT) such that they are only decrypted on sled boot. Sealing would mean that an attacker would have to steal K whole sleds and be able to boot them to recover the rack secret. The weight of K sleds makes this prohibitive for a casual attacker.With this rack secret we have a shared secret that is only available when enough of the N sleds in the group are plugged into the same rack and trust each other enough to distribute shares to each other. From this shared secret, we can do things like derive or wrap individual encryption keys for each individual U.2 device, and independently derive rack level root certificates for internal services.The purpose of this RFD is to identify:What data is protected by the rack secret?What is the temporal lifecycle of that data?What is the spatial locality of that data?Is it local to a specific drive/sled/rack/cluster?What physical or logical restrictions prevent the encrypted data from being moved?What is the key-hierarchy used to protect that dataWhat keys protect what data?Which keys are derived from which secrets and keys?Which keys are wrapped (encrypted by other keys)?How are keys derived or wrapped (Key Schedule)?What happens in the case of key compromise?These questions are critical to the security of our rack, and our determinations may evolve given our experiences of with the current solution and more time to think of and implement solutions necessary to harden and expand our posture. For now, though, this RFD must deliver strong enough answers to allow us to best protect our data at rest and move onto our other security goals in the near future.Leaf PeepingOur rack-level key-hierarchy starts at the root with the Rack Secret. Each member of the trust quorum can learn enough shares to recompute the rack secret. The rack secret itself is not a key, and so we must derive keys from it via a key derivation function (KDF) that can be used for vaious purposes required by our rack. What keys we derive, and what keys are derived from those keys, and what keys any of them may wrap, is hard to determine without more information about what data is actually being protected by keys in our system. To help construct our key hierarchy, and fill in the internal nodes in the tree, we will start by listing out the data to be protected within the rack, what unique keys will be used to protect that data, and the security guarantees we would like from those keys. After establishing the leaf-keys we can work backwards to fill in the rest of the key hierarchy.The below table shows the data we care about protecting by the rack secret key hierarchy for now. There are likely other things that need protecting, but they will almost certainly have similar requirements. And while we currently store Crucible encryption keys in CockroachDB, we anticipate having some sort of key management system in place post MVP that will house things like encryption keys, certs, and authentication tokens. We also do not currently have the internal certificate authority mechanism defined for managing how control plane services communicate over mutual TLS. We do know, however, that we will want such a system and therefore are being explicit here that we will at least have a root certificate and leaf certificates to protect. Whether or not we need intermediate certs is TBD.DataStorage TypeStorage DeviceLifetimeControl Plane DataCockroachDBSome U.2 devicesTemporary until lifetime of rackControl Plane MetricsClickhouseSome U.2 devices?Crucible ExtentsFilesMost U.2 devicesLifetime of a storage volumeCrucible volume encryption keysCockroachDBSome U.2 devicesLifetime of a storage volumeUser Authentication TokensCockroachDBSome U.2 devicesTemporary / user-session TTLInternal Service Leaf CertsFilesAll U.2 devices - although not all certs on all devicesCert TTLInternal CA Root Cert Private KeyFileAll sleds?Long enough to at least sign Leaf Cert CSRs, and then may be rederived for rotation of leaf certsInternal CA Root Cert Public KeyFileOne or more U.2 devices on all sledsUntil Cert rotationWe can see that all stored data resides on U.2 devices, either in files or databases, that end up in files. We know based on prior discussions that we do not want to use hardware backed full disk encryption (due to lack of trust in implementation, complexity of key management, and multiple possible vendors), and we also know that we will have one zpool per U.2 device. We want to encrypt almost the entire zpool via a root /crypt dataset using ZFS encryption, opting out specific datasets like crucible that provide their own encryption. We can therefore assume that all necessary customer and control plane data on each U.2. device is encrypted. To limit key compromise, such that if one disk is stolen and has its key compromised, not all other disks are compromised, we choose to use an individual key per disk. We know that each disk contains a lot of data, and that we do not want to re-encrypt that data when changing keys. Lucikly for us, ZFS allows us flexibility in how we generate and change these keys.Key rotation is managed internally by the ZFS kernel module and changing the user’s key does not require re-encrypting the entire dataset.— zfs(8) manpageAs we are encrypting all critical data on the U.2s using ZFS encryption (including crucible keys), and those ZFS keys are not stored on those disks, we cover our data at rest protection requirements. An attacker who cannot reconstruct the rack secret can not steal a subset of drives or sleds and retrieve any useful information. We now must figure out whether we want to derive or wrap those keys, and what the rest of the hierarchy looks like.Building a Key Hierarchy from a Rack SecretAssuming a static cluster, we now have a rack secret that can be computed from K key shares and used to derive child keys. In the following example, K = 2, just so we can keep the diagrams small.The rack secret protects every key in the system that is used after rack unlock. There are two mechanisms to get the next keys in the hierachy:Key DerivationKey Wrapping (Key Encryption)Key Derivation we already use to derive any primary child keys from the rack secret. But it can also be used to derive keys from keys. Key wrapping is just taking a key and encrypting it. Each has their benefits and drawbacks.Key derivation is nice because you never have to store the derived keys on disk. You can just regenerate them. The problem with it is that if the derived key changes any downstream derived keys will now change also change.Key wrapping is nice because if the wrapper key is rotated, the downstream keys do not have to change. This is particularly useful for encrypting large amounts of data on storage. You don’t want to be forced to decrypt and then re-encrypt just because a parent key was rotated. The downside of key wrapping is that you now have to store the wrapped (encrypted) key on disk somewhere. If the same key is used in multiple places you have to replicate it.For trust quorum, we have the particular problem that when nodes are added or removed we generate new shares. While it’s possible to maintain the same rack secret and key derivation with new shares, it is less secure over time because any single malicious node who retrieved the rack secret at one time and saved it can now recover any data on any existing drives or any data produced in the future. While this is always an issue, we prefer to allow the rotation of the rack secret in the case of a known compromise. We therefore always generate a new rack secret on every change of the trust quorum (reconfiguration), or other key-share rotation. This way, even if all existing data on the rack is compromised, at least no new new data will be compromised if the compromised