Kubernetes on Oxide: How Customer Needs Shaped Our Integrations | Oxide Computer Company

In late 2024, customers and prospects were eager to run Kubernetes on Oxide, but we had no supported integrations to help them do it.Kubernetes and Oxide are a natural fit. Kubernetes defines the infrastructure behavior it expects through standard extension points, while Oxide exposes the primitives needed to implement that behavior through APIs. The foundation for integration was there. What was missing was the software and an understanding of which integrations customers actually needed.That was the situation when I joined Oxide as its first Solutions Software Engineer,[1] focused on building software to solve customer problems. My first assignment was to make it easier to deploy and operate Kubernetes on Oxide.In my first week, I was handed two resources to help me get started:A customer-submitted pull request for a Rancher node driverAn early draft of RFD 493 Initial Kubernetes IntegrationsWhat began with those two resources grew into a team effort shaped by a feedback loop.

Our immediate goal was to unblock the customer who had submitted the Rancher node driver pull request. Working through their use case would also give us firsthand experience creating Kubernetes clusters on Oxide and help us uncover the next problems to solve.No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations.Rancher Node DriverBefore we could maintain the customer-submitted integration, we needed to understand the workflow it supported. I had never used Rancher or worked with a node driver, so reviewing the contribution meant learning both.A Rancher node driver is an executable plugin that teaches Rancher how to create and manage virtual machines on a particular infrastructure platform. The Oxide Rancher node driver translates those operations into Oxide API requests. Once installed in Rancher, it lets customers provision Oxide instances as nodes in Rancher-managed Kubernetes clusters.Testing confirmed that the customer’s implementation worked.

Oxide uses a FAT12 filesystem for cloud-init user-data, not ISO 9660, but Talos’s filesystem probe only attempted to read an ISO 9660 superblock from the NoCloud configuration disk.

Cluster API offered something our first two integrations did not—an upstream, provider-extensible API for managing clusters without requiring a third-party platform like Rancher or Omni.CAPI lets operators declaratively create, scale, upgrade, and delete Kubernetes clusters through Kubernetes custom resources.

Without that reconciliation, a cluster could not reliably determine whether an unreachable Kubernetes node was temporarily unavailable or whether its backing Oxide instance had been deleted.We needed a component that ran in each cluster, spoke to the Oxide API, and continuously reconciled Oxide infrastructure with Kubernetes state. Kubernetes provides a standard extension point for this purpose: the cloud controller manager (CCM). A CCM lets infrastructure-specific controllers integrate Kubernetes resources with an infrastructure provider’s API without adding provider-specific code to Kubernetes itself.We built the Oxide cloud controller manager to connect Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their backing Oxide instances, recording details such as instance IDs and network addresses, and reporting whether each instance is running, shut down, or no longer exists. Kubernetes uses this information to initialize nodes and safely remove them when their backing instances are deleted.The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver, the Omni infrastructure provider, and CAPOx. Instead, it provides a runtime integration shared across those provisioning workflows.Importantly, building the CCM gave us a durable extension point inside each cluster. As Oxide evolves, we can add new infrastructure-aware controllers to the CCM rather than update every provisioning integration.With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications.

The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem.How do I use LoadBalancer services?One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer. When a user creates one, Kubernetes asks the cloud provider’s service controller to provision the necessary infrastructure and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer.Oxide did, however, have floating IPs.

Floating IPs are addresses from a rack’s external IP pools that can be attached to and detached from instances, making those instances reachable from outside their VPCs. Using floating IPs offered a way to unblock LoadBalancer services.