Bare Metal Kubernetes Deployment

Kubernetes has revolutionized software development by leveraging containerization to provide easy resource allocation, scalability, and high availability. Cloud providers like Azure and AWS have extended that ease of use by combining their provisioning and managed services with Kubernetes to provide managed Kubernetes services that are easier to spin up and scale than ever. These services from major cloud providers combine the infrastructure abstraction and scalability of cloud with the benefits of Kubernetes to provide a one click deployment of Kubernetes.

Adopting a Kubernetes as a Service solution ties customers to the precise implementation of the provider. That’s not always the optimal solution for every organization’s needs and for many deployments, a non-managed (self hosted) approach is chosen. One could set up Kubernetes on virtualized servers such as EC2 but that leaves you at the mercy of unpredictable performance due to multi-tenancy bursts when other virtualization customers on the same underlying hardware experience a burst in resource consumption. Concerns about a virtualized neighbor impacting your application performance is just one of the reasons some technology groups elect a self-hosted bare metal deployment of Kubernetes.

Equinix Metal is one such option. When deciding between a managed solution vs. self hosted or considering the impact of a virtualized vs non-virtualized deployment, many questions need to be asked to identify the best path for the particular use case at hand. This article explores some of the background knowledge required and the opportunities and advantages of a self-hosted bare metal Kubernetes deployment.

Kubernetes is an open-source container-orchestration software. First announced by Google in 2014, it was released as an open-source technology managed by the Linux Foundation in 2015. Kubernetes takes advantage of container technology that was popularized by Docker which provided many of the same virtualization benefits that VMs provided with much less memory overhead and a vastly improved developer experience.

Containers vs Virtual Machines: WeaveWorks

Applications built on Kubernetes are able to abstract away the hardware providing one platform to develop for and enable high efficiency in utilizing resources, out-of-the-box autoscaling, simplified CI/CD, and increased durability and fault tolerance. However Kubernetes has a fairly steep learning curve, both in developing cloud-native applications that can take full advantage of Kubernetes and managing the Kubernetes deployment itself. Cloud vendors saw the growth of Kubernetes and released Managed Kubernetes offerings that abstracted away or automated many parts of the Kubernetes deployment and lifecycle management. However these Managed Kubernetes offerings were built on top of existing virtualization stacks.

Azure Stack Summary: CIO

By building on top of existing virtualization stacks, both performance and debugging are impacted. Part of the appeal of using containers was that they had lower system overhead than virtual machines, however when using managed Kubernetes services not only do you incur the performance cost of containers but also of the underlying virtualization software. In addition, debugging becomes a problem whenever you introduce more layers of abstraction. Ask any Kubernetes expert that has dealt with managed cloud Kubernetes offerings and they can provide horror stories about debugging networking, GPU management and more. However when compared to the hassle of managing your own infrastructure and the lack of scalability associated with on-prem Kubernetes clusters, the trade off seems reasonable and Managed Kubernetes offerings seem like the clear winner for most production workloads.

However the choice is no longer as clear. The rise of bare-metal server providers like Equinix Metal combine the scalability of cloud providers with the increased flexibility of bare metal. Requesting new servers is simple through their web portal or API.

Equinix Metal Web Portal: Equinix Metal

Similar to how other cloud providers work, you provision servers and deploy things on them. However the main difference is you are actually “buying a physical server that [they] … provision for you” to use. As the only tenant on the node, you never have to worry about “noisy neighbor” problems where you might start operating at half capacity because other tenants are gobbling up bandwidth or other limited resources. However operating on bare metal is certainly not without its downsides. Provisioning and installing OSes can impact auto-scaling as the time to install an OS on a bare metal machine is typically longer than on a cloud VM. You also have to deal with the physical hardware which comes with its own problems and benefits.

Deploying on bare metal has several Kubernetes specific benefits as well. GPU usage simplifies because you do not have to worry about any of the virtualization layers. Dealing with networking between nodes and in between pods becomes simpler because each node has a dedicated IP address that they can use. In addition the performance penalty shrinks allowing better utilization and optimizations of resources.

Equinix Dell Bare Metal Stack: Equinix Metal

The problems many had faced with deploying Kubernetes on bare metal is the lack of automation and features that have become staples of cloud deployments. Things like upgrading Kubernetes automatically, quick scale out and scale up, backups, and load balancing that are usually fairly tedious processes are automated in cloud Kubernetes services. However, bare metal providers like Equinix Metal recognize these problems and have begun developing a suite of automations and plugins that try to emulate those features in collaboration with industry leaders. Equinix Metal provides “automation to upgrade Kubernetes .. automatically”. Kube-vip provides features like load balancing to bare metal deployments of Kubernetes. Slowly but surely, bare metal implementation of Kubernetes is reaching feature parity with cloud deployments.

Choosing your deployment platform for your application is no longer simple. Before you had the binary choice between cloud provider and an on-prem deployment. The choice basically boiled down to cost and any type of security concerns or other constraints you were working with. Once companies started to buy into the cloud model, it was initially relatively easy to migrate from on-prem infrastructure to a set of cloud managed services. For many, cloud managed services were and still are perfectly adequate, but for an increasing number of companies, especially those with large or specialised requirements, new ways to achieve their goals are becoming attractive. Nowadays companies have multiple options including hybrid cloud software and bare metal providers. These offerings provide a richer menu of options around the flexibility and scalability of the Kubernetes deployments while still ensuring that you have full control and access to your hardware. Next time you are designing an application, be sure to weigh the pros and cons of each and choose wisely.