Kubernetes Guide

Infrastructure Automation

Infrastructure automation should have a dedicated repository, the infrastructure repository. It holds the configuration management code, the Kubernetes manifests for the cluster services and a CI/CD pipeline.

To quickstart your infrastructure repository you can follow the quickstart. It will get you up and running with Terraform configuration for a production ready Kubernetes cluster on Amazon, Azure or Google.

The implementation in the quickstart follows the principles of this guide, the guide aims to be vendor neutral.

Infrastructure as code

Treating infrastructure as code and using the same methods used for application development has proven its value for operating infrastructure in many organizations over the last couple of years.

But where previously, there was no clear definition where the configuration of the host OS ended and the configuration of the application running as a process on that host OS started, the clear separation of concerns that Kubernetes and containers enable makes following the infrastructure as code paradigm much easier.

Tracking your infrastructure configuration in version control, allows working with a well known pull-request, review and merge workflow. Once a commit is merged into master or a tag is pushed, it would trigger the infrastructure pipeline.

Triggers and pipeline stages

The pipeline usually consists of multiple stages that run the required steps to apply changes to one of the clusters. Its hard to give an abstract example, so for the sake of the example, we're using the quickstart's implementation based on Terraform and Kustomize.

+-----------------------------------------------------------------------------------------+
| terraform init stage                                                                    |
|                                                                                         |
| +-------------------------------------------------------------------------------------+ |
| |                                                                                     | |
| | terraform init                                                                      | |
| |                                                                                     | |
| +------------------------------------------+------------------------------------------+ |
|                                            |                                            |
|       triggered by                         |                       triggered by         |
|       master commit +----------------------+---------------------+ apps-deploy tag      |
|                     |                                            |                      |
|                     |                                            |                      |
| +-------------------v--------------------+   +-------------------v--------------------+ |
| |                                        |   |                                        | |
| | terraform workspace select ops         |   | terraform workspace select apps        | |
| |                                        |   |                                        | |
| +----------------------------------------+   +----------------------------------------+ |
|                                                                                         |
+-----------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------+
| terraform plan stage                                                                    |
|                                                                                         |
| +-------------------------------------------------------------------------------------+ |
| |                                                                                     | |
| | terraform plan -out=tfplan                                                          | |
| |                                                                                     | |
| +-------------------------------------------------------------------------------------+ |
|                                                                                         |
+-----------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------+
| terraform apply stage                                                                   |
|                                                                                         |
| +-------------------------------------------------------------------------------------+ |
| |                                                                                     | |
| | terraform apply tfplan                                                              | |
| |                                                                                     | |
| +-------------------------------------------------------------------------------------+ |
|                                                                                         |
+-----------------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------------+
| k8s manifest apply stage (runs per cluster pair)                                        |
|                                                                                         |
| +-------------------------------------------------------------------------------------+ |
| |                                                                                     | |
| | kustomize build manifests/ | kubectl apply -f -                                     | |
| |                                                                                     | |
| +-------------------------------------------------------------------------------------+ |
|                                                                                         |
+-----------------------------------------------------------------------------------------+

As you can see, the pipeline is setup to select a different terraform workspace, depending on if it was triggered from a commit in master branch or a apps-deploy tag. The workspace controls what Terraform state is used and based on that, which resources on the cloud provider side the configuration is applied to.

Once Terraform has run, the final step is to apply the Kubernetes manifests to the respective cluster or clusters. In a multi-region or multi-cloud setup, Terraform would update the configuration for all the clusters in the current workspace (ops or apps). It would also write a kubeconfig file for each of the clusters. The k8s manifest apply stage needs to run for each kubeconfig file. This way, depending on what clusters Terraform applied its configuration to, kubectl also applies the manifests to the same clusters from each cluster pair.

Cluster Service Manifests

Cluster services are all services that need to exist on the cluster before you can deploy your own applications. They are often times shared between multiple applications and environments like ingress controllers, monitoring, logging and service meshes. As such, just like the cluster configuration, changes to cluster service manifests need to be testable before they are applied to the Apps-cluster.

By being part of the infrastructure repository, changes to both cluster configuration and cluster service manifests are applied together, ensuring that a specific cluster configuration works with a specific set of cluster service manifests.

On-premise or homegrown vs. managed Kubernets in the cloud

The Kubestack quickstart and its example pipeline above heavily benefit from the fact, that its only provisioning managed Kubernetes clusters on Amazon, Azure or Google and only a limited amount of supporting cloud resources. The heavy lifting of managing updates to the cluster nodes is done by the cloud provider's managed service.

If running Kubernetes on-premise or on a provider that does not offer a managed Kubernetes service, following the approach this guide suggests with having a Ops- and Apps-cluster pair is still or arguably even more beneficial. But the stages that apply changes to the clusters in the CI/CD pipeline naturally get immensly more complex.