Infrastructure Automation - Kubestack

Infrastructure Automation

1. Prepare local environment

info_outline

Please note, the quickstart and infrastructure automation are a beta release. The resulting clusters are fully functional and automated using Terraform and Kustomize but CI/CD is not yet included.

To bootstrap your infrastructure automation this quickstart helps you setup a infrastructure repository, prepare your cloud environment and get your first cluster up and running.

You can easily add more clusters in different regions, on different cloud providers or both later. But for now pick one cloud and lets get started.

# Download and unpack the EKS quickstart. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/infra-quickstart-eks-v0.2.0-beta.1.zip
unzip infra-quickstart-eks-v0.2.0-beta.1.zip
rm infra-quickstart-eks-v0.2.0-beta.1.zip

# Change into the quickstart directory and initialize your Git repo.
cd infra-quickstart-eks
git init .
git add .
git commit -m "Initialized from infra-quickstart-eks"

# Download and unpack the AKS quickstart. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/infra-quickstart-aks-v0.2.0-beta.1.zip
unzip infra-quickstart-aks-v0.2.0-beta.1.zip
rm infra-quickstart-aks-v0.2.0-beta.1.zip

# Change into the quickstart directory and initialize your Git repo.
cd infra-quickstart-aks
git init .
git add .
git commit -m "Initialized from infra-quickstart-aks"

# Download and unpack the GKE quickstart. Remove the archive file.
curl -LO https://storage.googleapis.com/quickstart.kubestack.com/infra-quickstart-gke-v0.2.0-beta.1.zip
unzip infra-quickstart-gke-v0.2.0-beta.1.zip
rm infra-quickstart-gke-v0.2.0-beta.1.zip

# Change into the quickstart directory and initialize your Git repo.
cd infra-quickstart-gke
git init .
git add .
git commit -m "Initialized from infra-quickstart-gke"

To make it easier to provide all prerequisites like the cloud provider command line utilities, Terraform, Kustomize and more we provide a container image and use it for bootstrapping now and also CI/CD later.

# Build the bootstrap container
docker build -t kbst-infra-automation:bootstrap ci-cd/

# Exec into the bootstrap container
docker run --rm -ti \
    -v `pwd`:/infra \
    -u `id -u`:`id -g` \
    kbst-infra-automation:bootstrap

2. Bootstrap cloud environment

info_outline

We're not creating prerequisite cloud ressources like the state bucket, service account and organizations or projects automatically because of common chicken-and-egg, permissions, billing account and quota issues and the fact that it complicates the ability to terrafrom destroy. We prioritize seamless day to day operations over lower one time setup effort.

Initialize AWS CLI

You can use your personal account credentials for this. We will create a dedicated service account later to use for CI/CD.

# run aws configure and follow the instructions
# make sure to set a default region
aws configure

Initialize Azure CLI

You can use your personal account for this. We will create a dedicated service account later to use for CI/CD.

# run az login and follow the instructions
az login

Initialize Google Cloud SDK

You can use your personal account for this. We will create a dedicated service account later to use for CI/CD.

# run gcloud init and follow the instructions
gcloud init

Configure your cluster

To configure your cluster you will use a number of Terraform features. The quickstart example repository bootstraps this Terraform configuration for you. Lets go through the files in the repository one by one to understand their purpose and make any required changes.

  1. clusters.tf

    This file loads the cloud specific cluster modules once per cluster pair and passes the respective configuration. Modules are self-contained packages of reusable Terraform configurations. Kubestack provides a module that implements the Ops- and Apps-cluster approach and unifies the managed Kubernetes solution for each supported cloud.

    The quickstart already loads the module for your first cluster, so there are no changes required here.

  2. variables.tf and config.auto.tfvars

    Input variables are used set values specific to a cluster and its workspace. Terraform requires variables to be defined, which is what we do in variables.tf and then has multiple ways to set values for these variables. In our case that's what we do in config.auto.tfvars.

    The quickstart config file is commented to help you set values for required variables like the cluster name prefix, the base domain, the cloud provider region, the number of desired Kubernetes nodes and more. Make sure to change the example values in this file before you continue.

  3. providers.tf

    Providers are Terraform's implementation of the cloud provider APIs. In providers.tf we configure the Terraform provider for your chosen cloud. The majority of provider configuration, most importantly API credentials are read from the environment automatically, but depending on your cloud some configuration per provider is required. Open the provided example file and make sure you edit it accordingly.

  4. state.tf

    Terraform stores state about your infrastructure that it uses to map the real world resources to your configuration, keep track of metadata and to improve performance. Because we are using Terraform in a team and CI/CD scenario it is important to store state centrally and ensure only one change is applied at any given time. Both of this is achieved by using remote state with locking. Setting up Terraform remote state and its prerequisites is the topic of the next section.

Setup remote state

Terraform supports various backends for remote state including implementations for each major cloud provider's object storage which is what we'll use for the quickstart.

info_outline

Object storage bucket names have to be globally unique so we append the short git hash of our initial commit by default to enable copy-pasteable instructions. Alternatively, you can of course choose your own bucket name. Something like terraform-state-$company_name-$project_name has a fair chance to result in a unqiue name.

AWS S3

Run the following commands to create a bucket in AWS S3 and change the state.tf file to configure Terraform to use the created bucket for remote state.

# Create bucket and configure remote state
BUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`
REGION=`aws configure get region`
aws s3api create-bucket --bucket $BUCKET_NAME --region $REGION --create-bucket-configuration LocationConstraint=$REGION --acl private
aws s3api put-bucket-versioning --bucket $BUCKET_NAME --region $REGION --versioning-configuration Status=Enabled
aws s3api put-bucket-encryption --bucket $BUCKET_NAME --region $REGION --server-side-encryption-configuration '{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }
  ]
}'
sed -i 's/bucket = ""/bucket = "'${BUCKET_NAME}'"/g' state.tf
sed -i 's/region = ""/region = "'${REGION}'"/g' state.tf

Azure blob storage

Run the following commands to create a storage container in Azure blob storage and change the state.tf file to configure Terraform to use the created storage container for remote state.

# Select the resource group
az group list --output table
read -p "Resource group: " RESOURCE_GROUP

# Select the storage account name
az storage account list --output table
read -p "Storage account name: " STORAGE_ACCOUNT

# Create storage container and configure remote state
STORAGE_CONTAINER=terraform-state-kubestack-`git rev-parse --short HEAD`

export ARM_ACCESS_KEY=$(az storage account keys list --account-name ${STORAGE_ACCOUNT} --resource-group ${RESOURCE_GROUP} | jq -r '.[0].value')

az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT

sed -i 's/storage_account_name = ""/storage_account_name = "'${STORAGE_ACCOUNT}'"/g' state.tf
sed -i 's/container_name       = ""/container_name       = "'${STORAGE_CONTAINER}'"/g' state.tf

Google cloud storage

Run the following commands to create a bucket in Google cloud storage and change the state.tf file to configure Terraform to use the created bucket for remote state.

# Set the location of your multi-regional bucket
# valid values are `asia`, `eu` or `us`
read -p "Bucket location: " LOCATION
# Create bucket and configure remote state
BUCKET_NAME=terraform-state-kubestack-`git rev-parse --short HEAD`
gsutil mb -l $LOCATION gs://$BUCKET_NAME
gsutil versioning set on gs://$BUCKET_NAME
sed -i 's/bucket = ""/bucket = "'${BUCKET_NAME}'"/g' state.tf

Initialize remote state and workspaces

With the remote state prerequisites setup we need to initialize Terraform to use the remote state and create the two workspaces ops and apps that will hold the state for the Ops- and Apps-cluster respectively.

# Initialize Terraform
terraform init

# Create apps and ops workspaces
terraform workspace new apps
terraform workspace new ops

3. Bootstrap the Ops-and Apps-cluster pair

We use terraform workspaces to control which cluster the configuration is applied to. Because we created the ops workspace second, we're currently in the ops workspace and are ready to bootstrap the Ops-cluster.

1. Ops-cluster

# To bootstrap the Ops-cluster simply run
terraform apply

Check the output and if everything looks nice and green, confirm with yes.

2. Apps-cluster

# To bootstrap the Apps-cluster select the apps workspace and run apply
terraform workspace select apps
terraform apply

Again, check the output and confirm if you're happy with the plan.

4. Setup DNS

DNS name server records

Kubestack's DNS setup is designed to be per cluster to ensure full testability. To this purpose, a zone was created for each cluster based on the name_prefix and base_domain you set in config.auto.tfvars earlier.

To be able to resolve DNS names in those zones, we have to:

  1. get the created zones' nameservers made available as Terraform outputs
  2. set NS records in the base_domain for each zones' nameservers
info_outline

The zone setup here is what we refer to as the infrastructure domain. Meaning it's not user facing. The easiest way to expose applications on a user friendly URL like www.example.com is to use a CNAME.

AWS Route53

1. Ops-cluster
# get ops fqdn and name servers
terraform workspace select ops
terraform output --module=eks_zero.cluster_metadata fqdn
terraform output --module=eks_zero.cluster ingress_zone_name_servers

2. Apps-cluster
# get apps fqdn and name servers
terraform workspace select apps
terraform output --module=eks_zero.cluster_metadata fqdn
terraform output --module=eks_zero.cluster ingress_zone_name_servers

Azure DNS

1. Ops-cluster
# get ops fqdn and name servers
terraform workspace select ops
terraform output --module=aks_zero.cluster_metadata fqdn
terraform output --module=aks_zero.cluster ingress_zone_name_servers

2. Apps-cluster
# get apps fqdn and name servers
terraform workspace select apps
terraform output --module=aks_zero.cluster_metadata fqdn
terraform output --module=aks_zero.cluster ingress_zone_name_servers

Google Cloud DNS

1. Ops-cluster
# get ops fqdn and name servers
terraform workspace select ops
terraform output --module=gke_zero.cluster_metadata fqdn
terraform output --module=gke_zero.cluster ingress_zone_name_servers

2. Apps-cluster
# get apps fqdn and name servers
terraform workspace select apps
terraform output --module=gke_zero.cluster_metadata fqdn
terraform output --module=gke_zero.cluster ingress_zone_name_servers

Set NS records

Finally, set NS records in the base_domain for both the Ops- and Apps-cluster's DNS zone.

The goal is, to have a NS query return the cluster's name servers for your cluster domain similar to how it does for the example on Google below.

dig NS kbst-apps-us-east1.gcp.infra.kubestack.com

[...]
;; ANSWER SECTION:
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a1.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a2.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a3.googledomains.com.
kbst-apps-us-east1.gcp.infra.kubestack.com. 21599 IN NS ns-cloud-a4.googledomains.com.

5. Congratulations

If you've come this far, you now have a Ops- and Apps-cluster pair. Congratulations! With this setup you are almost ready for production. Almost? Yes, there is one last thing left to do. Setup CI/CD for your infrastructure repository.

Unfortunately CI/CD is not yet part of this beta but is coming soon.

To keep your configuration, lets commit the configuration to our repository.

# Run these commands outside of the bootstrap container
git add .
git commit -m "Add cluster configuration"

6. Clean-up

To clean-up and destroy all the cloud resources, you can use terraform destroy.

# First destroy the apps-cluster
terraform workspace select apps
terraform destroy

# Now also destroy the ops-cluster
terraform workspace select ops
terraform destroy