Extending Kubestack

TL;DR:

Use custom Terraform modules to extend your platform with on-premise or cloud resources.
Integrate custom modules into the framework by adopting Kubestack metadata and configuration inheritance.
Provision any Kubernetes YAML not available from the catalog using the custom-manifests module.

Introduction

You can extend a Kubestack platform to integrate with existing infrastructure or to provision resources that Kubestack does not provide modules for. There are two main use-cases.

For infrastructure requirements you can write custom Terraform modules and integrate them with the environments and outputs from the Kubestack modules.
For Kubernetes manifests you want to provision as part of your platform, Kubestack provides the custom-manifest module. Using the custom-manifest module, you can integrate and configure any upstream YAML the same way as all platform service modules.

Cluster Module Outputs

To integrate with cloud resources that are created by Kubestack modules, the modules provide Terraform outputs. You can combine the outputs with Terraform data sources, to query information about created resources that you may need as inputs to other resources.

Output `current_config`

The current_config output has the config used for the cluster in the current environment.

AmazonAzureGoogle

# module.eks_gc0_eu-west-1.current_config
eks_gc0_eu-west-1_current_config = {
  "base_domain"                = "kubestack.example.com"
  "cluster_availability_zones" = "eu-west-1a,eu-west-1b,eu-west-1c"
  "cluster_desired_capacity"   = "3"
  "cluster_instance_type"      = "t3a.xlarge"
  "cluster_max_size"           = "9"
  "cluster_min_size"           = "3"
  "name_prefix"                = "gc0"
}

Output `current_metadata`

The current_metadata output holds all the information that the metadata module generated for the cluster.

AmazonAzureGoogle

# module.eks_gc0_eu-west-1.current_metadata
{
  "domain"          = "aws.kubestack.example.com"
  "fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
  "label_namespace" = "kubestack.com/"
  "labels" = {
    "kubestack.com/cluster_domain"          = "aws.kubestack.example.com"
    "kubestack.com/cluster_fqdn"            = "gc0-ops-eu-west-1.aws.kubestack.example.com"
    "kubestack.com/cluster_name"            = "gc0-ops-eu-west-1"
    "kubestack.com/cluster_provider_name"   = "aws"
    "kubestack.com/cluster_provider_region" = "eu-west-1"
    "kubestack.com/cluster_workspace"       = "ops"
  }
  "name" = "gc0-ops-eu-west-1"
  "tags" = [
    "gc0-ops-eu-west-1",
    "ops",
    "aws",
    "eu-west-1",
  ]
}

Output `kubeconfig`

The kubeconfig output, provides a KUBECONFIG file that can be used to connect and authenticate to the Kubernetes API server of the respective cluster. This output is used to configure the kustomization providers used for platform service modules.

The following examples show how to use it with different Terraform providers focused on Kubernetes.

provider "kustomization" {
  alias = "gke_gc0_europe-west1"

  kubeconfig_raw = module.gke_gc0_europe-west1.kubeconfig
}

locals {
  gke_gc0_europe-west1_kubeconfig = yamldecode(module.gke_gc0_europe-west1.kubeconfig)
}

provider "kubernetes" {
  alias = "gke_gc0_europe-west1"

  host                   = local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["server"]
  cluster_ca_certificate = base64decode(local.gke_gc0_europe-west1_kubeconfig["clusters"][0]["cluster"]["certificate-authority-data"])
  token                  = local.gke_gc0_europe-west1_kubeconfig["users"][0]["user"]["token"]
}

Terraform Modules

Kubestack modules, are just Terraform modules. And consequently they can coexist alongside your custom modules in the same root module.

Query Using Data Sources

Terraform data sources can be used to query information about an existing resource. The primary resource is usually the cluster. Once you have the data source, you can access the attributes of the respective cluster resource.

The following examples show how to query information about an AKS, EKS or GKE clusters using a data source and the Kubestack module outputs.

AmazonAzureGoogle

data "aws_eks_cluster" "eks_gc0_eu-west-1" {
  provider = aws.eks_gc0_eu-west-1

  name = module.eks_gc0_eu-west-1.current_metadata.name
}

Attributes provided by the aws_eks_cluster data source are documented upstream.

Unique Names per Environment

Kubestack uses Terraform workspaces for its environments. When you create cloud resources, names usually have to be unique for one account or even globally. To make sure names are unique per environment, you can include the environment name in the resource name. The easiest way to achieve that is to include the terraform.workspace in the name.

locals {
  example_name = "example-${terraform.workspace}"
}

Kubestack itself goes a step further and includes a name_prefix, the terraform.workspace and the region in names. This provides names that are unique per environment, region and cloud provider.

The module that implements this, is used by all other Kubestack modules. And you can use the common metadata module to generate names and other metadata in your custom modules aswell.

module "example_metadata" {
  source = "github.com/kbst/terraform-kubestack//common/metadata?ref=v0.18.1-beta.0"

  name_prefix = "example"
  base_domain = var.base_domain

  provider_name   = "gcp" # kubestack uses aws, gcp or azure
  provider_region = module.gke_gc0_europe-west1.current_config["region"]

  # it's good practice to namespace labels
  label_namespace = "example.com/"
}

You can then access the same metadata that all Kubestack cluster modules provide. See the section about the current_metadata Terraform output above for more details.

output "example_metadata_name" {
  value = module.example_metadata.name
}

output "example_metadata_labels" {
  value = module.example_metadata.labels
}

Inheritance Based Configuration

To build custom modules that implement the same inheritance based configuration as Kubestack modules, use the common configuration module.

module "example_configuration" {
  source = "github.com/kbst/terraform-kubestack//common/configuration?ref=v0.18.1-beta.0"

  configuration = var.configuration
  base_key      = var.configuration_base_key
}

The configuration module expects to be given the configuration, as a map of maps. Where the keys of the outer map, are the environment names. And the base_key sets which environment all others inherit from. Most commonly you'll use this module inside another module, and the configuration and the base_key are passed in using variables when calling your module.

Kubernetes Manifests

Kubestack provides the custom-manifests module to apply any Kubernetes YAML. The custom-manifest module accepts the same configuration attributes as platform service modules. But unlike the modules from the catalog it does not bundle any upstream manifests. You can use it to integrate any Kubernetes YAML into your Kubestack platform the same way as any service from the catalog.

Usage example

To include custom YAML manifests:

place the YAML files in a directory under manifests/
call the custom-manifests module and reference the YAML files as resources

Below example uses the custom-manifests module to apply fictitious upstream YAML. It also sets an env label for all resources with the current Terraform workspace as the value.

If the Kubernetes service you need provides a helm chart, you can use helm template to vendor the rendered YAML in a file and apply them using the custom-manifests module.

module "example_custom_manifests" {
  providers = {
    kustomization = kustomization.gke_gc0_europe-west1
  }

  source  = "kbst.xyz/catalog/custom-manifests/kustomization"
  version = "0.4.0"

  configuration = {
    apps = {
      namespace = "example-${terraform.workspace}"

      resources = [
        "${path.root}/manifests/example/upstream.yaml"
      ]

      common_labels = {
        "env" = terraform.workspace
      }
    }

    ops = {}
  }
}

Custom-Manifest Configuration Attributes

The custom-manifest module configuration attributes are identical to platform service module configuration attributes. Since the custom-manifest module, unlike catalog modules, does not bundle any upstream YAML use resources instead of additional_resources to specify a Kustomization or the individual YAML files to provision.

module "eks_gc0_eu-west-1_service_example" {
  providers = {
    kustomization = kustomization.example_cluster_alias
  }

  # fictitious example module source and version
  source  = "kbst.xyz/catalog/example/kustomization"
  version = "0.0.0-kbst.0"

  configuration = {
    apps = {
      # list of paths to YAML files or Kustomizations to deploy (required)
      resources = [
        # kustomization
        "${path.root}/manifests/example-kustomization",

        # or individual YAML files
        "${path.root}/manifests/example/namespace.yaml",
        "${path.root}/manifests/example/deployment.yaml",
        "${path.root}/manifests/example/service.yaml",
      ]

      # set annotations on all Kubernetes resources
      # (optional), defaults to null
      common_annotations = {
        "example-annotation" = var.example_annotation
      }

      # set labels on all Kubernetes resources
      # (optional), defaults to null
      common_labels = {
        "example-label" = var.example_label
      }

      # generate Kubernetes configMaps
      # (optional), defaults to null
      config_map_generator = [{
        # name (required)
        # Sets 'metadata.name' of the configMap resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the configMap resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"  

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the configMap.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the configMap per line in the env file.
        envs = [
          "${path.root}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.root}/manifests/cfg.ini"           # data["cfg.ini"] = file_content
          "prod.ini=${path.root}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this configMap.
        options = {
          # see generator_options
        }
      }]

      # generate Kubernetes secrets
      # (optional), defaults to null
      secret_generator = [{
        # name (required)
        # Sets 'metadata.name' of the secret resource
        name = "example"

        # namespace (required)
        # Sets 'metadata.namespace' of the secret resource
        namespace = "example"

        # behavior (optional)
        # Valid values: 'create', 'replace' or 'merge'. Defaults to 'create'.
        behavior = "create"

        # type (optional)
        # 'type' attribute of the secret to generate.
        type = "generic"

        # literals (optional)
        # List of 'KEY=VALUE' strings. Sets 'data[KEY] = VALUE' in the secret.
        literals = [
          "KEY=VALUE"
        ]

        # envs (optional)
        # List of paths (strings) to env files (one KEY=VALUE pair per line).
        # Sets 'data[KEY] = VALUE' in the secret per line in the env file.
        envs = [
          "${path.root}/manifests/env"
        ]

        # files (optional)
        # List of paths (strings) to files.
        # Sets 'data[KEY] = file_content'. KEY defaults to the file's name.
        # Overwrite by prefixing 'customkey='.
        files = [
          "${path.root}/manifests/cfg.ini"           # data["cfg.ini"] = file_content
          "prod.ini=${path.root}/manifests/cfg.ini"  # data["prod.ini"] = file_content
        ]

        # options (optional)
        # Same as top level 'generator_options' but specific to this secret.
        options = {
          # see generator_options
        }
      }]

      # set options for all configMap and secret generators
      # (optional), defaults to null
      generator_options = {
        # annotations (optional)
        # Sets 'metadata.annotations' on the generated resources. Defaults to '{}'.
        annotations = {
          example-annotation = "example"
        }
        
        # labels (optional)
        # Sets 'metadata.labels' on the generated resources. Defaults to '{}'.
        labels = {
          example-label = "example"
        }

        # disable_name_suffix_hash (optional)
        # Disables hash suffix of generated resource's 'metadata.name'.
        # Defaults to 'false'.
        disable_name_suffix_hash = true
      }

      # patch image names, tags or digests
      # (optional), defaults to null
      images = [{
        # Refers to the 'pod.spec.container.name' to modify the 'image' attribute of.
        name     = "busybox"
        
        # Customize the 'registry/name' part of the image. The part before the ':'
        new_name = "new_name"
        
        # Customize the 'tag' part of the image. The part after the ':'.
        new_tag  = "new_tag"

        # Replace the 'tag' part of an image with a 'digest'.
        digest   = "sha256:..."
      }]

      # prefix or suffix to add to resource names
      # (optional), default to null
      name_prefix = "prefix-"
      name_suffix = "-suffix"

      # namespace to set for all resources
      # (optional), default to null
      namespace = "example"

      # patches to apply to resources
      # (optional), default to null
      patches = [{
        # path (optional)
        # Path to a file that defines a strategic merge or JSON patch.
        path = "${path.root}/manifests/patch.yaml"

        # patch (optional)
        # Inline string that defines a strategic merge or JSON patch.
        patch = <<-EOF
          - op: replace
            path: /metadata/name
            value: newname
        EOF

        # target (optional)
        # Target one or multiple resources to be patched.
        target = {
          group               = ""
          version             = "v1"
          kind                = "ConfigMap"
          name                = "example"
          namespace           = "example"
          label_selector      = "key=value"
          annotation_selector = "key=value"
        }
      }]

      # set replicas attribute of deployments and similar resources
      # (optional), default to null
      replicas = [{
        # Refers to the 'metadata.name' of the resource to scale
        name = "example"
        
        # Sets the desired number of replicas.
        count = 5
      }]

      # select upstream variant to deploy
      # available variants are documented on the respective catalog page
      # (optional), module specific default
      variant = "example"
    }

    ops  = {}
  }
}

low-level arguments

In addition to the convenience kustomization attributes documented above, platform service modules also pass the following low-level attributes through to the kustomization provider. These Kustomization attributes are less useful in the Terraform context and as such are not documented here.

components (optional) List of paths to Kustomize components.
crds (optional) List of paths to Kustomize CRDs.
generators (optional) List of paths to Kustomize generators.
transformers (optional) List of paths to Kustomize transformers.
vars (optional) List of objects to define Kustomize vars.

Framework Documentation