Provision and configure a Kubernetes cluster in minutes

Learn how to use Terraform to create a Kubernetes cluster on Google Cloud platform in a few minutes, along with a working Helm installation and convenience charts, like an ingress controller. In this article I will explain how to define a Terraform plan that is able to create a new cluster, install Helm, and make it publicly accessible with an ingress controller. All with a single command.

This is very useful if you need to have an isolated environment created on the fly to allow testing of an experimental feature. Or for your personal projects, keeping the resources running the bare minimum amount of time you need and avoiding extra costs. Being so fast and easy, there are a lot of uses.

Although the plan we are going to create is mostly geared towards small clusters for development or test, hints and notes will be provided about how some things should be done for a production cluster. This article is mostly focused on Google Kubernetes Engine, but the underlying concept and idea could be modified to work on other cloud provider.

A sample plan can be found on this Github repository.

Prerequisites

This article assumes a basic knowledge of all the tools and technologies involved.

Before you begin, ensure that you have an account already set up with Google Cloud Platform, that you have created a project, and that Kubernetes Engine is activated.

You will also need a working Terraform installation. To allow the tool to manage resources inside GCP, I would recommend creating a service account with the following roles: Compute Network Admin, Kubernetes Engine Admin and Kubernetes Engine Cluster Admin. You can look into the documentation for details on how to do so.

Part 1: Cluster creation

Create a new folder to place the Terraform files. The first file you should create is variables.tf, which will be used to define input variables. While following this article, remember that a code line like foo = "${var.bar}" is referring to a variable called bar inside that file. You can find a sample file in the Github repository but, generally, names will be self explanatory.

Again in that folder, create a new file cluster.tf. This file will hold the resources needed to create the cluster itself. Start by defining the Google provider:

provider "google" {  
   project = "${var.project}"  
   region  = "${var.region}"  
}

By itself, this will not work, because it requires credentials. Here is where the service account that you created will come in handy. Run a command like export GOOGLE_APPLICATION_CREDENTIALS="[path to credentials JSON]" to indicate Terraform which credentials to use. Or explore other methods of providing credentials that better suit your needs.

Also, declare a data source that will be used to access the configuration for the provider:

data "google_client_config" "current" {}

The next step will be to define the cluster resource itself. In GKE, a cluster is comprised of one or more masters, and one or more nodes. The master runs the control plane processes, deciding when and where running pods, attending API requests, etc. This control plane is automatically handled by the cloud providers on their managed offerings. The cluster also has one or more nodes, which are responsible for executing the containers with the workloads, as instructed by the master. In GKE the nodes are grouped into node pools, which is a group of nodes that share the same configuration. You can define multiple node pools depending on your workload needs, and use features like node taints and tolerations to distribute the pods onto the different nodes.

Defining a new google_container_cluster resource is simple enough:

resource "google_container_cluster" "primary" {  
  name = "${var.cluster_name}"  
  location = "${var.cluster_location}"  
  remove_default_node_pool = false  
  initial_node_count = "${var.pool_min_nodes}"

  node_config {  
    machine_type = "${var.pool_machine_type}"  
    preemptible = true  
    oauth_scopes = [  
      "https://www.googleapis.com/auth/compute",  
      "https://www.googleapis.com/auth/devstorage.read_only",  
      "https://www.googleapis.com/auth/logging.write",  
      "https://www.googleapis.com/auth/monitoring"  
    ]  
  }

  master_auth {  
    username = ""  
    password = ""  
    client_certificate_config {  
      issue_client_certificate = false  
    }  
  }  
}

At this point, if we executed terraform init and terraform apply, we should be able to create a small, fully functional cluster. Let us go through the different parameters individually.

Cluster location

This is the location in which the cluster will be created. The created cluster will be very different depending on whether you specify a region or a zone.

If a region is specified (e.g. europe-west1), a regional cluster will be created. This will have multiple masters spread across the zones in the region. It will also have at least a node in each zone, so the minimum number of nodes will be 3. If a zone is specified (e.g. europe-west1-b) a zonal cluster will be created, with a single cluster master. The minimum number of nodes will be 1.

For a production cluster, you should probably go with a regional cluster, as it provides better availability and resilience. For a temporary cluster, a zonal cluster with a single node is definitely cheaper.

Node pools

You cannot create a cluster in GKE without a default node pool, although you can delete all the node pools. The most common approach to node pools provisioning, recommended by Terraform, and that you can find on most articles, is setting remove_default_node_pool to true and specifying a separate node pool using a google_container_node_pool resource. This launches the cluster with the smallest possible default node pool, deletes it immediately, and then creates a new node pool.

Dissociating the management of the cluster from the management of the node pools provides several benefits, like performing modifications on the node pool without bringing down the whole cluster. It is what you definitely should do in your production cluster.

However, for a temporary cluster, there is one caveat to that approach. As we need to wait for the cluster creation, and then for the separate node pool creation, we essentially double our execution time from ~4 mins to ~8 mins. That is not relevant for a permanent cluster, but when the cluster is going to be temporary and disposed of quickly, waiting times start to add up. That is why we have opted for using the default node pool.

Authentication

We are disabling basic authentication by specifying and empty username and password, and we are also disabling client certificate authorization. For a production cluster, you should take a look at the security documentation and use the system that better suits your particular needs.

Preemptible instances

A preemptible instance is a virtual machine that will terminate itself after 24 hours, or at any other moment if the capacity is needed elsewhere. That is, a virtual machine that can be shutdown at any given moment. Why would you use that? Because it is extremely cheap! You can save up to 80% compared to a regular instance.

If you workload is mostly stateless, you can save a ton using preemptible instances. With enough nodes, there will not be any downtime. If your workload is mixed, you can have several node pools, and run your stateless applications on preemptible instances, and your stateful applications on normal ones. The node pools will take care of recreating the instances that get shutdown. You can even intercept the preemption signal (happens 30 seconds before the machine is actually shutdown) for not losing a single request.

On the sample repository, we are using a single preemptible node. This is as cheapest as it can get (running 24/7 would be ~7 USD/month!), but our cluster would not be available all the time. Needless to say, you do not want to do this on a production cluster but, still, consider using preemptible instances whenever possible to keep your costs down.

Part 2: Configuring the cluster

Helm

After creating a cluster, it is very frequent that you need to perform additional configuration and installation of different tools. This would get cumbersome pretty fast. Even if you automated it with another tool, you would have to execute two commands. I will explain how to leverage Terraform to perform those actions as part of the plan execution, using Helm and an ingress controller as an example. You could easily extend them to include other tools.

Let us start with Helm. It has two components: a command-line tool (helm), which is a client, communicating with a server-side component running in the cluster (tiller). As a GKE cluster has Role-Based Access Control (RBAC) enabled by default, installing tiller on our cluster requires creating a ServiceAccount and assigning it a role through a ClusterRoleBinding. On a production cluster you may want to have a look at further security configuration, using TLS, etc.

Create a new file and name it kubernetes.tf. We will start by defining our kubernetes provider. We could think of it as configuring the context for kubectl:

provider "kubernetes" {  
  load_config_file = false

  host = "${google_container_cluster.primary.endpoint}"  
  cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"  
  token = "${data.google_client_config.current.access_token}"  
}

As you can see, endpoints and authentication information is dynamically retrieved from the cluster resource that just got created. Terraform is smart enough to detect that there is a dependency on the cluster resource. And, because those attributes are computed and will not be populated until the cluster has been created, which is exactly what we want, as it would be impossible to create resources on a cluster that does not exist yet.

After the provider is defined, we create two resources required by tiller:

resource "kubernetes_service_account" "helm_account" {  
  depends_on = [  
    "google_container_cluster.primary"  
  ]

  metadata {  
    name = "${var.helm_account_name}"  
    namespace = "kube-system"  
  }  
}

resource "kubernetes_cluster_role_binding" "helm_role_binding" {  
  metadata {  
    name = "${kubernetes_service_account.helm_account.metadata.0.name}"  
  }

  role_ref {  
    api_group = "rbac.authorization.k8s.io"  
    kind      = "ClusterRole"  
    name      = "cluster-admin"  
  }  
  subject {  
    api_group = ""  
    kind = "ServiceAccount"  
    name      = "${kubernetes_service_account.helm_account.metadata.0.name}"  
    namespace = "kube-system"  
  }  
  provisioner "local-exec" {  
    command = "sleep 15"  
  }  
  depends_on = [  
    "kubernetes_service_account.helm_account"  
  ]  
}

It should be noted that here we are defining explicit dependencies (using depends_on). The dependency on the ServiceAccount exists mostly to avoid errors when running terraform destroy, because otherwise, it would try to delete the resources after the cluster is already destroyed. The dependency on the ClusterRoleBinding is explicit because the attributes we are dependent on (metadata.0.name) are not computed, so they are available right from the start of the execution. That would cause Terraform to try to create the resources at once, which would fail because the role binding requires the account to be already there.

At this point, we are ready to install Helm itself. For that, we create a new file called helm.tf, and then configure our third and last provider:

provider "helm" {  
  service_account = "${kubernetes_service_account.helm_account.metadata.0.name}"  
  tiller_image = "gcr.io/kubernetes-helm/tiller:${var.helm_version}"  
  install_tiller  = true

  kubernetes {  
    host  = "${google_container_cluster.primary.endpoint}"  
    token = "${data.google_client_config.current.access_token}"

    client_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"  
    client_key = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"  
    cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"  
  }  
}

A key point here is that it automatically tries to install tiller if it is not already installed, which is exactly what we want. The rest of the attributes are there to set the credentials to the cluster, similar like what we did for the kubernetes provider. If we did not provide them, it would look for the default credentials for the machine kubectl configuration.

Ingress controller

Once Helm is working, we can deploy any chart. As a sample, we are going to deploy an ingress controller, but you could add any other chart, including your own application. Instead of directly specifying Kubernetes resources, we will leverage our shiny new Helm installation to deploy a chart.

First, though, on cluster.tf define an IP address to be used with the ingress. This way, we do not have to rely on Google creating it automatically, and we also can declare an output to get the ingress address after it is created:

resource "google_compute_address" "cluster_ip" {  
  name = "${var.cluster_name}-ingress"  
  address_type = "EXTERNAL"  
}

output "cluster_address" {  
  description = "IP for accessing the ingress resource"  
  value = "${google_compute_address.cluster_ip.address}"  
}

Afterwards, we define a new Helm release for the ingress resource:

resource "helm_release" "ingress" {  
  chart = "stable/nginx-ingress"  
  name = "nginx-ingress"  
  namespace = "kube-system"

  set {  
    name  = "controller.service.loadBalancerIP"  
    value = "${google_compute_address.cluster_ip.address}"  
  }

  depends_on = [  
    "kubernetes_cluster_role_binding.helm_role_binding"  
  ]  
}

Et voilà! As simple as that, and we have our ingress controller set up. It should be accessible on the IP we just created. Refer to the documentation on Terraform and on the chart to view other settings that you can use to configure its behavior.

The ingress controller itself is running on the cluster, but requires underlying cloud infrastructure to work. As our objective is to create a cheap cluster, you need to be aware of the costs of those underlying resources to avoid unpleasant surprises. Especially the load balancer. Pricing is determined by the amount of forwarding rules created, with a minimum of five. That is, up to five rules, the cost is the same. And that cost is ~18 USD per month, which is quite expensive when comparing it to other resources (e.g. our preemptible node goes by ~7 USD monthly, which is less than half). You only pay for these resources as long as they exist, so if your cluster is quickly disposed off and does not live long, you should not worry too much. But if you want to keep your cluster running all the time, and are on a budget, you may consider not having an ingress controller.

Summary

If at this point you run terraform apply, in a few minutes your cluster should be up and running. You can add further refinements, like installing your own application chart, a monitoring system, a database, or whatever you want to have on your cluster. You could also configure your Terraform state so it is properly saved on remote storage. These are just the basics, but should be enough to get you started. And remember to destroy the resources when they are no longer in use!

Remember that the code is available on this GitHub repository.

Thanks a lot for your time, and happy terraforming!