Installing and maintaining Kubernetes clusters correctly and securely is difficult. Thankfully, all the major cloud providers, such as Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP), facilitate installing and maintaining clusters.
Different ways to create an AKS cluster
Using the portal: The portal offers a graphical user interface (GUI) for deploying your cluster through a wizard. This is a great way to deploy your first cluster. For multiple deployments or automated deployments, one of the following methods is recommended.
Using the Azure CLI: The Azure command-line interface (CLI) is a cross‑platform CLI for managing Azure resources. This allows you to script your cluster deployment, which can be integrated into other scripts.
Using Azure PowerShell: Azure PowerShell is a set of PowerShell commands used for managing Azure resources directly from PowerShell. It can also be used to create Kubernetes clusters.
Using ARM templates: Azure Resource Manager (ARM) templates are an Azure‑native way to deploy Azure resources using Infrastructure as Code (IaC). You can declaratively deploy your cluster, allowing you to create a template that can be reused by multiple teams.
Using Terraform for Azure: Terraform is an open‑source IaC tool developed by HashiCorp. The tool is very popular in the open‑source community for deploying cloud resources, including AKS. Like ARM templates, Terraform also uses declarative templates for your cluster.
Creating your first AKS cluster
We will start our initial cluster deployment using the Azure portal. The Azure portal is a web‑based management console. It allows you to build, manage, and monitor all your Azure deployments worldwide through a single console.
To start, browse to the Azure portal on https://portal.azure.com. Enter the keyword aks in the search bar at the top of the Azure portal. Click on Kubernetes services under the Services category in the search results:
This will take you to the AKS pane in the portal. As you might have expected, you don't have any clusters yet. Go ahead and create a new cluster by hitting the + Add button, and select the + Add Kubernetes cluster option:
This will take you to the creation wizard to create your first AKS cluster. The first step here is to create a new resource group. Click Create new, give your resource group a name, and hit OK. If you want to follow along with the examples in this book, please name the resource group rg-handsonaks:
Next, change the node count to 2. The default Standard DS2 v2 node size is sufficient.
The final view of the first pane should look like this:
In the final view, Azure will validate the configuration that was applied to your first cluster. If you get the message Validation passed, click Create:
Deploying the cluster should take roughly 10 minutes. Once the deployment is complete, you can check the deployment details as shown:
The Kubernetes resources section gives you an insight into the workloads that are running on your cluster. You could, for instance, see running deployments and running pods in your cluster. It also allows you to create new resources on your cluster. We will use this section later in the chapter after you have deployed your first application on AKS.
In the Node pools pane, you can scale your existing node pool (meaning the nodes or servers in your cluster) either up or down by adding or removing nodes. You can add a new node pool, potentially with a different virtual machine size, and you can also upgrade your node pools individually.
In the Cluster configuration pane, you can instruct AKS to upgrade the control plane to a newer version. Typically, in a Kubernetes upgrade, you first upgrade the control plane, and then the individual node pools separately. This pane also allows you to enable role-based access control (RBAC) (which is enabled by default), and optionally integrate your cluster with Azure AD.
Finally, the Insights pane allows you to monitor your cluster infrastructure and the workloads running on your cluster. Since your cluster is brand new, there isn't a lot of data to investigate.
AKS nodes and node pools
Creating an AKS cluster automatically creates and configures a control plane, which provides core Kubernetes services and application workload orchestration. The Azure platform provides the AKS control plane at no cost as a managed Azure resource. The control plane and its resources exist only in the region where you created the cluster. The nodes, also called agent nodes or worker nodes, host the workloads and applications.
In AKS, customers fully manage and pay for the agent nodes attached to the AKS cluster. To run applications and supporting services, an AKS cluster needs at least one node: An Azure virtual machine (VM) to run the Kubernetes node components and container runtime. Every AKS cluster must contain at least one system node pool with at least one node.
AKS groups nodes of the same configuration into node pools of VMs that run AKS workloads. System node pools serve the primary purpose of hosting critical system pods such as CoreDNS. User node pools serve the primary purpose of hosting workload pods. If you want to have only one node pool in your AKS cluster, for example in a development environment, you can schedule application pods on the system node pool.
You can also create multiple user node pools to segregate different workloads on different nodes to avoid the noisy neighbor problem, or to support applications with different compute or storage demands. Every agent node of a system or user node pool is a VM provisioned as part of Azure Virtual Machine Scale Sets and managed by the AKS cluster.
You can add a node pool to a new or existing AKS cluster by using the Azure portal, Azure CLI, the AKS REST API, or infrastructure as code (IaC) tools such as Bicep, Azure Resource Manager (ARM) templates, or Terraform.
When you create a new node pool, the associated virtual machine scale set is created in the node resource group, an Azure resource group that contains all the infrastructure resources for the AKS cluster. These resources include the Kubernetes nodes, virtual networking resources, managed identities, and storage.
In the Node pools pane, you can scale your existing node pool (meaning the nodes or servers in your cluster) either up or down by adding or removing nodes. You can add a new node pool, potentially with a different virtual machine size, and you can also upgrade your node pools individually.
Virtual nodes
You can use virtual nodes to quickly scale out application workloads in an AKS cluster. Virtual nodes give you quick pod provisioning, and you only pay per second for execution time. You don't need to wait for the cluster autoscaler to deploy new worker nodes to run more pod replicas. Virtual nodes are supported only with Linux pods and nodes. The virtual nodes add-on for AKS is based on the open-source Virtual Kubelet project.
Virtual node functionality depends on Azure Container Instances.
Azure Application Gateway as a Kubernetes ingress
An ingress in Kubernetes is an object that is used to route HTTP and HTTPS traffic from outside the cluster to services in a cluster. Exposing services using an ingress rather than exposing them directly, as you've done up to this point—has a number of advantages. These advantages include the ability to route multiple hostnames to the same public IP address and offloading TLS termination from the actual application to the ingress.
To create an ingress in Kubernetes, you need to install an ingress controller. An ingress controller is software that can create, configure, and manage ingresses in Kubernetes. Kubernetes does not come with a preinstalled ingress controller. There are multiple implementations of ingress controllers, and a full list is available.
There are two ways of configuring the AGIC, either using Helm or as an Azure Kubernetes Service (AKS) add-on. Installing AGIC using the AKS add-on functionality will result in a Microsoft-supported configuration. Additionally, the add-on method of deployment will be automatically updated by Microsoft, ensuring that your environment is always up to date.
To find this, look for agic (or the name you gave your application gateway) in the Azure search bar, and select your application gateway.
This will show you your application gateway in the Azure portal:
To verify that it has been created successfully, browse to the DNS name you configured for the public IP address. Note that the error message shown is expected since you haven't configured any applications yet behind the application gateway. You will configure applications behind the application gateway using AGIC in the Adding an ingress rule for the guestbook application section.
Now that you've created a new application gateway and were able to connect to it, we will move on to integrating this application gateway with your existing Kubernetes cluster.
Adding TLS to an ingress
You will now add HTTPS support to your application. To do this, you need a TLS certificate. You will be using the cert-manager Kubernetes add-on to request a certificate from Let's Encrypt.
There are a couple of steps involved. The process of adding HTTPS to the application involves the following:
1. Install cert-manager, which interfaces with the Let's Encrypt API to request a certificate for the domain name you specify.
2. Install the certificate issuer, which will get the certificate from Let's Encrypt.
3. Create an SSL certificate for a given Fully Qualified Domain Name (FQDN). An FQDN is a fully qualified DNS record that includes the top-level domain name (such as .org or .com). You created an FQDN linked to your public IP in step 2 in the section Creating a new application gateway.
4. Secure the front-end service by creating an ingress to the service with the certificate created in step 3.
It will again take about a minute for the certificate to become active. Once the new certificate is issued, you can browse to your DNS name again and shouldn't see any more warnings regarding invalid certificates. If you click the padlock icon in the browser, you should see that your connection is secure and uses a valid certificate:
Monitoring applications
Monitoring the health of applications deployed on Kubernetes as well as the Kubernetes infrastructure itself is essential for providing a reliable service to your customers. There are two primary use cases for monitoring:
- Ongoing monitoring to get alerts if something is not behaving as expected.
- Troubleshooting and debugging application errors.
When observing an application running on top of a Kubernetes cluster, you'll need to examine multiple things in parallel, including containers, pods, services, and the nodes in the cluster. For ongoing monitoring, you'll need a monitoring system such as Azure Monitor or Prometheus.
When you are experiencing issues in AKS, a good place to start your exploration is the AKS Diagnostics pane. It provides you with tools that help investigate any issues related to underlying infrastructure or system cluster components.
AKS Diagnostics gives you two tools to diagnose and explore issues. One is Cluster Insights, and the other is Networking. Cluster Insights uses cluster logs and configuration on your cluster to perform a health check and compare your cluster against best practices. It contains useful information and relevant health indicators in case anything is misconfigured in your cluster.
Networking and network security in AKS
By default, the control plane is exposed over the internet and is accessible to everybody that is connected to the internet. This doesn't mean that the control plane is not secure though. Even if an attacker had network access to your control plane, they would still need to have cluster credentials to execute commands against the control plane.
Frequently, though, organizations still want to limit network access to the control plane of their AKS clusters. There are two functionalities in AKS that enable you to limit network access to the control plane of a cluster.
The first functionality is called authorized IP address ranges. By configuring authorized IP address ranges on your AKS, you configure which IP addresses are allowed to access your API server. This means that IP addresses that are not allowed to access your API server cannot interact with your API server.
Private Link is an Azure feature that allows you to connect to managed services using a private IP address in your VNet. When using Private Link, a Private Link endpoint is created in your VNet. To connect to this Private Link endpoint, you would have to connect from either a VM hosted in the same VNet, in a peered VNet, or through a VPN or Azure ExpressRoute that is connected to that VNet.
You'll first need to choose the networking model with which you'll deploy your cluster. This configuration has a limited impact on security, but it is important to understand from a networking perspective. There are two options:
Kubenet networking (default): By using kubenet networking, cluster nodes get an IP address from a subnet in a VNet. The pods running on those nodes get an IP address from an overlay network, which uses a different address space from the nodes. Pod-to-pod networking is enabled by Network Address Translation (NAT). The benefit of kubenet is that only nodes consume an IP address from the cluster subnet.
Azure Container Network Interface (CNI) networking (advanced): With Azure CNI, the pods and the nodes all get an IP address from the subnet that the cluster is created in. This has the benefit that pods can be accessed directly by resources outside the cluster. It has the disadvantage that you need to execute careful IP address planning, since each pod requires an IP address in the cluster subnet.
In both networking models, you can create the cluster in an existing VNet or have AKS create a new VNet on your behalf.
The second network security configuration to consider is the routing of inbound and outbound traffic through an external firewall. This could either be an Azure Firewall or a third-party network virtual appliance (NVA). By routing traffic through an external firewall, you can apply centralized security rules, do traffic inspection, and log traffic access patterns. To configure this, you would configure a user-defined route (UDR) on the cluster subnet, to route traffic from your cluster through the external firewall.
Another network security option is the use of NSGs in Azure to limit inbound and outbound traffic. By default, when you create a service of the LoadBalancer type in AKS, AKS will also configure an NSG to allow traffic from everywhere to that service. You can tune the configuration of this NSG in AKS, to limit which IPs can access those services.
Finally, you can limit traffic in your cluster by using a Kubernetes feature called network policies. A network policy is a Kubernetes object that allows you to configure which traffic is allowed on certain pods. With network policies, you can secure pod-to-pod traffic, external to pod traffic, and pod to external traffic. It is recommended that you use network policies mainly for pod-to-pod traffic (also called east-west traffic), and to use an external firewall or NSGs for external-topod or pod-to-external traffic (also called north-south traffic).
Workload network security
You have now learned about how to protect the network of your control plane of your AKS cluster. This, however, hasn't influenced the network security of your workloads. In this section, you will explore three ways in which you can protect your workloads. First, you will create a Kubernetes service using an Azure internal load balancer. Then, you'll secure traffic to a service in Kubernetes using NSGs. Finally, you will use network policies to secure pod-to-pod traffic.
You have used the service type load balancer multiple times before to have AKS create an Azure load balancer. These have always been public load balancers. You can also configure AKS in such a way that it will create an internal load balancer. This is useful in cases where you are creating a service that only needs to be accessible from within a VNet or networks connected to that VNet.
AKS created an internal load balancer to expose this service. You can see this load balancer in the Azure portal as well. To see this internal load balancer, start by searching for load balancer in the Azure search bar:
Click on the kubernetes-internal load balancer.
However, you never restricted access to your application, except in the previous section, by deploying an internal load balancer. This means that the application was always publicly accessible. In the following example, you will create a service on your Kubernetes cluster that will have a public IP, but you will restrict access to it using an NSG that is configured by AKS.
By adding additional configuration to a Kubernetes service, you were able to limit who was able to connect to your service. You were able to confirm that only the public IP that was allowed to connect to the service was able to connect to it. A connection not coming from this public IP address timed out.