This chapter is from the book
This chapter is from the book
This chapter is from the book
Container Engine for Kubernetes
As containerized applications grow in scale, they tend to become smaller and more distributed. Modern distributed applications are designed as a network of microservices that each implement a specific feature and communicate with each other using well-defined interfaces. When combined with container-based packaging, this design paradigm enables each of these smaller services to scale, update, and expand in their features, independent of each other. This also helps increase the overall development velocity and supports a more frequent release of smaller changes. As the number of these containers rises, however, so does their management overhead; the overall application also increases in its complexity. Manually or statically wiring together the containers and keeping track of their status and health soon becomes an untenable approach. This calls for more automation to orchestrate the container workloads. For large container-based workloads that require significant higher-level abstractions and orchestration, Kubernetes is the platform of choice. Kubernetes offers much beyond container management, and it provides features such as autoscaling, resource management, service discovery, load balancing, and deployment management. Kubernetes is a CNCF graduated open-source project and can be installed and run on public cloud, hybrid, or on-premises infrastructures.
Kubernetes exemplifies the “pets versus cattle” approach. The premise is that you do not treat your infrastructure as individual hosts or resources, each with a designated purpose and name, like a pet. Instead, you see your infrastructure as a fleet of servers, with none serving any special role and all being completely replaceable. For instance, with Kubernetes, you can configure an application container so that it is allowed to use two cores and 8GB of memory to run, and you can request that three instances of the application be running at any one time. Using this configuration, called a manifest and typically represented in YAML, Kubernetes can create the required number of containers to meet your specification and keep track of their health. Kubernetes can move your containers as the fleet’s status changes and failures occur, all without intervention. In this manner, Kubernetes enables you to describe the configuration you desire in the deployment manifests; you can then apply these manifests to a Kubernetes cluster that keeps track of the containers, nodes, and other resources and ensures that your configuration defined in the manifest is always met. Because these manifests can be versioned, releasing new changes and rolling back to previous configurations becomes trivial for most applications.
Oracle Cloud Infrastructure Container Engine for Kubernetes (OKE) is the managed Kubernetes platform for developing modern applications from Oracle. Although you can install Kubernetes on any infrastructure yourself, the installation and upkeep of the administrative and platform components in Kubernetes can be challenging. Figure 3-5 illustrates the various components of an OKE cluster.
)
FIGURE 3-5 Components in an OKE Cluster, Showing the Oracle-Managed Control Plane and the Data Plane Where the User Has Full Control
As a software suite that manages container-based workloads, Kubernetes has a set of administrative components that manage and control the cluster for tasks such as keeping track of the nodes, workloads, configurations, health status, and so on. The nodes on which these control components run are called the control plane nodes. In a managed Kubernetes platform such as OKE, these are installed and managed by the cloud provider. The control plane nodes do not run any workloads other than the management processes for the cluster itself. Users do not have access to these nodes.
The workloads themselves run on compute instances called worker nodes. The cluster control plane processes monitor and record the state of the worker nodes and schedule workloads onto them. A node pool is a subset of worker nodes within a cluster that all have the same configuration. A cluster must have a minimum of one node pool, but a node pool need not contain any worker nodes.
OKE supports two types of node pools that differ in how the nodes in the pool are managed. Managed node pools have nodes that are controlled by the user. Virtual node pools, on the other hand, are fully managed by OCI. Managed node pools and virtual node pools address different use cases and usage models. A managed node is a compute instance of the user’s choice of shape. Users have full access to these nodes, including SSH access and the capability to customize the nodes with user-created OS images and cloud-init scripts. Nodes run the kubelet process, which is responsible for ensuring that the pods scheduled on the node are running and reporting on the node health conditions acting as a node agent for Kubernetes. Virtual nodes, on the other hand, leverage Kubernetes pod configurations to create an isolated compute environment for the pod. Each Kubernetes pod is therefore isolated from other pods at a hypervisor level. The configuration of the execution environment, such as the number of CPU cores and memory, is inferred from the resource requests and limits set on the containers in the pod configuration. The execution environment for the pod is fully managed by Oracle and runs abstracted, away from the user. Virtual node pools therefore completely remove the need to manage infrastructure when deploying Kubernetes workloads and can be considered to be a serverless Kubernetes platform. Although managed nodes give users a high degree of control in accessing and managing their nodes (as with using custom cloud-init scripts to customize nodes), they come with the additional overhead of managing the node’s OS and Kubernetes upgrades. Virtual nodes, on the other hand, offer an experience that is focused on your workload, with little or no infrastructure management overhead. However, that comes at the expense of having control over the configuration of nodes. A single cluster can have both provisioned and virtual node pools.
Table 3-1 offers a comparison of managed and virtual nodes.
Table 3-1 Managed versus Virtual Nodes
Managed Nodes |
Virtual Nodes |
|
|---|---|---|
Infrastructure control |
Users maintain control over nodes. |
Users can control the workload but not the infrastructure. |
Upgrades |
Users upgrade the nodes. |
Upgrades are fully managed by OKE. |
Isolation |
A node’s resources are shared by the pods that run on it. |
A virtual node has no physical resources. Each pod runs in its own hypervisor-level isolated compute environment. |
Resource management |
Users decide the shapes of the nodes and set resource requirements and limits for pods. The Kubernetes scheduler matches pods to nodes based on availability. |
Nodes need not be created or managed. Users should set resource requirements and limits on pods, to create dedicated compute environments for each pod. |
Node pools can also have placement configurations that control the placement of the nodes in the node pool. These placement configurations can be used to spread the nodes in a node pool across multiple availability domains or fault domains to ensure better resiliency. Creating multiple node pools enables you to create groups of machines within a cluster that have different configurations. Figure 3-6 shows two node pools, one with E3.Flex shapes and another with A1 bare-metal machines. Here, the first node pool is based on AMD (x86)–based Flex shape virtual machine instances; the second node pool is an ARM-based bare-metal shape. This example also demonstrates that different node pools can be of different shapes, CPU architectures, and bare metal or virtual machines. Having this flexibility in your cluster resources lets you right-size the workloads and progressively introduce infrastructure changes to your environments—for example, introducing ARM-based compute for a subset of workloads or using bare-metal or GPU-enabled nodes for compute-heavy workloads and VMs for supporting workloads. Node pools also let you control the placement of nodes across availability domains and fault domains in OCI. Figure 3-6 shows the first node pool placing nodes across all three availability domains and the second node pool restricting nodes to just two of the three availability domains.
)
FIGURE 3-6 Node Pools in a Cluster Can Be Used to Control the Node Types and Their Placement, as Well as Create Clusters with Multiple Types of Nodes in Separate Node Pools
The node pools act as the control unit for scaling and can be used to scale the number of compute instances up or down, to add or remove compute capacity in the cluster. The scaling also can be automated based on metrics. Autoscaling is covered in more detail in Chapter 4, “Understanding Container Engine for Kubernetes.”
Note that the number of pods that can be scheduled or placed on a node is still dependent on the network address space available on a node, up to a maximum of 110. Larger nodes with more CPU cores and memory are therefore ideal to accommodate pods with much higher resource consumption needs. The memory and network throughput are also important considerations when choosing a shape for your nodes. The maximum available memory and network bandwidth changes, based on the shape of the node and the number of OCPUs (an OCPU is a complete core, not just a hardware thread). Thus, the choice of shape for the nodes also depends on the memory and network throughput expectations for the workloads.
As one of the highest-velocity open-source projects, Kubernetes provides support for three minor versions. This support policy is sometimes also called an N-2 support policy, in which the latest version and the two preceding minor versions of Kubernetes get patches for security and bug fixes. OKE as a managed service does not force users to upgrade as new versions of Kubernetes are released, although keeping your Kubernetes version up to date with the latest security fixes and bug fixes is an important consideration. For creating new clusters, OKE always supports at least three versions of Kubernetes. Version choice for new clusters moves like a rolling window as well. When OKE adds a new version of Kubernetes as a choice for creating new clusters, the oldest version choice remains a choice for at least 30 days, beyond which it may be removed. Exiting clusters that use that version are unaffected; the removal simply means that new clusters will have newer version choices. As support for new Kubernetes versions is added to OKE, you can update the control plane to the new version with a click of a button or an API call. The control plane upgrades are completely managed by Oracle and are transparent to the user. The Oracle-managed control plane is always in a highly available configuration, and the upgrade is performed in a rolling fashion so that it does not impact the cluster’s normal operations. After the control plane has been upgraded, the node pools can be upgraded as well. Chapter 4 does a deep dive into OKE, providing best practices, strategies, and tips for building and deploying applications to OKE.