Kubernetes Architecture

Mahmoud Mokhtar

Kubernetes Architecture

Kubernetes is a Container Orchestration Platform. It's considered a distributed platform consisting of Master nodes and Worker nodes we can describe the master nodes as a server and worker nodes as clients.

  • Master controls and manages the cluster.
  • Worker exposes infrastructure resources for the cluster (Storage, Compute, Network).

 

K8s Architecture

Master Nodes

The control plane node provides a running environment for the control plane agents responsible for managing the state of a Kubernetes cluster we can describe it as it is the brain behind all operations inside the cluster.
Users can send requests to the control plane via a CLI tool, a Web UI Dashboard, or an Application Programming Interface (API).


The master node consists of the below components

  • API server.
  • Etcd.
  • Scheduler.
  • Controller Manager.
In addition, the control plane node runs:
• Container Runtime
• Node Agent
• Proxy
• Optional add-ons for cluster-level monitoring and logging.

now we will describe the function of each component in the Kubernetes Master node.

API server

The K8s API server is considered the primary management component of the Kubernetes cluster. The API server is the central communication point in the cluster and all other services communicate to each other through the K8s API server.

  • Expose the k8s and listen for communications from the outside cluster (user interactions with the cluster) and communications from the cluster components. The API Server receives RESTful calls from users, administrators, developers, operators, and external agents, and then validates and processes them.
  • During processing the API Server reads the Kubernetes cluster's current state from the key-value store (Etcd), and after a call's execution, the resulting state of the Kubernetes cluster is saved in the key-value store for persistence.
  • The API Server is the only control plane component to talk to the key-value store (Etcd), both to read from and to save Kubernetes cluster state information - acting as a middle interface for any other control plane agent inquiring about the cluster's state.

Controller Manager

  • Kube Controller Manager is the brain behind the Kubernetes cluster
  • Manager manages various controllers in Kubernetes.
  • In Kubernetes terms, a controller is a process that continuously monitors the state of the components within the system and works towards bringing the whole system to the desired functioning state.
  • In Kubernetes, a controller is a control loop that watches the API server to get updates from Etcd through the API server and do the needed actions to match the current state with the desired state.
  • Each object has a controller inside the controller manager (node controller, replication controller, Namespace controller, pod controller, ….etc)

Scheduler

  • Identifies the right node to place the container on based on container resource requirements, worker node capacity, and any other policies (tents and toleration, node affinity roles )
  • responsible for assigning and scheduling the PODs on the worker nodes according to pre-configured policies. 
  • API Server sends the new workload object's requirements to the scheduler 
    • These requirements may include constraints that users and operators set, such as scheduling work on a node labeled with disk==ssd key-value pair.
    • The scheduler also takes into account Quality of Service (QoS) requirements, data locality, affinity, anti-affinity, taints, toleration, cluster topology, etc.
  • The scheduler obtains the cluster state, resource usage, and the data about each worker in the cluster from the key-value store via the API Server.
  • Once all the cluster data is available, the scheduling algorithm filters the nodes with predicates to isolate the possible node candidates which then are scored with priorities to select the one node that satisfies all the requirements for hosting the new workload.
  • The outcome of the decision process is communicated back to the API Server, which then delegates the workload deployment to other control plane agents.

Etcd

Etcd is an open-source, distributed key-value storage system that maintains the K8s cluster configurations. Also contains the current state of all objects in the cluster.

  • Etcd is the cluster DB and API server used to store cluster configs and data.
    • The ETCD Datastore stores information regarding the cluster such as Nodes, PODS, Configs, Secrets, Accounts, Roles, Bindings and, Others.
  • Every information you see when you run the kubectl get command is from the ETCD Server. And every change we made on our cluster is updated in ETCD.
  • It can be deployed on the muster nodes or on a separate cluster with High availability for it.
  • If Etcd becomes down the whole cluster will be down.

Worker Nodes

worker nodes are where the actual workload is executed within a Kubernetes cluster the worker node components:

  • container runtime
  • Kubelet
  • kube proxy.

Let's go deeper on each of the worker node components.

container runtime

Container runtime is the software responsible for running and managing containers on the host server. At the beginning of using containers there was just Docker as container run time then introduced some other container run times like Rocket (rkt).

  • K8s came to orchestrate docker then k8s introduced an interface called Container runtime interface CRI.
  • CRI allowed any vendor to work as a container runtime for k8s as that container runtimes adhere to the OCI standards.
  • In K8s v1.24 release of Kubernetes to remove dockershim completely and so support for Docker was removed.

Kubernetes currently supports other container runtimes, such as Containerd, Rocket, and CRI-O

Kubelet

We can consider it as an agent on each worker node, Kubelet is responsible for communicating with the Kubernetes control plane and registering the worker node in the K8s API server.

Kubelet communicates container run time through the CRI (container runtime interface) and instructs it to deploy or destroy containers on the node.

It manages the containers on the node, ensuring that they are running and healthy according to the specifications provided by the Kubernetes API server.

kube proxy

Kube-proxy is a network proxy that runs on each worker node and maintains network rules required to implement Kubernetes networking concepts such as Service abstraction and load balancing.

  • Manages the service using the IP table rules.
  • It always watches the API server for new services to start creating their IP table rules on each node to forward traffic to those services to backend pods
  • It always watches the API server for new services to start creating their rules on the IP tables
  • It also watches the API server for EP these endpoints are created by the endpoint controller when the created POD is working and healthy and get EP IP can receive traffic on that IP

Note

Everything in the cluster is containerized even the Master node components (Application, cluster components) so we need container runtime on all nodes including the master nodes.


EX to know how cluster components communicate

In this example, we will assume that the user uses cli to create pods using a file this file contains 3 replicas, a container with Ngnix name, using a specific image, and expose port 80.

  1. Through CLI the Yaml file will be sent to the API server
  2. The API server will run some checks and validations (Authentication, Authorization, Schema validation, Mutating Admission control, Validate Admission control)
  3. If all validation is passed will store the new object in the Etcd db.
  4. The Etcd will reply that the object stored successfully
  5. The API server will ACK to the user that the object deployment is successful (actually the deployment is just stored in the Etcd and not created on worker nodes yet)
  6. the controller manager watches the API server and will find a new desired state (new deployment) not match the current state.
      - It has a replica set controller that will create the required 3 replicas and create 3 pods for these replicas
      - Will send this update to the API server to store it in the Etcd
  7. The scheduler watches the API server and will find 3 new pending pods.
      - Will start to schedule the required pods according to the available resources on the worker nodes and the required resources for the pods (affinity role, specific node,….etc)
  8. The scheduler will update the pods specifications to the API server to store it in the Etcd
  9. On the worker nodes the Kubelet watch the Master API server will find pending pods with its worker node ID 
      - Kubelet will create the required pods
  10. The kubelet also manages the container run time on its worker node and instructs the container runtime to create the required containers inside the created pods