Kubernetes Deepdive

Core Concepts

Kubernetes Components

API Server - Frontend to Kubernetes; Users, Mgmt Devices, CLI talk to this.
etcd Key Store - Distributed, Reliable Key Values Store used to store cluster data, implements lock to avoid conflicts for multiple Master nodes scenario.
Kubelet - Agent that runs on each Node; responsible to make sure containers are running on nodes as expected.
Container Runtime - Underlying software used to run containers(eg: Docker, etc)
Controller - Brains behind orchestration; Notice & respond when Nodes, Containers, end points goes down; make decisions to bring up new containers
Scheduler - Distributes work or containers across multiple Nodes; looks for newly created containers and assigns them to nodes.

• Worker nodes were known as Minions earlier.
• Worker Nodes host the containers

Master Node               Worker Nodes
[Kube-ApiServer] -------> [Kubelet]
[etcd]
[controller]
[scheduler]               [Container Runtime]  Docker, Rocket, CRI-O

Kubectl used to deploy and manage applications on a Cluster to get cluster information.

kubectl run hello-minikube
kubectl run nginx --image nginx
kubectl cluster-info
kubectl get nodes
kubectl get pods
kubectl describe pod myapp-pod

PODS

Containers are encapsulated into kubenetes objects call Pods.

Pod is a single Instance of Application.
Pod is the smallest object that can be created in Kubernetes.
If the no of users increases, we do not increase no of app instances(containers) inside pod, we add new Pods.
You can deploy new pods in a new node if the capacity reaches for a single pod.

Multi-Container PODs

Rarely used
Helper Containers can run inside your same pod as the main application pod.
{[Python] [Helper pod]}

We run 4 application instances:

docker run python-app
docker run python-app
docker run python-app
docker run python-app

Then add 4 helper applications:

docker run helper -link app1
docker run helper -link app2
docker run helper -link app3
docker run helper -link app4

Need to take care of networking and shared volumes Also need to monitor the containers.

With Pods, all of this is done automatically.

YAML in Kubernetes

Used as input for creating objects like Pods, Replicasets, depoyments, services, etc.

Kubernetes definition file always contains 4 required root/top level properties:

apiVersion:    --> version of Kubernetes API; eg: v1, apps/v1-beta, extensions/v1-beta, etc  => String
kind:          --> Pod, Service, ReplicaSet, Deployment                                      => String
metadata:      --> Data about object - Name, Label, etc; Spacing should be like python       => Dictionary
spec:          --> Specify which image to be used, etc. Check documentation of Image         => Dictionary

Kind        Version
POD         v1
Service     v1
ReplicaSet  apps/v1
Deployment  apps/v1

• API Version is specific to what we are creating.

Example Definition file

> nano pod-definition.yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  labels:                 ==> Used to identify pods in case you have 100s of them, else it will be difficult to manage them
      app: myapp
      type: front-end
spec:
   containers:                      --> List/Array
     - name: nginx-container        --> '-' Indicates first item in the list. 
       image: nginx

> kubectl create -f pod-definition.yml

ReplicaSets

• Helps run multiple instances of single Pod
• Provides High Availability.
• If only a single instance of POD is running, it will make sure that it continues to run.
• In case the instance fails, it will run another instance.

Replication Controller vs Replica Set

• Replication Controller is older technology.
• Usage:

apiVersion: v1
kind: ReplicationController
metadata:
  name: my-app-rc
  labels:
      app: myapp
      type: front-end
spec:                          ==> Replication Controller
  template:                    ==> Below is data copied from another definition file
    metadata:
      name: my-app-pod
      labels:
        app: myapp
        type: front-end
    spec:                      ==> POD
      containers:
      - name: nginx-container
        image: nginx
  replicas: 3

Create Replication Controller:

kubectl create -f rc-definition.yml
kubectl get replicationcontroller

• ReplicaSet is newer technology.
• Usage:

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: myapp-replicaset
  labels:
      app: myapp
      type: front-end
spec:                          ==> Replication Controller
  template:                    ==> Below is data copied from another definition file
    metadata:
      name: my-app-pod
      labels:
        app: myapp
        type: front-end
    spec:                      ==> POD
      containers:
      - name: nginx-container
        image: nginx
  replicas: 3
  selector:                    ==> Used because it can also manage pods not created during replication set creation; Mandatory
    matchLabels:
      type: front-end

Create ReplicaSet:

kubectl create -f rs-definition.yml
kubectl get replicaset

Labels and Selectors

Scaling

1st Method:

Change replicas: => 6
kubectl replace -f rs-definition.yml

2nd Method:

kubectl scale --replicas=6 -f rs-definition.yml

OR

 kubectl scale --replicas=6 replicaset myapp-replicaset
                               (type)   (name)

3rd Method:

Scaling based on Load

Deployments

Use Case:

You do not want to update all of your Webservers at once.
Rolling Updates - Upgrade webserver one by one.
Deployments allow us to upgrade instances seamlessly using rolling updates, undo changes, pause and resume changes as required.

Containers < POD < Replica Set < Deployment

Usage:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-replicaset
  labels:
      app: myapp
      type: front-end
spec:
  template:
    metadata:
      name: my-app-pod
      labels:
        app: myapp
        type: front-end
    spec: 
      containers:
      - name: nginx-container
        image: nginx
  replicas: 3
  selector:
    matchLabels:
      type: front-end

kubectl create -f deployment-definition.yml
kubectl get deployments
kubectl get relicaset
kubectl get pods

Check all objects:

kubectl get all

NameSpaces

[Mark Smith]    [Mark Williams]

Kubernetes creates a set of PODs and services for its internal purposes - required by Networking, DNS,etc.
To isolate these from users & prevent accidental deletions or modification, kubernetes creates them under different Namespaces.
These are created at cluster startup under namespace: kube-system
kube-public is also created where resources available to all users are placed.
For small environments you can continue using default namespace.

NameSpace Isolation:

Use the same cluster for Dev & Production.
Isolate them.
Also you can apply Quota of resources to them.

Within NameSpace a POD can reach another service by using its hostname:

mysql.connect("db-service")

For connecting into another NameSpace called 'dev':

mysql.connect("db-service.dev.svc.cluster.local")
              
db-service    =>   service-name
dev           =>   namespace 
svc           =>   service
cluster.local =>   domain

This command list pods from default name space only:

kubectl get pods

Specify namespace to see its pods:

kubectl get pods --namespace=kube-system

POD is created in default name space:

kubectl create -f pod-definition.yml

To create it in another namespace:

kubectl create -f pod-definition.yml --namespace=dev

Otherwise specify it in the Definition files:

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  namespace: dev
  labels:
      app: myapp
      type: front-end
spec:
   containers:
     - name: nginx-container
       image: nginx

Create Namespace

1st Method:

apiVersion: v1
kind: Namespace
metadata:
  name: dev

kubectl create -f namespace-dev.yml

2nd Method:

kubectl create namespace dev

Permanently change Namespace

kubectl config set-context $(kubectl config current-context) --namespace=dev
kubectl get pods                            ==> This will show pods under 'dev'
kubectl get pods --namespace=default        ==> This will show pods under 'default'

To see PODs in all namespaces:

kubectl get pods --all-namespaces

Create Quota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-qouta
  namespace: dev
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 5Gi
    limits.cpu: "10"
    limits.memory: 10Gi

kubectl create -f compute-qouta.yml

Configuration

Commands and Arguments in Docker

Containers are not meant to run OS Thats is why if you run below command it will fail to start container:

docker run ubuntu

Docker files of popular containers has:

Nginx  => CMD ["nginx"]
MySql  => CMD ["mysqld"]

Ubuntu  => CMD ["bash"]

Run Commands in Ubuntu bash:

docker run ubuntu [command]

E.g:

docker run ubuntu sleep 5

Making this change permanent:

FROM Ubuntu
CMD sleep 5

Or:

FROM Ubuntu
CMD ["sleep", "5"]

Create a custom image from above:

docker build -t ubuntu-sleeper .
docker run ubuntu-sleeper

Changing the hard-coded values:

docker run ubuntu-sleeper sleep 10

Using Default values and applying custom value too:

FROM Ubuntu
ENTRYPOINT ["sleep"]
CMD ["5"]

docker run ubuntu-sleeper       ==> value = 5 (default)
docker run ubuntu-sleeper 20

For changing entrypoint on the fly:

docker run --entrypoint sleep2.0 ubuntu-sleeper 10     ==> Command run at startup = sleep2.0 10

Commands and Arguments in Kubernetes

Docker command:

docker run --name ubuntu-sleeper --entry-point sleep2.0 ubuntu-sleeper 10

Docker file:

FROM Ubuntu
ENTRYPOINT ["sleep"] 
CMD ["5"]

Definition File:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper-pod
spec:
   containers:
     - name: ubuntu-sleeper
       image: ubuntu-sleeper
       command: ["sleep2.0"]
       args: ["10"]

Environment Variables

Docker Command:

docker run -e APP_COLOR=pink simple-webapp-color

Definition file:

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
   containers:
     - name: simple-webapp-color
       image: simple-webapp-color
       ports: 
         - containerPort: 8080
       env:
         - name: APP_COLOR
           value: pink

• Plain Key Value:

env:
 - name: APP_COLOR
   value: pink

• ConfigMap:

env:
 - name: APP_COLOR
   valueFrom: 
        configMapKeyRef:

• Secrets:

env:
 - name: APP_COLOR
   valueFrom: 
        secretKeyRef:

Config Maps

Two methods to create a definition file

• Create using Commands line:

kubectl create configmap

• Definition file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
 APP_COLOR: blue
 APP_MODE: prod

Create the config map:

kubectl create -f config-map.yml

• Using Config Maps for a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
   containers:
     - name: simple-webapp-color
       image: simple-webapp-color
       ports: 
         - containerPort: 8080
       envFrom:
         - configMapRef:
                name: app-config

kubectl create -f pod-definition.yml

• Check Config Maps:

kubectl get configmaps
kubectl describe configmaps

Secrets

• Used to store passwords

• 2 ways to create them

Using CLI:

kubectl create secret generic\
      app-secret --from-literal=DB_Host=mysql \
                 --from-literal=DB_user=root \
                 --from-literal=DB_Password=passwd

Using Definition file:

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
data:
  DB_Host=mysql
  DB_user=root
  DB_Password=passwd

• Convert the secrets into encoded format:

$ echo -n 'mysql' | base64
bXlzcWw=
$ echo -n 'root' | base64
cm9vdA==
$ echo -n 'passwd' | base64
cGFzc3dk

• Now use these values in the Definition file:

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
data:
  DB_Host=bXlzcWw=
  DB_user=cm9vdA==
  DB_Password=cGFzc3dk

• Using the secret in a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
  labels:
     name: simple-webapp-color

spec:
  containers:
      - name: simple-webapp-color
        image: simple-webapp-color
        ports: 
           - containerPort: 8080
        envFrom:
            - secretRef:
                  name: app-secret

Create the Pod:

kubectl create -f pod-definition.yml

• Check Secrets:

kubectl get secrets
kubectl describe secrets
kubectl get secret app-secret -o yaml         ==> This will show the hash values

• Decode the hash values from above command:

$ echo -n 'bXlzcWw=' | base64 --decode
mysql

• Mount secrets as a Volume:

If you mount the secret as a volume, each attribute in secret is created as file with value of secret as its content.

 
 volumes:
    - name: app-secret-volume
      secret:
         secretName: app-secret

Check Volumes:

ls /opt/app-secret-volume
cat /opt/app-secret-volume/DB_Password

Docker Security

• Docker runs a process as user 'root' with PID = 1.
• This process runs on the host in different namespace.
• It can be seen on host with different PID.
• But this root account is not having all privileges as a normal process with root privileges.

docker run --user=1001 ubuntu sleep 3600             ==> run as a non root user
docker run --cap-add MAC_ADMIN ubuntu sleep 3600     ==> enable custom privileges
docker run --privileged ubuntu sleep 3600            ==> run with full fledged root privileges

• In Kubernetes you may choose to configure security at container level or Pod level.
• Security applied at container level overrides security applied at Pod level.

• Pod Level Security:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  securityContext:
     runAsUser: 1000
  containers:
      - name: ubuntu
        image: ubuntu
        command: ["sleep","10"]

• Container Level Security:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: ubuntu
        image: ubuntu
        command: ["sleep","10"]
        securityContext:
            runAsUser: 1000
            capabilities:
                 add: ["MAC_ADMIN"]

Service Account

• There are two types of accounts in kubernetes:

 User Account     - Used by Humans; eg - to deploy admin task or to deploy service
 Service Account  - Used by Machines; eg - account used by service to interact with Kubernetes cluster; like Prometheus to monitor Cluster; Jenkins to deploy services. 

• Create service account:

kubectl create serviceaccount testacnt 
kubectl get serviceaccount
kubectl describe serviceaccount testacnt        => Shows Token object
kubectl describe secret testacnt-token-kddbm    => Shows Token

curl https://10.12.33.1:6443/api -insecure --header "Authorization: Bearer eyTdgf....."

• Steps to use Service Accounts in Application:

Create the Service Account
Assign right permissions using RBAC - Role Based Access Control.
Export tokens and use it in the application.

• For every namespace, there is a service account 'default' created by itself.

• In case the application is hosted inside a Pod in cluster itself, we can simplify the authentication mechanism:

Mount the Secret as a volume inside the Pod running Application.

• By default every Pod is have a secret - 'default' mounted in it:

kubectl describe pod my-pod

Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount

kubectl exec -it my-pod ls /var/run/secrets/kubernetes.io/serviceaccount
=> ca.crt  namespace  token

• To view the Token:

kubectl exec -it my-pod cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyTdgf.....    

• 'default' service account is very much restrictive; can be used to run very basic kubernetes APIs.

• Use a custom Service Account for a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: ubuntu
        image: ubuntu
  serviceAccount: testacnt

• You cannot change the Service Account for a POD. You must delete and recreate a POD.
• However you can change the Service Account for a deployment. as any change to the POD deployment file will automatically trigger out a new rollout for the deployment.

Resource Requirements

• Kubernetes Scheduler decides which node a Pod goes to.
• Schedular takes into account the amount of resources required and amount of resources required.
• If not enough resources are available, the Pod will remain in 'Pending' stuck state.
• By default Kubernetes assumes that a Pod or a Container within Pod requires: 0.5 CPU; 256MB RAM

• CPU => 0.1 == 100m (mili)

Minimum value  = 1m

1 CPU => 1 AWS vCPU or
         1 GCP Core or
         1 Azure Core
         1 Hyperthread

• Memory:

256Mi == 268435456
1G
1Gi 

• Docker do not have any limit on the amount of resources it can consume, suffocating other nodes.
• By default there is a limit of 1 vCPU & 512Mi RAM to a container.

• Define Resource Request:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: ubuntu
        image: ubuntu
  resources:
     requests:
        memory: "1Gi"
        cpu: 1
     limits:
        memory: "2Gi"
        cpu: 2       

Taints and Tolerations

• Define what pods are placed on what nodes.
• If Taint applied to a node, no Pod can be placed on it except for ones where you apply Toleration.

• Apply Taint:

kubectl taint nodes node-name key=value:taint-effect
kubectl taint nodes node1 app=blue:NoSchedule

taint-effect = NoSchedule          => Do not place new Pods
               PreferNoSchedule    => Try not to place new pods, no guarantees
               NoExecute           => Evict any Pods that do not tolerate the taint

• Tolerations are added to Pods:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: ubuntu
        image: ubuntu
  tolerations:
      - keys: "app"
        operator: "Equal"
        value: "blue"
        effect: "NoSchedule"

• Master Node is by dafault run with Taint: NoSchedule

kubectl describe node kubemaster | grep Taint
=>   Taints:      node-role:kubernetes.io/master:NoSchedule

Node Selectors

• Used to run Heavy Applications on Bigger Nodes.
• Define a Pod to be assigned to a Large Node:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: ubuntu
        image: ubuntu
  nodeSelector:
      size: Large

• Label Nodes:

kubectl label node <node-name> <label-key>=<label-value>
kubectl label node node1 size=Large

Multi-Container PODs

• Used when you need 2 services: WebServer + Log Agent
• Both Apps have same life cycle - created together, destroyed together, Share Network space & have access to same Storage volumes

• Creating Mulit-Container Pods:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: web-app
        image: web-app
      - name: log-agent
        image: log-agent

• Design Patterns:

Sidecar
Adapter
Ambassador

Observability

• Pod Status

Pending              - run 'kubectl describe pod' to find why it is in pending state.
ContainerCreating
Running

• Pod Conditions

PodScheduled
Initialized
ContainersReady
Ready

To check the conditions:

kubectl describe pod 
kubectl get pods         ==> READY  ->  Means -> ContainersReady & Ready = True

Readiness and Liveness Probes

• Some pods (like jenkins) take 15 seconds to become ready.
• But the Describe command shows it in Ready state during initialization too.
• This is not correct.
• Specify below readiness probe, kubernetes will not immediately set Ready state.

• HTTP Test - /api/ready:

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: web-app
        image: web-app
        ports: 
           - containerPort: 8080
        readynessProbe: 
              httpGet:
                  path: /api/ready
                  port: 8080

• TCP Test - 3306

       readynessProbe: 
             tcpSocket:
                 port: 3306

• Exec Command

       readynessProbe: 
             exec:
                command:
                   - cat
                   - /app/is_ready

Liveness Probes

• If web server crashes, Nginx process exists & Container exits as well.

docker ps -a
 container id 4ygre874hufew         Exited(1) 41 seconds ago

• Since Dockers is not an orchestration engine, Container continues to stay dead & denies services to users until you manually create a new container.
• If Kubernetes Orchestration is used, every time the application crashes, Kubernetes makes an attempt to restart the container to restore service to the users.

with every crash, "Restart" count increases.

• Problem occurs if the application is not working but the container continues to stay alive.

e.g: Due to a bug, application is stuck in an infinite loop. For Kubernetes, container is up, so the application is assumed to be up, but the users are not served with application.
Now container needs to be restarted or destroyed and a new container needs to be brought up.

• Liveness probe tests if the application within the container is healthy or not.
• Parameters to be tested:

Http Application - /api/healthy
TCP Test - port 3306
Exec Command

• HTTP Test - /api/healthy:

nano pod-definitaion.yaml

apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  containers:
      - name: web-app
        image: web-app
        ports: 
           - containerPort: 8080
        livenessProbe: 
              httpGet:
                  path: /api/healthy
                  port: 8080
              initialDelaySeconds: 10
              periodSeconds: 5
              failureThreshod: 8

TCP Test - 3306

       livenessProbe: 
             tcpSocket:
                 port: 3306

Exec Command

       livenessProbe: 
             exec:
                command:
                   - cat
                   - /app/is_ready

Container Logging

• If you run an application which generates events & send them to standard output:

docker run kodecloud/event-simulator

• If you run the container in background in detached mode, tose logs are not visible:

docker run -d kodecloud/event-simulator

So you need to run below command to see logs:

docker logs <container-id>
docker logs -f <container-id>                 shows Live logs

• In Kubernetes, we can see the logs:

kubectl logs -f event-simulator-pod           shows Live logs

• If there are multiple containers in Pod, then you need to specify the name of the container you want to see the logs:

kubectl logs -f  event-simulator-pod   event-simulator

Monitor and Debug Applications

• Resource consumption monitoring

No of nodes in cluster
How many are healthy
Cpu,memory & network & Disk utilization
No of pod
Performance metrics of each pod - cpu, memory usage

• Kubernetes does not come with full monitoring tool.

• There are opensource tools to monitor these resources - Metrics Server, Prometheus, Elastic Stack
• Proprietary tools: - DataDog, Dynatrace

• Metrics Server:

Heapster was original monitoring tool for Kubernetes, which is not deprecated.
Its slim-down version is called Metrics Server.
We can have 1 metrics server per Kubernetes Cluster.
It retrieves metrics from each Kubernetes Nodes & Pods, aggregates them & stores them in memory.
We cannot get historical data from it, need to install one of the other tool for this.

• Kubelet is an agent which runs on each node, which takes instructions from kubernetes API master server & runs pods on nodes.
• Kubelet also contains a sub-component called cAdvisor or Container Advisor.
• cAdvisor is responsible for retreiving performance metrics from pods & exposing them through Kubelet API to make metrics available for the metrics server.

• If you use minikube for local cluster:

minikube addons enable metrics-server

• For other environments:

git clone https://github.com/kubernetes-incubator/metrics-serve
kubectl create -f deploy/1.8+/

• Usage:

kubectl top node
kubectl top pod

POD Design

Labels, Selectors and Annotations

• Labels & Selectors are used to group similar objects based on their class or kind.
• Can filter based on multiple items
• Labels are properties attached to each item - Class, Kind, Color, etc
• Selectors help to filter these items. e.g: Class=Mammal; Color=Green

• Kubernetes has objects like: Pods, Services, Replica Sets, Deployments,etc.
• Can group objects by their Types(Pods, services,etc), Application(App1, App2,etc), Functionality(Front-End, DB, Auth,etc)
• Apply labels & Using Selectors:

app = App1
function = Front-End

app = App2
function = Auth

app = App3
function = DB

Filter:

app=App1

• Applying Labels:

nano pod-definition.yaml

apiVersion: v1
kind: Pod
metadata:
   name: simple-webapp
   labels:
       app: App1
       function: Front-end
spec:
   containers:
   -  name: simple-webapp
      image: simple-webapp
      ports:
        - containerPort: 8080 

• Using Selectors:

kubectl get pods --selector app=App1

• Applying Labels to ReplicaSets:

nano replicaset-definition.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
   name: simple-webapp
   labels:                                ==> These are labels for Replica Sets
       app: App1
       function: Front-end
spec:
   replicas: 3
   selector:                              ==> Selectors
      matchLabels:
         app: App1
   template:                              ==> These are the labels for Pods
      metadata:
         labels:
            app:App1
            function: Front-end
      spec:
         containers:
         -  name: simple-webapp
            image: simple-webapp
      ports:
        - containerPort: 8080 

Annotations

• Used to record tool details like Build, Version, name, contact details, etc

apiVersion: apps/v1
kind: ReplicaSet
metadata:
   name: simple-webapp
   labels:                                ==> These are labels for Replica Sets
       app: App1
       function: Front-end
   annotations:
       buildversions: 1.34

Rolling Updates & Rollbacks in Deployments

• New Rollout creates new revisions - Revision 1, Revision 2, etc

kubectl rollout status deployment/myapp-deployment
  Waiting for Rollout to finish: 0 of 10 updated replicas are available...
  Waiting for Rollout to finish: 1 of 10 updated replicas are available...
  Waiting for Rollout to finish: 2 of 10 updated replicas are available...
  ........................................................................
  Waiting for Rollout to finish: 9 of 10 updated replicas are available...
  deployment "myapp-deployment" successfully rolled out.

kubectl rollout history deployment/myapp-deployment

• Deployment Strategy

Recreate Startegy - Delete all old version pods & create new ones - Event - Scaled down replica set from 5 to 0.
Rollout Update - Default - Updates one by one. - scaled up/down 5,4,3,2,1

• Upgrades:

Creates a new replica set.
Deploys new Pods in new Replica set
Deletes old Pods one by one from old Replica set

• Rollback:

kubectl rollout undo deployment/myapp-deployment

Commands Summary

• Create Deployment by Kubectl run

kubectl run nginx --image=nginx

Create:

kubectl create -f deployment-definition.yml

Get:

kubectl get deployments

Update:

kubectl apply -f deployment-definition.yml 
kubectl set image deployment/myapp-deployment nginx=nginx:1.9.1

Status:

kubectl rollout status deployment/myapp-deployment
kubectl rollout history deployment/myapp-deployment

Rollback

kubectl rollout undo deployment/myapp-deployment

Jobs

Normal Jobs run long time:

App
Web
DB

Below processes carry out specific task & then finish:

Batch
Analytics
Reporting

E.g of set of workloads meant for a short period of time:

Performing a computation.
Image processing
Performing analytics on a large data set
Generating report and sending email

In Docker:

docker run ubuntu expr 3 + 2

After finishing this task:

docker ps -a
  STATUS: Exited (0) 41 seconds ago (0 = Task completed successfully)

For Kubernetes:

apiVersion: v1
kind: Pod
metadata:
   name: math-pod
spec:
   containers:
   - name: math-add
     image: ubuntu
     command: ['expr', '3', '+', '2']
   restartPolicy: Never

kubectl create -f pod-definition.yml

kubectl get pods
   STATUS: complete   RESTARTS: 1,2,3,4......   (continues to running because restartPolicy is set to Always by default)

• ReplicaSet - Helps make sure that specified no of Pods are running all the time.
• Jobs - Used to run a set of pods to perform a given task to completion.

nano job-definition.yaml

apiVersion: batch/v1
kind: Job
metadata:
   name: math-add-job
spec:
   template:
      spec:
        containers:
        - name: math-add
          image: ubuntu
          command: ['expr', '3', '+', '2']
        restartPolicy: Never

Create Job:

kubectl create -f job0definition.yaml

Check Jobs:

kube get jobs

Check Pods:

kubectl get pods
  READY: 0/1     STATUS: Completed     RESTARTS: 0     AGE: 2m

Check Output/logs:

kubectl logs math-add-job-ld87pn
   5

Delete Job:

kubectl delete job math-add-job
  job.batch "math-add-job" deleted

Multiple Pods

nano job-definition.yaml

apiVersion: batch/v1
kind: Job
metadata:
   name: random-error-job
spec:
   completions: 3          ==> Pods run sequentially, one after another's completion
   parallelism: 3          ==> Creates all 3 pods in parallel
   template:
      spec:
        containers:
        - name: random-error
          image: kodekloud/random-error
        restartPolicy: Never

CronJobs

• CronJobs can be scheduled and run periodically.

nano cron-job-definition.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
   name: reporting-cron-job
spec:                               ==> for Cronjob
   schedule: "*/1 * * * *"
   jobTemplate: 
      spec:                         ==> for Job
         completions: 3
         parallelism: 3
         template:
            spec:                   ==> for Pod
              containers:
              - name: reproting-tool
                image: reporting-tool
              restartPolicy: Never

kubectl create -f cron-job-definition.yaml
kubectl get cronjob

Services & Networking

Services

• Enable communication withing and outside of Application.
• Help to connect application together with other applications or users
• They enable connectivity between Client & Frontend, Frontend with Backend, Backend with External Datasources.

• NodePort Service: Listens to a port on the node & forwards the requests to the pods.
• ClusterIP: Service creates a virtual IP inside the cluster to enable communication between different services such as set of frontend sevrers to a set of backend servers.
• LoadBalancer: Distributes load across nodes.

NodePort

[PC]-----------------{NodePort-30008}[Kube Node]{Port-80}--------------------{TargetPort-80}[POD]
192.168.1.10              192.168.1.2           10.106.1.12                       10.244.0.1

NodePort Service[30008]
curl http://192.168.1.2:30008
Node Port Range - 30000 - 32767

• Labels & Selectors used to connect TargetPort to Pod.

• Create a NodePort Service:

nano service-definition.yml

apiVersion:v1
kind: Service
metadata:
   name: myapp-service
spec:
   type: NodePort
   ports:
      - targetPort: 80       ==> If not provided, assumed to be same as targetPort
        port: 80             ==> Mandatory Field
        nodePort: 30008      ==> If not provided, automatically allocated from range(30000-32767)
   selector:
      app: myapp
      type: front-end

kubectl create -f service-definition.yml
kubectl get services

Check service:

curl http://192.168.1.2:30008 

• For Mapping multiple Pods, automatically mapping is created for all matching Pods with correct labels.
• In case pods are on multiple Nodes:

curl http://192.168.1.2:30008
curl http://192.168.1.3:30008
curl http://192.168.1.4:30008

Cluster IP

• Full Stack Web Application:

Front-end   [POD-10.244.0.3]  [POD-10.244.0.4]   [POD-10.244.0.5]  
                            \        |          /
                                 {back-end}
Back-end    [POD-10.244.0.6]  [POD-10.244.0.7]   [POD-10.244.0.8]  
                            \        |          /
                                  {redis}
Redis       [POD-10.244.0.9]  [POD-10.244.0.10]  [POD-10.244.0.11]  

• The IPs of PODs are not static, they may change at any time.
• Therefore Networking cannot be done on 1-1 IP/port basis.
• Also it is difficult to decide which front end POD connect to which back end POD.
• ClusterIP Service can be used to map each layer to an IP address/Port.

• Creating ClusterIP:

nano service-definition.yml

apiVersion: v1
kind: Service
metadata:
   name: back-end
spec:
   type: CLusterIP
   ports:
      - targetPort: 80
        port: 80
   selector:
      app: myapp
      type: back-end

kubectl create -f service-definition.yml
kubectl get services

• Service can be accessed by the ClusterIP or the Service Name.

Ingress Networking

• Normal Access:

[MySQL-POD]---------------------[WebApp-POD]--------------------------[Client]
      [service-ClusterIP]              [service-NodePort-38080]

Client access: http://192.168.1.10:38080
DNS Server : http://my-online-store.com:38080

• Replacing the Port number with standard port:

[MySQL-POD]---------------------[WebApp-POD]------------------------[Proxy-Server]--------------[Client]
      [service-ClusterIP]              [service-NodePort-38080]                 [80]

Client access: http://my-online-store.com

• In case hosting is done on Public Cloud, create a service of type LoadBalance, then Cloud (e.g: GCP) will automatically create a GCP-Loadbalancer:

[MySQL-POD]---------------------[WebApp-POD]------------------------[GCP-LoadBalancer]--------------[Client]
      [service-ClusterIP]              [service-NodePort-38080]                 [80]

Client access: http://my-online-store.com

• For multiple services you need multiple GCP-LoadBalncers, Need to pay more & re-configure for every additional service.
• Also you need to configure SSL for the applications, which becomes difficult to manage in case you have multiple applications, etc
• Also need to configure firewall rules for each application too.
• Ingress helps users access application using a single externally accessible URL.
• It can be configured to route different services and can also provide SSL configuration point.
• It can be thought of as a L7 LoadBalancer
• Still it needs to be published as a NodePort or cloud native LoadBalancer, but that is just a one time configuration.

• Components:

Ingress Controller -  Nginx(Supported), HAProxy, Traefik, GCP HTTPS LoadBalancer(GCE)(Supported), Contour, Istio
Ingress Resources: Rules

• Configuring Nginx Controller

apiVersion: extensions/v1beta1
kind: Deployment
metadata: 
   name: nginx-ingress-controller
spec:
   replicas: 1
   selector:
      matchLabels:
         name: nginx-ingress
   template:
      metadata:
         labels:
            name: nginx-ingress
      spec:
         containers:
            - name: nginx-ingress-controller
              image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
         args:
            - /nginx-ingress-controllee
            - --configmap=$(POD_NAMESPACE)/nginx-configuration                                ==> This is below created ConfigMap
         env:
           - name: POD_NAME
              valueFrom:
                  fieldRef:
                     fieldPath: metadata.name
           - name: POD_NAMESPACE
              valueFrom:
                  fieldRef:
                     fieldPath: metadata.namespace
         ports:
           - name: http
             containerPort: 80
           - name: https
             containerPort: 443

• In order to decouple Nginx Config data like: error-log-path, keep-alive, ssl-protocols, Config-Map object is created and passed.
• This blank will also do, because you dont need to change the nginx config files later on.

• Config Map:

kind: ConfigMap
apiVersion: v1
metadata:
   name: nginx-configuration

• Service to expose the ingress controller to the external world:

apiVersion: v1
kind: Service
metadata:
   name: nginx-ingress
spec:
   type: NodePort
   ports:
   - port: 80
     targetPort: 80
     protocol: TCP
     name: http
   - port: 443
     targetPort: 443
     protocol: TCP
     name: https
   selector:
     name: nginx-ingress

• Ingress Controllers monitor the kubernetes cluster for ingress resources and configure the underlying nginx server when something is changed.
• For this, a service account is needed which has - correct Roles, ClusterRoles & RoleBindings:

apiVersion: v1
kind: ServiceAccount
metadata:
   name: nginx-ingress-serviceaccount

• Therefore below objects are needed for a simple Ingress Controller:

Deployment
Service
ConfigMap
Auth

• Ingress Resources:

www.mysite.com/wear-------------->[wear-POD]
www.mysite.com/video------------->[video-POD]

wear.mysite.com------------------>[wear-POD]
video.mysite.com----------------->[video-POD]

nano ingress-wear.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: ingress-wear
spec:
   backend:
       serviceName: wear-service
       servicePort: 80

kubectl create -f ingress-wear.yaml
kubectl get ingress

• Using Paths:

nano ingress-wear-watch.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: ingress-wear-watch
spec:
   rules:
      - http:
          paths:
          - path: /wear
               backend:
                   serviceName: wear-service
                   servicePort: 80
          - path: /watch
               backend:
                   serviceName: watch-service
                   servicePort: 80

Check the Backends:

kubectl describe ingress ingress-wear-watch

• Using Host names:

nano ingress-wear-watch.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
   name: ingress-wear-watch
spec:
   rules:
     -  host: wear.mysite.com
        http:
          paths:
          -    backend:
                   serviceName: wear-service
                   servicePort: 80
     -  host: video.mysite.com
        http:
          paths:
          -    backend:
                   serviceName: watch-service
                   servicePort: 80

Network Policies

• Kubernetes is by default configured with "All Allow" rule.
• If you do not want Frontend POD to communicate directly with DB POD, need to create a Network Policy.
• Solutions that support Network Policy:

Kube-router
Calico
Romana
Weave-net

• Solutions that DO NOT support Network Policy(can create policy, will will not have any impact):

Flannel

• Applying a Network Policy:

nano policy-definition.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
   name: db-policy
spec:
   podSelector:
      matchLabels:
         role: db
   policyTypes:
   - Ingress
   ingress:
   - from:
     - podSelector:
         matchLabels:
            name: api-pod
     ports:
     - protocol: TCP
       port: 3306

kubectl create -f policy-definition.yaml

State Persistence

Volumes

• Docker container(& Kubernetes PODs) & its volumes are generally temporarily required.
• Container & volume is destroyed after termination.
• To persist Data, we attach a volume to the containers when they are created.

apiVersion:v1
kind: Pod
metadata:
   name: random-number-generator
spec:
   containers:
   - image: alpine
     name: alpine
     command: ["/bin/sh","-c"]
     args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
     volumeMounts:
     - mountPath: /opt
       name: data-volume
   volumes:
   - name: data-volume
   hostpath:
      path: /data                    ==> mounting host path directly is fine for single POD, but not for multiple PODs
      type: Directory

• For multiple PODs use storage solutions - NFS, GlusterFS, Flocker, ceph, Scaleio, AWS Elastic Block Storage, AZURE, GCP storage
• For AWS EBS:

   volumes:
   - name: data-volume
     awsElasticBlockStore:
       volumeID: <volume-id>
       fstype: ext4

Persistent Volumes

• With large environment, there is a need to configure a large number of volumes for each pod.
• For every change, they all need to be changed.
• Administrator can create a large pool of storage from where small volumes can be allocated to PODs.
• Persistent Volumes can help here, it is a cluster wide pool of storage volumes.

nano pv-definition.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
   name: pv-vol1
spec:
   accessModes:
      - ReadWriteOnce
   capacity:
      storage: 1Gi
   hostPath:
      path: /tmp/data

kubectl create -f pv-definition.yaml
kubectl get persistentvolume

• Access Modes:

 ReadOnlyMany
 ReadWriteOnce
 ReadWriteMany

Persistent Volume Claims

• PVC is used to make the volume available to the Nodes.
• PV & PVC are two separate objects in Kubernetes name space.
• Administrator creates PV & users creates PVCs.

• Create a PVC:

nano pvc-definition.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: myclaim
spec:
   accessModes:
      - ReadWriteOnce
   resources:
      requests:
         storage: 500Mi

kubectl create -f pvc-definition.yaml

kubectl get persistentvolumeclaim
     STATUS: Bound

Delete PVC:

kubectl delete persistenvolumeclaim myclaim

persistentVolumeReclaimPolicy: Retain         ==> other options are Delete, Recycle

Kubernetes Challenge Series

Challenge - 1 - Wordpress MySQL

References

{{#widget:DISQUS |id=networkm |uniqid=Kubernetes Deepdive |url=https://aman.awiki.org/wiki/Kubernetes_Deepdive }}