Create aws private eks cluster using eksctl commands

$ eksctl create cluster — name <cluster name> — region <your region> — vpc-private-subnets=subnet-xxxxxxxx,subnet-xxxxxxxxxxxx — node-private-networking — version 1.xx — nodegroup-name standard-workers — node-type t3.medium — nodes 3 — nodes-min 1 — nodes-max 4 — node-ami auto

Readings,

https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html

https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html

https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

https://www.bluematador.com/blog/my-first-kubernetes-cluster-a-review-of-amazon-eks

https://eksctl.io/usage/iam-policies/

How to setup rolling update in kubernetes deployments

Please note that below rolling update exercise is only applicable if you maintaining container image versioning for each deployment. That means you cannot perform rolling update for two deployments with same version number.

If you are looking for a way to do this while keeping the same version number, please check this post J

First we need add rolling update strategy to the deployment yaml.

Below is the required part to be added to the yaml file,

strategy:

    type: RollingUpdate

    rollingUpdate:

      maxSurge: 1 # this is limit to create extra pods while rolling update

      maxUnavailable: 1 # this is the limit that can unavailable during rolling update

   minReadySeconds: 25 # this is the minimum time to start your application

Below is a sample deployment yaml file with above strategy,

apiVersion: apps/v1

kind: Deployment

metadata:

  name: test

  labels:

    app: test

spec:

  replicas: 2

  selector:

    matchLabels:

      app: test

   strategy:

    type: RollingUpdate

    rollingUpdate:

      maxSurge: 1

      maxUnavailable: 1

   minReadySeconds: 25

  template:

    metadata:

      labels:

        app: test

        version: v1

    spec:

      imagePullSecrets:

      - name:

      containers:

      - name: test

        image: test:1.0.0  # different versioning number is a must to perform rolling update    

        ports:

          - name:

            containerPort: 8080

Now apply the deployment with –record flag.

$ kubectl.exe apply -f <your deployment name>.yml --record

Now you can run below command to start rolling update. You must mention a different version number here.

$ kubectl set image deployment test test=test:1.0.1 --record

Now you can check rollout status using below command,

$ kubectl.exe rollout status deployment <your deployment name>

Also you can check rollout history by using below command,

$ kubectl.exe rollout history deployment <your deployment name>

You can use below commands to undo roll outs,

Undo to previous deployment,

$ kubectl rollout undo deployment <your deployment name>

Undo to a specific roll out revision,

$ kubectl rollout undo deployment <deployment> --to-revision=<revision Number that you get from rollout history command (1,2,…)>

References,

https://tachingchen.com/blog/kubernetes-rolling-update-with-deployment/

https://blog.sebastian-daschner.com/entries/zero-downtime-updates-kubernetes

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/

https://medium.com/platformer-blog/enable-rolling-updates-in-kubernetes-with-zero-downtime-31d7ec388c81

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#rolling-update

Kubernetes Rolling update / restart without changing the version of pods

With the default kubernetes we cannot maintain rolling update while keeping the same container version but there is a workaround for this,

This can be done by adding a patch label to the deployment. Once the patch updated, pods getting rolling restart by pulling images,

$ kubectl patch deployment <your deployment name> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<your container name","env":[{"name":"LAST_ROLLOUT","value":"'$(date +%s)'"}]}]}}}}'

You can test your deployment in kubernetes for the new label,

$ kubectl.exe get deployment <your deployment name> -o yaml

spec:

  template:

...

    spec:

      containers:

      - env:

        - name: LAST_ROLLOUT

          value: "1553772196"

       

You can get pods list to check whether new ones created and existing ones terminated,

$ kubectl.exe get pods

NAME                                           READY     STATUS             RESTARTS   AGE

test-776885db4c-rdb6j                  2/2       Running            0          5h

test-84fdbf5cc8-wv4qv                  0/2       PodInitializing    0          4s

$ kubectl.exe get pods

NAME                                           READY     STATUS             RESTARTS   AGE

test-776885db4c-rdb6j                  0/2       Terminating        0          5h

test-84fdbf5cc8-wv4qv                  2/2       Running              1          40s

Now you can check rollout status using below command,

$ kubectl.exe rollout status deployment <your deployment name>

deployment "<your deployment name>" successfully rolled out

Also you can check rollout history by using below command,

$ kubectl.exe rollout history deployment <your deployment name>

deployments "<your deployment name>"

REVISION  CHANGE-CAUSE

1         kubectl.exe patch deployment <your deployment name> --patch={"spec":{"template":{"spec":{"containers":[{"name":"<your deployment name>","env":[{"name":"LAST_ROLLOUT","value":"1553772196"}]}]}}}}

You can use below commands to undo roll outs,

Undo to previous deployment,

$ kubectl rollout undo deployment <your deployment name>

Undo to a specific roll out revision,

$ kubectl rollout undo deployment <deployment> --to-revision=<revision Number that you get from rollout history command (1,2,…)>

References,

https://github.com/kubernetes/kubernetes/issues/13488

https://github.com/kubernetes/kubernetes/issues/27081

How to set istio ingress gateway to an application to access from outside the network

To see current gateways and their ips with ports,

# kubectl get svc istio-ingressgateway -n istio-system

Below is the network traffic plan for the application via istio-system,

Client/Browser à http://<Istio ingressgateway External IP> :< gateway port>/<application URL> à Gateway (istio) à VirtualService(istio) à Service(k8s) à Deployment(Pods)

First we need to apply our deployment,

Below is a basic deployment.yaml file content,

apiVersion: apps/v1

kind: Deployment

metadata:

  name: test

  labels:

    app: test

spec:

  replicas: 1

  selector:

    matchLabels:

      app: test

  template:

    metadata:

      labels:

        app: test

        version: v1

    spec:

      containers:

      - name: test

        image: <image location>test:latest   

        ports:

          - name: test

            containerPort: 80 # this is the application port exposing via pod

Use below command to apply the deployment,

#  kubectl.exe deploy -f <your deployement file>.yml

Now you need to apply service.yaml to create a service with clusterIP,

apiVersion: v1

kind: Service

metadata:

  name: test

  labels:

    app: test

spec:

  ports:

  - name: http

    protocol: TCP

    port: 80

  selector:

    app: test

Use below command to apply the service,

#  kubectl.exe deploy -f <your service file>.yml

Now you need to create virtualservice to send the traffic to the service created above,

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: test

spec:

  hosts:

  - "*"

  gateways:

  - test-gateway #this is the gateway referring to get the traffic

  http:

  - match:

    - uri:

        exact: /test-service/getall

    - uri:

        exact: /login

    - uri:

        exact: /logout

    route:

    - destination:

        host: test

        port:

          number: 80

Use below command to apply the virtual service,

#  kubectl.exe deploy -f <your virtual service file>.yml

Finally you need to create a gateway to get the traffic from outside world to send the traffic between virtual services,

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: test-gateway

spec:

  selector:

    istio: ingressgateway # this is the default selector

  servers:

  - port:

      number: 80

      name: http

      protocol: HTTP

    hosts:

    - "*"

That’s it. Now you have ingress traffic path to your application cluster.

Use below commands to check created resources,

#  kubectl.exe get deployments

#  kubectl.exe get services

#  kubectl.exe get virtualservices

#  kubectl.exe get gateways

Now you can access created application using istio-ingressgateway exteranal IP,

http://<Istio ingressgateway External IP> :< gateway port>/<application URL>

Ex. According to above sample deployments,

http:// ://<Istio ingressgateway External IP> :80/test-service/getall

How to view kubernetes container/pod logs in an istio installed k8s cluster

Error

$ kubectl logs -f <pod name>

Error from server (BadRequest): a container name must be specified for pod <pod name>, choose one of: [<pod name> istio-proxy] or one of the init containers: [istio-init]

Solution

$ kubectl logs -f <pod name> -c <container name>

*** Most of the time container name is same as the pod name without hash value.

How to copy kubernetes secrets & configmaps between namespaces and between clusters

Kubernetes Namespaces provide a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. Namespaces are a way to divide cluster resources between multiple users.

Kubernetes Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a Secret object allows for more control over how it is used, and reduces the risk of accidental exposure.

The ConfigMap API resource provides mechanisms to inject containers with configuration data while keeping containers agnostic of Kubernetes. ConfigMap can be used to store fine-grained information like individual properties or coarse-grained information like entire config files or JSON blobs (https://unofficial-kubernetes.readthedocs.io/en/latest/tasks/configure-pod-container/configmap/).

Copy secrets between namespaces

$ kubectl get secret <secret-name> --namespace=<source-namespace> --export -o yaml |\  kubectl apply --namespace=<destination-namespace> -f -

Copy configmaps between namespaces

$ kubectl get configmap <configmap-name> --namespace=<source-namespace> --export -o yaml |\  kubectl apply --namespace=<destination-namespace> -f -

Copy secrets between kubernetes clusters

First log in to the source kubernetes cluster.

Then run below command to export required secret to a yaml file,

$ kubectl get secret <secret-name> -n <source-namespace>  --export -o yaml > <secret-name>.yml

You can view the content of the file using below command,

$ cat <secret-name>.yml

Now you can easily deploy exported secret to another kubernetes cluster,

$ kubectl apply --namespace=<destination-namespace> –f <secret-name>.yml

Copy configmaps between kubernetes clusters

First log in to the source kubernetes cluster.

Then run below command to export required configmap to a yaml file,

$ kubectl get configmap <configmap-name> -n <source-namespace> --export -o yaml > <configmap-name>.yml

You can view the content of the file using below command,

$ cat <configmap-name>.yml

Now you can easily deploy exported configmap to another kubernetes cluster,

$ kubectl apply --namespace=<destination-namespace> –f <configmap-name>.yml

ERROR: SECRETS "KIALI" NOT FOUND

Error: secrets "kiali" not found

add below ,

$ echo -n 'admin' | base64
YWRtaW4=
$ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: $NAMESPACE
  labels:
    app: kiali
type: Opaque
data:
  username: YWRtaW4=
  passphrase: MWYyZDFlMmU2N2Rm
EOF

Source:

https://github.com/istio/istio/blob/44798c747a5a4483bafc810c35e568494d78cc99/install/kubernetes/helm/istio/README.md