Deploy applications with Helm 3 to OpenShift

Helm is a package manager for Kubernetes which helps users create templated packages called Helm Charts to include all Kubernetes resources that are required to deploy a particular application. Helm then assists with installing the Helm Chart on Kubernetes, and afterwards it can upgrade or rollback the installed package when new versions are available. Helm Charts are particularly useful for installation and upgrade of stateless applications given that the Kubernetes resources and the application image can simply be updated to newer versions. 

Helm 2 was based on a server-side component named Tiller which was responsible for performing Helm operations on Kubernetes clusters. Tiller was designed prior to Kubernetes role-based access control (RBAC) and although useful for single-tenant clusters, its permissive configuration could grant users a wide array of unintended permissions. Therefore it was recognised as a major security concern on multi-tenant clusters, which prevented many enterprise users from using Helm in production environments. OpenShift is an enterprise Kubernetes platform, and therefore we didn’t recommend the use of Helm 2 in production, even though it was possible to disable OpenShift security features in order for Helm 2 to be used on OpenShift.

Helm 3 was recently released as GA in the Helm community, and a major update has been removing Tiller and pivoting to a client-side architecture to address the aforementioned security concerns, removing the barrier tor using Helm in enterprise environments.

Starting from OpenShift 4.3 you can download Helm 3 via OpenShift Web Console or directly from our mirror as per documentation

Nevertheless if you are using earlier version of OpenShift or would like to try latest Helm 3 version you can do it easly.

$ #Download latest Helm 3 from https://github.com/helm/helm/releases 

$ tar -xvf helm-v3*.tar.gz 

$ mv helm /usr/local/bin
$ helm version

$ helm repo add stable https://kubernetes-charts.storage.googleapis.com

Once Helm 3 is up and running on your workstation let's deploy sample tomcat Chart to OpenShift.

The easiest way would be to install it directly to OpenShift using helm install command but this will fail due to OpenShift Security Context Constrains configuration which prohibits deployments of pods exposing host ports. Of course we can disable this configuration, but I don't want to introduce unnecessary security vulnerability to my OpenShift cluster. Hence first we need to pull and modify tomcat Chart locally:

$ helm pull stable/tomcat
$ sed -i "s/hostPort/#hostPort/g" ./tomcat/values.yaml

Now we can install tomcat chart to OpenShift:

$ oc new-project helm-tomcat-demo
$ helm install ./tomcat --generate-name

After a while you should see tomcat pod up ad running. Since this chart doesn't define ingress you can easily create OpenShift Route to access the tomcat sample application from outside of the cluster:

$ oc expose svc $(oc get svc --no-headers | awk '{print $1}')
$ curl $(oc get route --no-headers | awk '{print $2'})/sample/

Finally you can also use Helm to uninstall your tomcat Chart deployment:

$ helm uninstall $(helm ls | grep tomcat | awk '{print $1}')
$ oc delete project helm-tomcat-test

Learn more about OpenShift 4.3 and Helm 3 integration here.

Limiting the Bandwidth Available to Pods

Using Kubernetes Resource Quotas you can limit number of Kubernetes objects, compute resources (CPU, RAM memory) and Persistent Volumes storage size in the project or multiple projects as per documentation. However with Resource Quotas you can't set ingress and egress traffic bandwidth quotas on a pod level. In this blog post I'll explain how you can do that in OpenShift 3.x or 4.x and Kubernetes. Remember this fuctionality must be supported by your SDN network plugin and is required for CNI networking plugins.

 

You can apply quality-of-service traffic shaping to a pod and effectively limit its available bandwidth. Egress traffic (from the pod) is handled by policing, which simply drops packets in excess of the configured rate. Ingress traffic (to the pod) is handled by shaping queued packets to effectively handle data. The limits you place on a pod do not affect the bandwidth of other pods.

To limit the bandwidth on a pod:

1. Write an object definition and specify the data traffic speed using kubernetes.io/ingress-bandwidth and kubernetes.io/egress-bandwidth annotations. For example, to limit both pod egress and ingress bandwidth to 10M/s in pod definition:

{
    "kind": "Pod",
    "spec": {
        "containers": [
            {
                "image": "openshift/hello-openshift",
                "name": "hello-openshift"
            }
        ]
    },
    "apiVersion": "v1",
    "metadata": {
        "name": "iperf-slow",
        "annotations": {
            "kubernetes.io/ingress-bandwidth": "10M",
            "kubernetes.io/egress-bandwidth": "10M"
        }
    }
}

or in OpenShift deployment config definition in template section:

{
    "apiVersion": "apps.openshift.io/v1",
    "kind": "DeploymentConfig",
    ...
    "spec": {
        ...
        "template": {
            "metadata": {
                "annotations": {
                    "kubernetes.io/egress-bandwidth": "10M",
                    "kubernetes.io/ingress-bandwidth": "10M",
                    "openshift.io/generated-by": "OpenShiftWebConsole"
                },
                ...
             },
             ...
         },
         ...
    },
    ...
}

2. Create the pod using the object definition:

oc create -f <file_or_dir_path>

Quarkus - game changer for Java developers

Recently I came across Quarkus framework. With this framework you can significantly (even in order of magnitude) reduce Java application startup time and RAM memory consumption. This makes it very attractive for running Java applications in containers on Kubernetes or OpenShift. 

In Quarkus Java applications can be executed in OpenJDK JVM. However in order to get highest memory and startup time reduction you should compile your Java application code into native executable using GraalVM. 

Here is example based on OpenJDK s2i image how you can build Quarkus Java application image:

# In this example I'm using OpenJDK 11 image from Red Hat image registry which requires authentication, hence in first step you might need to get your image pull secret. In general you'll need s2i OpenJDK image containing Maven >= 3.5.3 otherwise build might fail.


# get image pull secret from 

# https://access.redhat.com/terms-based-registry/#/accounts 

# and save it to my-pull-secret.yaml file

$ oc create -f my-pull-secret.yaml
$ oc secrets link builder my-pull-secret

$ oc new-app \

registry.redhat.io/openjdk/openjdk-11-rhel8~https://github.com/jstakun/hello-quarkus.git --name=java-quarkus

# or first import OpenJDK image to your cluster
$ oc import-image registry.redhat.io/openjdk/openjdk-11-rhel8 --confirm \

--all=true -n openshift

$ oc new-app \

openshift/openjdk-11-rhel8~https://github.com/jstakun/hello-quarkus.git --name=quarkus-java

Here is single command example for building Native executable for the same Java application:

$ oc new-app \

quay.io/quarkus/ubi-quarkus-native-s2i:19.2.1~https://github.com/jstakun/hello-quarkus.git --name=native-quarkus

This command produces very large image:

$ sudo podman images
REPOSITORY                                                                                         TAG                                                                       IMAGE ID       CREATED          SIZE
image-registry.openshift-image-registry.svc:5000/my-quarkus-app   sha256:4e0feed735fdee1403b8f23a35ad0b9b0ae42baec41e324824f72fd45af2a424   ab444dc9a48f   3 minutes ago    1.38 GB

If you would like to build much smaller image you can build native executable locally and then use s2i binary build with Universal Base Image minimal version:

#Build native executable locally:
$MAVEN_OPTS="-Xmx4G -Xss128M \
-XX:MetaspaceSize=1G -XX:MaxMetaspaceSize=2G  \
-XX:+CMSClassUnloadingEnabled" 
mvn clean package -Pnative -DskipTests 

#create binary s2i build
$ oc new-build --name=hello-quarkus \
--dockerfile=$'FROM registry.access.redhat.com/ubi7/ubi-minimal:latest\nCOPY *-runner /application\nRUN chgrp 0 /application && chmod +x /application\nCMD /application\nEXPOSE 8080'


#--from-file specifies native executable file created above
$ oc start-build hello-quarkus --from-file=/projects/hello-quarkus/target/hello-1.0.0-SNAPSHOT-runner 

#run pod after image build finish
$ oc new-app hello-quarkus

This should produce much smaller image:

$ sudo podman images
REPOSITORY                                                                                         TAG                                                                       IMAGE ID       CREATED          SIZE
image-registry.openshift-image-registry.svc:5000/my-quarkus-app/hello-quarkus-ubi-minimal                  sha256:169396aa1199bcf7d8bfab444357ccffe35a01f366a572e3f1486a0214271c35   bc050c6c6ccd   27 minutes ago   129 MB

You can try to decrease size of your image even further using Father Linux UBI micro image which I built and pushed to my quay registry:

$ oc new-build --name=hello-quarkus --dockerfile=$'FROM quay.io/jstakun/ubi8-micro:0.1\nCOPY *-runner /application\nRUN mkdir /vertx && chgrp -R 0 /vertx && chmod -R g=u /vertx && chgrp 0 /application && chmod +x /application\nCMD /application -Djava.io.tmpdir=/vertx\nEXPOSE 8080'

This should produce even smaller image:

$ sudo podman images
REPOSITORY                                                                                         TAG                                                                       IMAGE ID       CREATED          SIZE
image-registry.openshift-image-registry.svc:5000/my-quarkus-app/hello-quarkus-ubi-micro                      sha256:a2f6e6c81487ccf174f568223e65477c07736370331d77cebd1122c862eddd33   5044755e8456   32 minutes ago   91.2 MB 

If you want to try to build smallest image you can try to do that from the scratch and add only libraries which are referenced by your executable and sh:

#check what libraries are referenced by your native executable:
$ ldd /projects/hello-quarkus/target/hello-1.0.0-SNAPSHOT-runner
        linux-vdso.so.1 (0x00007ffc25b54000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f298e977000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f298e757000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f298e553000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f298e33c000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f298e133000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f298dd70000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f298ecf9000)

#and sh
$ ldd /bin/sh
    linux-vdso.so.1 =>  (0x00007ffc4dfb4000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f0ec5407000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0ec5203000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0ec4e36000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0ec5631000)

#build container from the scratch using buildah
container=$(buildah from scratch)
mnt=$(buildah mount $container)
mkdir $mnt/bin
mkdir $mnt/lib64
buildah config --workingdir /bin $container
buildah copy $container /projects/hello-quarkus/target/hello-1.0.0-SNAPSHOT-runner /bin/application
buildah copy $container /bin/sh /bin/sh
buildah copy $container /lib64/libtinfo.so.5 /lib64
buildah copy $container    /lib64/ld-linux-x86-64.so.2 /lib64
buildah copy $container /lib64/libm.so.6 /lib64
buildah copy $container /lib64/libpthread.so.0 /lib64
buildah copy $container /lib64/libdl.so.2 /lib64
buildah copy $container /lib64/libz.so.1 /lib64
buildah copy $container /lib64/librt.so.1 /lib64
buildah copy $container /lib64/libc.so.6 /lib64      
buildah copy $container /lib64/ld-linux-x86-64.so.2 /lib64
buildah config --port 8080 $container
buildah config --entrypoint /bin/application $container
buildah commit --format docker $container hello-quarkus-minimal:latest

$ podman images
REPOSITORY                                    TAG      IMAGE ID       CREATED         SIZE
localhost/hello-quarkus-minimal                             latest   2dd6b83e8432   8 seconds ago   28 MB
 
#run container
$ podman run localhost/hello-quarkus-minimal:latest -d
2019-12-06 08:48:11,158 INFO  [io.quarkus] (main) hello 1.0.0-SNAPSHOT (running on Quarkus 1.0.0.Final) started in 0.009s. Listening on: http://0.0.0.0:8080
2019-12-06 08:48:11,158 INFO  [io.quarkus] (main) Profile prod activated.
2019-12-06 08:48:11,158 INFO  [io.quarkus] (main) Installed features: [cdi, resteasy]

Now check startup time in the logs and RAM memory consumption for both running pods. You should see big difference. If you compare with simplest Spring Boot example difference should be even bigger in terms of startup time, RAM consumption and image size.

Quarkus is community project but soon will get productized as part of OpenShift Container Platform. Enjoy!

Resize storage in your CodeReady Containers virtual machine

CodeReady Containers (CRC) provides a pre-built development environment based on Red Hat Enterprise Linux and OpenShift Container Platform for quick container-based application development which you can run on your workstation. After installation you can easly configure CRC virtual machine using crc config command to adjust CPU and memory settings according to your needs. However if you would like to resize storage in your CRC virtual machine (which is by default only 30 GB) you'll need to use external tools. Here is how I did it on my RHEL workstation with KVM virtualization using gparted live iso:

1. Increase CRC virtual machine disk size on your host.

For example add additonal 10 GB storage to CRC virtual machine:

$ qemu-img resize ~/.crc/machines/crc/crc +10G

Check actual virtual disk size:

$ qemu-img info ~/.crc/machines/crc/crc | grep 'virtual size'

Now your virtual disk size is increased but you need to resize filesytem inside your CRC Virtual machine.

If you are running CRC 1.6+ it is as easy as:

$ crc start

$ ssh -i /home/jstakun/.crc/machines/crc/id_rsa core@192.168.130.11

$ sudo xfs_growfs /sysroot

$ df -h

If you are running CRC 1.5 or earlier here is my procedure:

2. Download gparted live iso from https://gparted.org/download.php

3. Configure gparted live iso as cd-rom device in you CRC virtual machine

4. Set gparted live iso cd-rom device as first on top in Boot Options

 
5. Boot CRC virtual machine

6. Click 4 times enter until you open gparted GUI as below

7. Select /dev/vda3 partion on the list. Click Resize/Move the selected partition button in the Toolbar on top. Resize partition using unallocated disk space as below

8. Shutdown virtual machine

9. Move gparted live iso cd-rom device down in Boot Options

9. Start your CRC virtual machine with crc start command
 
Now you can ssh to your CRC virual machine and check that partiton size has been increased!

$ ssh -i ~/.crc/machines/crc/id_rsa core@192.168.130.11
Red Hat Enterprise Linux CoreOS 42.80.20191010.0
---
[core@crc-847lc-master-0 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         12G     0   12G   0% /dev
tmpfs            12G   84K   12G   1% /dev/shm
tmpfs            12G   28M   12G   1% /run
tmpfs            12G     0   12G   0% /sys/fs/cgroup
/dev/vda3        40G   27G   14G  66% /sysroot
/dev/vda2       976M   76M  833M   9% /boot

Switching context between clusters using oc command line tool

When using Openshift oc command line tool you might want to connect to multiple Openshift clusters over the time. In order to easily switch between different clusters you can use oc config context subcommands. 

Let's create two contexts for different Openshift clusters. First context for AWS cluster:

$ oc login -u admin http://my_aws_cluster

$ oc config rename-context $(oc config current-context) aws

and second one for Openshift Online Pro cluster:

$ oc login -u my_oso_user https://api.pro-us-east-1.openshift.com

$ oc config rename-context $(oc config current-context) oso
 

Now we've got 2 oc contexts aws and oso. Contexts are saved in kubeconfig file (typically ~/.kube/config). You can check what contexts are currently set in your environment using command:

$ oc config get-contexts

You can switch between contexts using oc config use-context command. For example let's switch to oso context:

$ oc config use-context oso

Now we can check what is our current user and project we are in:

$ oc whoami && oc project -q
my_oso_user
my_oso_project

You can find more subcommands to manage contexts using command:

$ oc config --help

Context management commands will work in oc command line tools for both Openshift 3 and Openshift 4 versions.

Alerting on ElasticSearch data in Openshift

Openshift Container Platform contains built in container log aggregation service based on ElasticSearch Kibana and Fluentd (EFK) stack. However it lacks built in framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. In order to deliver this functionality I've created container images based on this Dockerfile and application template to deploy it in just a few steps to Openshift. For detailed instructions on how to deploy this template to Openshift please refer to my git repo. 

This template has been tested on Openshift 4.1, 4.2 and 4.4

Etcd cluster database recovery

Etcd is cluster database in Kubernetes and Openshift. This is critical component to keep your cluster up and running. By design it is fault tolerant, but of course some failures might require administator intervention. In Openshift Etcd is running on master nodes collocated with api servers and controller.

In order to keep Etcd fully operational you need to have more than half cluster members running. If less than half Etcd cluster members are running cluster will switch to read only mode and in practice you won't be able to manage your cluster with kubectl/oc or web/admin console. In this situation you must recover cluster nodes or add new cluster members to have more than half cluster members running.

Here are Etcd failure scenarios and how you can recover in Openshift environment:

• Majority masters are up, minority master are down.

When failed masters comes back they will automatically recover and join the cluster.

• Network partition.

If there is any side where majority is running this side will remain fully operational. Once the network partition clears, the minority side automatically recognizes the leader from the majority side and recovers its state.

• Minority masters are up, majority masters are down.

First create a single-node etcd cluster following Restoring etcd quorum for static pods procedure.

Secondly add more cluster members following Adding etcd nodes after restoring procedure. 3 master clusters are recommended.

• All masters are down.

If all masters are down first you should check if you can get any of then up and running reasonably quickly. If yes you can proceed with scenario 3. From my experience if there is no file system corruption this should work pretty well in most cases. Otherwise you'll need to recover your Etcd cluster from the backup:

First Restore etcd from snapshot which will create single-node etcd cluster.

Secondly add more cluster members following Adding etcd nodes after restoring procedure. 3 master clusters are recommended.

---

Read more about Etcd failures.