Pod priorities

As number of apps running on Openshift is growing you might want to set priorities for one apps over other to instruct scheduler which apps should be deployed first and evicted last.

In Openshift Conatiner Platform 3.11 new feature Pod Priority and Preemption has been introduced to solve this challenge. It is quite simple: first you can create your own priority classes or use default ones. Secondly you need to configure your pod with class you want to use:

apiVersion: v1
kind: Pod

...

spec:
  containers:
  ...

  priorityClassName: high-priority

Check this docs for more details how to set pod priorities up.

Setting quotas for users (tenants)

One of the first things you should do after you install Openshift cluster is to setup resource quotas in order to avoid creative users exhausting your cluster compute resources instantly. In Openshift you can leverage Cluster Resource Quota which allows you to create multi project quotas for example for different users (tenants). Here is how I'm typically doing that:

First create Cluster Resource Quota files. I typically create 3 seperate files for non-terminating compute resources, terminating compute resources and storage resources. When this files are in place execute following commands to create user (tenant) quota. From now on each project created by the user (tenant) will be covered by his quota.

Last two things you should do to get this working is to setup limit range object in each project your user (tenant) will create and create maximum duration for builds executed by your user (tenant).


Best way to setup limit range is to create default project template  which contains limit range object as described here.

In order to set maximum build duration please follow docs here.

Speeding up s2i builds

I hear quite offen complaints about slow performance of s2i builds (especially Java). In most cases this is beacause you rely on default build pod resource limits. Here you can read how to set this limits. I recommend to set CPU limit to at least 1 core:

spec:
  resources:
    limits:
      cpu: "1" 
      memory: "512Mi"

Deleting pods stuck in Terminating state for a long time

From time to time due to some edge case you might find pods stuck in Terminating state for a long time, and Openshift (kubelet) trying constanty to delete them without success. First of all in that case I stongly encourage you to contact Red Hat Support for assistance. Nevertheless here is the solution which should help you to get rid of this pods.

First check with docker ps if there are no containers running that belongs to your terminating pod on the node where your pod was scheduled. If you'll find any delete them, and check if this will solve the issue.

If this is not a case in Openshift 3.10+ please rsh to etcd pod. On earliar Openshift versions you'll need to use etcdctl cli installed on master nodes.

oc project kube-system

oc rsh master-etcd-ip-10-0-2-82.ec2.internal

Find your pod entry in etcd database:

source /etc/etcd/etcd.conf
export ETCDCTL_API=3
etcdctl --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --cacert=/etc/etcd/ca.crt --endpoints=https://$(hostname):2379 get / --prefix --keys-only | grep sso-1-qxvlv

You should receive path to your pod entry:

/kubernetes.io/pods/sso/sso-1-qxvlv

Finally delete pod entry:

etcdctl --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --cacert=/etc/etcd/ca.crt --endpoints=https://$(hostname):2379 del --prefix /kubernetes.io/pods/sso/sso-1-qxvlv

In reply you should receive number of affected records:

1

That's it. Your terminating pod has been deleted.

Running container processes as root in Openshift

Containers running processes as root user are essentially bad thing, which might open attack surface. But still I see many containers running this way. Openshift will block by default execution of containers running processes as root user. Nevertheless if this is really what you want to do this is how you can do it the right way: https://gist.github.com/jstakun/b4d9c5ed567433ecd84fb203c2a85d96

If you want to learn more about running container processes as root user in Openshift check this blog post: https://blog.openshift.com/getting-any-docker-image-running-in-your-own-openshift-cluster/