czwartek, 25 października 2018

Speeding up s2i builds

I hear quite offen complaints about slow performance of s2i builds (especially Java). In most cases this is beacause you rely on default build pod resource limits. Here you can read how to set this limits. I recommend to set CPU limit to at least 1 core:

spec:
  resources:
    limits:
      cpu: "1" 
      memory: "512Mi"

środa, 10 października 2018

Deleting pods stuck in Terminating state for a long time

From time to time due to some edge case you might find pods stuck in Terminating state for a long time, and Openshift (kubelet) trying constanty to delete them without success. First of all in that case I stongly encourage you to contact Red Hat Support for assistance. Nevertheless here is the solution which should help you to get rid of this pods.

First check with docker ps if there are no containers running that belongs to your terminating pod on the node where your pod was scheduled. If you'll find any delete them, and check if this will solve the issue.

If this is not a case in Openshift 3.10+ please rsh to etcd pod. On earliar Openshift versions you'll need to use etcdctl cli installed on master nodes.

oc project kube-system

oc rsh master-etcd-ip-10-0-2-82.ec2.internal

Find your pod entry in etcd database:

source /etc/etcd/etcd.conf
export ETCDCTL_API=3
etcdctl --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --cacert=/etc/etcd/ca.crt --endpoints=https://$(hostname):2379 get / --prefix --keys-only | grep sso-1-qxvlv

You should receive path to your pod entry:

/kubernetes.io/pods/sso/sso-1-qxvlv

Finally delete pod entry:

etcdctl --cert=/etc/etcd/peer.crt --key=/etc/etcd/peer.key --cacert=/etc/etcd/ca.crt --endpoints=https://$(hostname):2379 del --prefix /kubernetes.io/pods/sso/sso-1-qxvlv

In reply you should receive number of affected records:

1

That's it. Your terminating pod has been deleted.