Managing cluster nodes configuration in Openshift v4

Openshift v4 introduces new set of APIs to manage cluster nodes configuration called Machine Config. Machine Config Pools manage a cluster of nodes and their corresponding Machine Configs. Machine Configs contains configuration information for a cluster including nodes configuration files. You can check what Machine Configs and Machine Config Polls exists in your cluster by calling:

$ oc get machineconfigpools
NAME     CONFIG                                                                      UPDATED   UPDATING   DEGRADED
master   rendered-master-e192851e43f1ab347b3a565c9c71d2b8   True      False      False
worker   rendered-worker-5c35596867d37b22ca2daac46351cda5   True      False      False

$ oc get machineconfig
NAME                                                      
00-master                                                  
00-worker                                                  
01-master-container-runtime                                
01-master-kubelet                                          
01-worker-container-runtime                                
01-worker-kubelet                                          
50-worker-container-registries                             
99-master-4d4106e2-8c0f-11e9-a9ed-02607282474a-registries  
99-master-ssh                                              
99-worker-4d668e54-8c0f-11e9-a9ed-02607282474a-registries  
99-worker-ssh                                              
rendered-master-e192851e43f1ab347b3a565c9c71d2b8           
rendered-worker-5c35596867d37b22ca2daac46351cda5           
rendered-worker-da1dff08dd891cb37c0ffc31e2276fe0     

By default you should see two Machine Config Pools for master and worker nodes and a bunch of Machine Configs in each of pools. At this time I encourage you to have a look at each of Machine Config to learn what configuration file they manage. For example:

$ oc describe machineconfig 01-worker-container-runtime

Nodes configuration is managed by Machine Config Operator. One important thing you should know is how Machine Configs are applied by the Operator to the nodes. The Machine Configs are read in order (from 00* to 99*). Labels inside the Machine Configs identify the type of node it belongs to (master or worker). If the same file appears in multiple Machine Config files, the last one wins. So, for example, any file that appears in a 99* file would replace the same file that appeared in a 00* file. The input Machine Config objects are unioned into a "rendered" Machine Config object, which will be used as a target by the operator and is the value you can see in the Machine Config Pool.

To see what files are managed from a Machine Config, look for “Path:” inside a particular Machine Config. For example:

$ oc describe machineconfigs 01-worker-container-runtime | grep Path:
            Path:            /etc/containers/registries.conf
            Path:            /etc/containers/storage.conf
            Path:            /etc/crio/crio.conf

Now lets try a simple example: I'd like to add quay.io image registry to list of search registires on my worker nodes. This configuration is stored in /etc/containers/registries.conf file. As you can see above this file is configured in  01-worker-container-runtime Machine Config object.

First thing you need to do is to create Machine Config object which will contain your augmented version of /etc/containers/registries.conf

cat <<EOF > 50-worker-container-registries.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-worker-container-registries
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
 source: data:,%5Bregistries.search%5D%0Aregistries%20%3D%20%5B'registry.access.redhat.com'%2C%20'docker.io'%2C%20'quay.io'%5D%0A%0A%5Bregistries.insecure%5D%0Aregistries%20%3D%20%5B%5D%0A%0A%5Bregistries.block%5D%0Aregistries%20%3D%20%5B%5D%0A
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.conf
EOF

As you can see content of registries.conf file data is url encoded. You can use any url encoding tool or online service to encode/decode your configuration files data. 

There are two important metadata in this object. First one is labels section where you specify to what Machine Config Pool this configuration should be added (master or worker). In our example this is machineconfiguration.openshift.io/role: worker. Second is name: 50-worker-container-registries which should start with higher number than Machine Config you want to override (if you want to override existing configuration file and not create new one).

Now you can create this Machine Config in your cluster:

$ oc create -f 50-worker-container-registries.yaml -n openshift-config

This should trigger automatically rolling upgrade of your worker nodes. You should see worker nodes being restarted one by one. You can also check if your Machine Config Pool is in updating status:

$ ./oc get machineconfigpools
NAME     CONFIG                                                                   UPDATED   UPDATING   DEGRADED
master   rendered-master-e192851e43f1ab347b3a565c9c71d2b8   True      False      False
worker   rendered-worker-5c35596867d37b22ca2daac46351cda5   True      True      False 

After your nodes are upgraded you can check if new configuration has been applied successfully:

$ oc get nodes

NAME                           STATUS                     ROLES          AGE   VERSION
ip-10-0-135-132.ec2.internal   Ready                      worker         15h   v1.13.4+cb455d664
ip-10-0-139-98.ec2.internal    Ready,SchedulingDisabled   worker         47h   v1.13.4+cb455d664
ip-10-0-140-77.ec2.internal    Ready                      infra,worker   46h   v1.13.4+cb455d664
ip-10-0-143-102.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664
ip-10-0-154-138.ec2.internal   Ready                      worker         47h   v1.13.4+cb455d664
ip-10-0-159-103.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664
ip-10-0-160-135.ec2.internal   Ready                      worker         47h   v1.13.4+cb455d664
ip-10-0-172-230.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664

$ oc debug node/ip-10-0-135-132.ec2.internal
 

Starting pod/ip-10-0-135-132ec2internal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
 

sh-4.2# chroot /host
 

sh-4.4# cat /etc/containers/registries.conf

[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'quay.io']

[registries.insecure]
registries = []

[registries.block]
registries = []

That's it! You have learned how to apply custom configuration to your Openshift v4 cluster node.

Getting started with Openshift v4 - part 2 - installing on vSphere

After successfull installation of Openshift v4 on AWS I had a opportunity to install Openshift v4 on vSphere. This time installation had to be done following User Provisioned Infrastructure procedure. This means before the cluster could be bootstrapped I had to prepare required infrastucture myself following the documentation. 

First thing you need to prepare is networking: 2 load balancers for masters and worker nodes and DNS configuration for your cluster.

Secondly you need to download and run openshift-install but this time only to generate your machines ignition config files. Don't forget to create install-config.yaml file first!

Last thing you need to do is to create virtual machines in your vSphere cluster. You'll need to download RHCOS OVA image and create 1 bootstrap, 3 masters and 3 worker machines.

Now you are almost done!

At this point I decided to slighty disobey documentation and boot seperately masters and worker nodes. In order to do that first I have started only bootstrap node and 3 master nodes and had to wait until bootstrap process has finished. This will take a while and you can either ssh to bootstrap machine and follow journal log, or call openshift-install with wait-for bootstrap-complete parameters to monitor bootstrap progress. 

To confirm master bootstrap is done you can call oc get nodes to verify if all 3 masters are in ready status.

$ oc get nodes

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  63m  v1.13.4+b626c2fe1
master-1  Ready     master  63m  v1.13.4+b626c2fe1
master-2  Ready     master  64m  v1.13.4+b626c2fe1

At this point you should delete bootstrap virtual machine and remove it from the load balancer.

Now you can simply start worker nodes (one by one, or in pararel) and wait for a while until they'll appear in the nodes list with status ready by calling again oc get nodes command.

Next thing you'll need to do is to approve pending csrs for your worker machines.

Last but not least don't forget to configure storage for you image registry operator.

That's it. Your Openshift v4 cluster is up and running on vSphere!

In part 3 I'll show you how to proceed with cluster configuration.

Getting started with Openshift v4 - part 1 - Installing on AWS

We've just announced release of new Red Hat Openshift 4, which will be available in a few weeks. However if you are interested in trying Openshift 4 eariler there is already beta version available for you. Please refer to Openshift 4 documentation for more details. 

Openshift 4 installation is currently possible only on AWS, vSphere and bare metal. Support for other platforms including Azure, GCP, OpenStack, Red Hat Virtualization will follow in subsequent minor releases within next few months.

I gave first try to install Openshift 4 on AWS with Installer Provisioned Infrastructure which means installer installed for me Openshift 4 cluster nodes as well as the platform itself.

I must admit this has been very pleasant experience. I had to only set up my AWS account, and register or transfer internet domain at Route53. The rest has been done automatically by the installer.

After about 30 mins I had my cluster of 3 masters and 3 worker nodes up and running.

What is equally important (a least for me) it was also very straightforward to uninstall the cluster using the same installer. In fact I've already installed and uninstalled the cluster a couple of times without any issues.

Nice!

In part 2 I'll share my experience with Openshift 4 installation on vSphere so stay tuned...

Becoming Red Hat Certified Architect in Enterprise Applications

Are you application architect or developer working with Red Hat JBoss, Openshift, J2EE, Microprofile or Camel technologies?  Would you like to become Red Hat Certified Architect in Enterprise Applications? You can achieve it by taking Enterprise Applications RHCA certification track (Go to "Cerification details" and select tab "For RHCEMDs & RHCJDs").

For example list of Openshift and JBoss exams you need to pass check my exams transcript.

Prune all evicted or failed Pods

Sometimes you might have a lot of pods in Evicted or other state which requires manual deletion by cluster administrator. This is how you can do that with single command:  

for n in $(oc get projects --no-headers=true | awk '{print $1}'); do echo $n; for pod in $(oc get pods -n $n --no-headers=true | grep "Evicted" | awk '{print $1}'); do oc delete pod ${pod} -n ${n}; done; done;

Similarly you can delete with single command all pods in CrashLoopBackOff state:

for n in $(oc get projects --no-headers=true | awk '{print $1}'); do echo $n; for pod in $(oc get pods -n $n --no-headers=true | grep "CrashLoopBackOff" | awk '{print $1}'); do oc delete pod ${pod} -n ${n}; done; done;

Automatic updates of Images

Openshift Container Platform has nice feature called Image Streams which aggregates all tags of container image in single object. This makes it easier to reference images from other configuration objects in Openshift i.e. Deployment Configs. Other nice feature of Image Steams is that you can schedule automatic updates of existing and new tags from Image Stream sourcing image registry. In order to make your Image Stream schedulable for updates you need to add --scheduled=true parameter to oc import-image command which creates new Image Stream from existing container image:

oc import-image ruby --from=registry.redhat.io/rhscl/ruby-25-rhel7 --confirm --all --scheduled=true

Image Streams update scheduler is enabled by default and is configured in master-config.yaml in imagePolicyConfig section:

imagePolicyConfig:
  MaxScheduledImageImportsPerMinute: 10
  ScheduledImageImportMinimumIntervalSeconds: 1800
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 3

You can adjust this configuration according to your needs but first remember to refer Openshift docs and restart master api and controller processes after you made any changes.

Garbage Collector

When your nodes are running for some time you might find a lot of terminated containers and container images unreferenced by any pods consuming your container storage. In Docker environment you would execute docker rm and docker rmi to get rid of this objects. In Openshift alternatively you could leverage built in Garbage Collector which can do this for you automatically in the background. 

Garbage Collector is enabled by default in Openshift however with no limits for number of terminated containers and very high limits for unreferenced images storage space. You can adjust this defaults by setting kubeletArguments section of the node-config.yaml: 

kubeletArguments:

  ...
  maximum-dead-containers-per-container:
  - '1'
  maximum-dead-containers:
  - '50'
  minimum-container-ttl-duration:
  - 10s
  image-gc-high-threshold:
  - '70'
  image-gc-low-threshold:
  - '60'

You could learn more about this useful functionality in Openshift docs.