Managing cluster nodes configuration in Openshift v4

Openshift v4 introduces new set of APIs to manage cluster nodes configuration called Machine Config. Machine Config Pools manage a cluster of nodes and their corresponding Machine Configs. Machine Configs contains configuration information for a cluster including nodes configuration files. You can check what Machine Configs and Machine Config Polls exists in your cluster by calling:

$ oc get machineconfigpools
NAME     CONFIG                                                                      UPDATED   UPDATING   DEGRADED
master   rendered-master-e192851e43f1ab347b3a565c9c71d2b8   True      False      False
worker   rendered-worker-5c35596867d37b22ca2daac46351cda5   True      False      False

$ oc get machineconfig
NAME                                                      
00-master                                                  
00-worker                                                  
01-master-container-runtime                                
01-master-kubelet                                          
01-worker-container-runtime                                
01-worker-kubelet                                          
50-worker-container-registries                             
99-master-4d4106e2-8c0f-11e9-a9ed-02607282474a-registries  
99-master-ssh                                              
99-worker-4d668e54-8c0f-11e9-a9ed-02607282474a-registries  
99-worker-ssh                                              
rendered-master-e192851e43f1ab347b3a565c9c71d2b8           
rendered-worker-5c35596867d37b22ca2daac46351cda5           
rendered-worker-da1dff08dd891cb37c0ffc31e2276fe0     

By default you should see two Machine Config Pools for master and worker nodes and a bunch of Machine Configs in each of pools. At this time I encourage you to have a look at each of Machine Config to learn what configuration file they manage. For example:

$ oc describe machineconfig 01-worker-container-runtime

Nodes configuration is managed by Machine Config Operator. One important thing you should know is how Machine Configs are applied by the Operator to the nodes. The Machine Configs are read in order (from 00* to 99*). Labels inside the Machine Configs identify the type of node it belongs to (master or worker). If the same file appears in multiple Machine Config files, the last one wins. So, for example, any file that appears in a 99* file would replace the same file that appeared in a 00* file. The input Machine Config objects are unioned into a "rendered" Machine Config object, which will be used as a target by the operator and is the value you can see in the Machine Config Pool.

To see what files are managed from a Machine Config, look for “Path:” inside a particular Machine Config. For example:

$ oc describe machineconfigs 01-worker-container-runtime | grep Path:
            Path:            /etc/containers/registries.conf
            Path:            /etc/containers/storage.conf
            Path:            /etc/crio/crio.conf

Now lets try a simple example: I'd like to add quay.io image registry to list of search registires on my worker nodes. This configuration is stored in /etc/containers/registries.conf file. As you can see above this file is configured in  01-worker-container-runtime Machine Config object.

First thing you need to do is to create Machine Config object which will contain your augmented version of /etc/containers/registries.conf

cat <<EOF > 50-worker-container-registries.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-worker-container-registries
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
 source: data:,%5Bregistries.search%5D%0Aregistries%20%3D%20%5B'registry.access.redhat.com'%2C%20'docker.io'%2C%20'quay.io'%5D%0A%0A%5Bregistries.insecure%5D%0Aregistries%20%3D%20%5B%5D%0A%0A%5Bregistries.block%5D%0Aregistries%20%3D%20%5B%5D%0A
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/containers/registries.conf
EOF

As you can see content of registries.conf file data is url encoded. You can use any url encoding tool or online service to encode/decode your configuration files data. 

There are two important metadata in this object. First one is labels section where you specify to what Machine Config Pool this configuration should be added (master or worker). In our example this is machineconfiguration.openshift.io/role: worker. Second is name: 50-worker-container-registries which should start with higher number than Machine Config you want to override (if you want to override existing configuration file and not create new one).

Now you can create this Machine Config in your cluster:

$ oc create -f 50-worker-container-registries.yaml -n openshift-config

This should trigger automatically rolling upgrade of your worker nodes. You should see worker nodes being restarted one by one. You can also check if your Machine Config Pool is in updating status:

$ ./oc get machineconfigpools
NAME     CONFIG                                                                   UPDATED   UPDATING   DEGRADED
master   rendered-master-e192851e43f1ab347b3a565c9c71d2b8   True      False      False
worker   rendered-worker-5c35596867d37b22ca2daac46351cda5   True      True      False 

After your nodes are upgraded you can check if new configuration has been applied successfully:

$ oc get nodes

NAME                           STATUS                     ROLES          AGE   VERSION
ip-10-0-135-132.ec2.internal   Ready                      worker         15h   v1.13.4+cb455d664
ip-10-0-139-98.ec2.internal    Ready,SchedulingDisabled   worker         47h   v1.13.4+cb455d664
ip-10-0-140-77.ec2.internal    Ready                      infra,worker   46h   v1.13.4+cb455d664
ip-10-0-143-102.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664
ip-10-0-154-138.ec2.internal   Ready                      worker         47h   v1.13.4+cb455d664
ip-10-0-159-103.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664
ip-10-0-160-135.ec2.internal   Ready                      worker         47h   v1.13.4+cb455d664
ip-10-0-172-230.ec2.internal   Ready                      master         47h   v1.13.4+cb455d664

$ oc debug node/ip-10-0-135-132.ec2.internal
 

Starting pod/ip-10-0-135-132ec2internal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
 

sh-4.2# chroot /host
 

sh-4.4# cat /etc/containers/registries.conf

[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'quay.io']

[registries.insecure]
registries = []

[registries.block]
registries = []

That's it! You have learned how to apply custom configuration to your Openshift v4 cluster node.