Getting started with Openshift v4 - part 2 - installing on vSphere

After successfull installation of Openshift v4 on AWS I had a opportunity to install Openshift v4 on vSphere. This time installation had to be done following User Provisioned Infrastructure procedure. This means before the cluster could be bootstrapped I had to prepare required infrastucture myself following the documentation. 

First thing you need to prepare is networking: 2 load balancers for masters and worker nodes and DNS configuration for your cluster.

Secondly you need to download and run openshift-install but this time only to generate your machines ignition config files. Don't forget to create install-config.yaml file first!

Last thing you need to do is to create virtual machines in your vSphere cluster. You'll need to download RHCOS OVA image and create 1 bootstrap, 3 masters and 3 worker machines.

Now you are almost done!

At this point I decided to slighty disobey documentation and boot seperately masters and worker nodes. In order to do that first I have started only bootstrap node and 3 master nodes and had to wait until bootstrap process has finished. This will take a while and you can either ssh to bootstrap machine and follow journal log, or call openshift-install with wait-for bootstrap-complete parameters to monitor bootstrap progress. 

To confirm master bootstrap is done you can call oc get nodes to verify if all 3 masters are in ready status.

$ oc get nodes

NAME      STATUS    ROLES   AGE  VERSION
master-0  Ready     master  63m  v1.13.4+b626c2fe1
master-1  Ready     master  63m  v1.13.4+b626c2fe1
master-2  Ready     master  64m  v1.13.4+b626c2fe1

At this point you should delete bootstrap virtual machine and remove it from the load balancer.

Now you can simply start worker nodes (one by one, or in pararel) and wait for a while until they'll appear in the nodes list with status ready by calling again oc get nodes command.

Next thing you'll need to do is to approve pending csrs for your worker machines.

Last but not least don't forget to configure storage for you image registry operator.

That's it. Your Openshift v4 cluster is up and running on vSphere!

In part 3 I'll show you how to proceed with cluster configuration.

Getting started with Openshift v4 - part 1 - Installing on AWS

We've just announced release of new Red Hat Openshift 4, which will be available in a few weeks. However if you are interested in trying Openshift 4 eariler there is already beta version available for you. Please refer to Openshift 4 documentation for more details. 

Openshift 4 installation is currently possible only on AWS, vSphere and bare metal. Support for other platforms including Azure, GCP, OpenStack, Red Hat Virtualization will follow in subsequent minor releases within next few months.

I gave first try to install Openshift 4 on AWS with Installer Provisioned Infrastructure which means installer installed for me Openshift 4 cluster nodes as well as the platform itself.

I must admit this has been very pleasant experience. I had to only set up my AWS account, and register or transfer internet domain at Route53. The rest has been done automatically by the installer.

After about 30 mins I had my cluster of 3 masters and 3 worker nodes up and running.

What is equally important (a least for me) it was also very straightforward to uninstall the cluster using the same installer. In fact I've already installed and uninstalled the cluster a couple of times without any issues.

Nice!

In part 2 I'll share my experience with Openshift 4 installation on vSphere so stay tuned...

Becoming Red Hat Certified Architect in Enterprise Applications

Are you application architect or developer working with Red Hat JBoss, Openshift, J2EE, Microprofile or Camel technologies?  Would you like to become Red Hat Certified Architect in Enterprise Applications? You can achieve it by taking Enterprise Applications RHCA certification track (Go to "Cerification details" and select tab "For RHCEMDs & RHCJDs").

For example list of Openshift and JBoss exams you need to pass check my exams transcript.

Prune all evicted or failed Pods

Sometimes you might have a lot of pods in Evicted or other state which requires manual deletion by cluster administrator. This is how you can do that with single command:  

for n in $(oc get projects --no-headers=true | awk '{print $1}'); do echo $n; for pod in $(oc get pods -n $n --no-headers=true | grep "Evicted" | awk '{print $1}'); do oc delete pod ${pod} -n ${n}; done; done;

Similarly you can delete with single command all pods in CrashLoopBackOff state:

for n in $(oc get projects --no-headers=true | awk '{print $1}'); do echo $n; for pod in $(oc get pods -n $n --no-headers=true | grep "CrashLoopBackOff" | awk '{print $1}'); do oc delete pod ${pod} -n ${n}; done; done;

Automatic updates of Images

Openshift Container Platform has nice feature called Image Streams which aggregates all tags of container image in single object. This makes it easier to reference images from other configuration objects in Openshift i.e. Deployment Configs. Other nice feature of Image Steams is that you can schedule automatic updates of existing and new tags from Image Stream sourcing image registry. In order to make your Image Stream schedulable for updates you need to add --scheduled=true parameter to oc import-image command which creates new Image Stream from existing container image:

oc import-image ruby --from=registry.redhat.io/rhscl/ruby-25-rhel7 --confirm --all --scheduled=true

Image Streams update scheduler is enabled by default and is configured in master-config.yaml in imagePolicyConfig section:

imagePolicyConfig:
  MaxScheduledImageImportsPerMinute: 10
  ScheduledImageImportMinimumIntervalSeconds: 1800
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 3

You can adjust this configuration according to your needs but first remember to refer Openshift docs and restart master api and controller processes after you made any changes.

Garbage Collector

When your nodes are running for some time you might find a lot of terminated containers and container images unreferenced by any pods consuming your container storage. In Docker environment you would execute docker rm and docker rmi to get rid of this objects. In Openshift alternatively you could leverage built in Garbage Collector which can do this for you automatically in the background. 

Garbage Collector is enabled by default in Openshift however with no limits for number of terminated containers and very high limits for unreferenced images storage space. You can adjust this defaults by setting kubeletArguments section of the node-config.yaml: 

kubeletArguments:

  ...
  maximum-dead-containers-per-container:
  - '1'
  maximum-dead-containers:
  - '50'
  minimum-container-ttl-duration:
  - 10s
  image-gc-high-threshold:
  - '70'
  image-gc-low-threshold:
  - '60'

You could learn more about this useful functionality in Openshift docs.





 

Deleting projects stuck in Terminating state

Sometimes you'll experience OpenShift projects stuck in Terminating state. One of the reasons could be your project has orphan serivceinstance or servicebinding objects. You can list this objects by calling explicitly oc get command:

$ oc get serviceinstance -n your_project_name
$ oc get servicebinding -n your_project_name

If this commands will return any objects you can get rid of them and likely get the project deleted automatically by OpenShift with following commands:

$ for i in $(oc get projects | grep Terminating | awk '{print $1}'); do echo $i; oc get serviceinstance -n $i -o yaml | sed "/kubernetes-incubator/d" | oc apply -f - ; done

$ for i in $(oc get projects | grep Terminating | awk '{print $1}'); do echo $i; oc get servicebinding -n $i -o yaml | sed "/kubernetes-incubator/d" | oc apply -f - ; done