Verifying container images signatures in OpenShift 4

In this article, I’ll demonstrate how to configure a container engine to validate signatures of container images from the Red Hat registries for increased security of your containerized applications. 

Images are signed using gpg keys hence you need to make sure public keys are stored on OpenShift 4 worker nodes. Fortunetally Red Hat images are signed with the same key as RHEL rpm packages, so the key is already present on worker nodes at /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release. Furthermore images signatures are exposed on publically available registry server: https://access.redhat.com/webassets/docker/content/sigstore

Configuring Linux container tools to only run container images that pass signature checking is a two-step process:

1. Create a YAML file under /etc/containers/registries.d that specifies the location of detached signatures for a given registry server.
   
2. Add an entry to /etc/containers/policy.json that specifies the public GPG key that validates signatures of a given registry server.

In order to create or modify this files in OpenShift 4 worker nodes you'll need to create machine config containing this files in base64 encoded format:

$ cat << EOF | base64
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports":
    {
      "docker-daemon":
        {
          "": [{"type":"insecureAcceptAnything"}]
        },
      "docker":
        {
          "registry.redhat.io": [
            {
              "type": "signedBy",
              "keyType": "GPGKeys",
              "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
            }]
        }
    }
}
EOF

$ cat << EOF | base64
docker:
  registry.redhat.io:
    sigstore: https://access.redhat.com/webassets/docker/content/sigstore
EOF

Using the base64 content of the above commands you can create machine config:

echo "---
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 10-redhat-image-signature-verify
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,ewogICJkZWZhdWx0IjogWwogICAgewogICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgfQogIF0sCiAgInRyYW5zcG9ydHMiOgogICAgewogICAgICAiZG9ja2VyLWRhZW1vbiI6CiAgICAgICAgewogICAgICAgICAgIiI6IFt7InR5cGUiOiJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIn1dCiAgICAgICAgfSwKICAgICAgImRvY2tlciI6CiAgICAgICAgewogICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pbyI6IFsKICAgICAgICAgICAgewogICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvZXRjL3BraS9ycG0tZ3BnL1JQTS1HUEctS0VZLXJlZGhhdC1yZWxlYXNlIgogICAgICAgICAgICB9XQogICAgICAgIH0KICAgIH0KfQo=
        filesystem: root
        mode: 420
        path: /etc/containers/policy.json
      - contents:
          source: data:text/plain;charset=utf-8;base64,ZG9ja2VyOgogIHJlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tOgogICAgc2lnc3RvcmU6IGh0dHBzOi8vYWNjZXNzLnJlZGhhdC5jb20vd2ViYXNzZXRzL2RvY2tlci9jb250ZW50L3NpZ3N0b3JlCg==
        filesystem: root
        mode: 420
        path: /etc/containers/registries.d/redhat.yaml
" | oc create -f - -n openshift-config

This should trigger automatically rolling upgrade of your worker nodes. You should see worker nodes being restarted one by one. You can also check if your Machine Config Pool is in updating status. Please refer to my previous post for more details on how to do that.

 

If you want to verifiy signatures of images signed with custom gpg keys the procedure will be a bit more complicated. First you'll need to sign the image using custom gpg key i.e. using skopeo copy --sign-by command, then you must store signature on custom registry server (web server), and finally define machine config similar to the above but with custom gpg public key file included (output of gpg --output pubring.asc --armor --export username@email). 

 

This is a great topic for yet another post!