This post is an follow up to my previous post on the same topic. This time I'd like to show you how you can configure OpenShift 4 to verify container images signatures signed using custom gpg keys.
When you sign your image with your gpg key you need to remember to store signature on the registry server (web server) which will be accessible from OpenShift cluster worker nodes.
When you sign your image using skopeo i.e:
$ skopeo copy --sign-by jstakun@example.com registry.redhat.io/rhscl/httpd-24-rhel7:2.4 quay.io/jstakun/httpd-signed:2.4
image signature will be saved by default in subdirectory of /var/lib/atomic/sigstore. You need to make sure this signature is copied to registry server (web server) with the same directory structure.
Once this is done you can follow this steps to verify image signature:
1. Create base64 encoded /etc/containers/policy.json file:
$ cat << EOF | base64
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports":
{
"docker-daemon":
{
"": [{"type":"insecureAcceptAnything"}]
},
"docker":
{
"registry.redhat.io": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}],
"registry.access.redhat.com": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
}],
"quay.io/jstakun/httpd-signed": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/rpm-gpg/jstakun-pub"
}]
}
}
}
EOF
2. Create base64 encoded /etc/containers/registries.d/redhat.yaml file
$ cat << EOF | base64
docker:
registry.access.redhat.com:
sigstore: https://access.redhat.com/webassets/docker/content/sigstore
registry.redhat.io:
sigstore: https://access.redhat.com/webassets/docker/content/sigstore
EOF
3. Create base64 encoded /etc/containers/registries.d/httpd-signed.jstakun.quay.io.yaml file. This is where you need to set the url of your registry server (web server) hosting your images signatures.
$ cat << EOF | base64
docker:
quay.io/jstakun/httpd-signed:
sigstore: http://my-web-server.example.com:8000
EOF
4. Create URL encoded file containing your gpg public key exported using following command:
$ gpg --output pubring.asc --armor --export jstakun@example.com
5. Create machine config object in your OpenShift 4 cluster using output of the above commands as files contents source values like in example below.
$ echo "---This should trigger automatically rolling upgrade of your worker nodes. You should see worker nodes being restarted one by one in rolling fashion. You can also check if your Machine Config Pool is in updating status. Please refer to my previous post for more details on how to do that.
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 20-image-signature-verify
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,ewogICJkZWZhdWx0IjogWwogICAgewogICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgfQogIF0sCiAgInRyYW5zcG9ydHMiOgogICAgewogICAgICAiZG9ja2VyLWRhZW1vbiI6CiAgICAgICAgewogICAgICAgICAgIiI6IFt7InR5cGUiOiJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIn1dCiAgICAgICAgfSwKICAgICAgImRvY2tlciI6CiAgICAgICAgewogICAgICAgICAgInJlZ2lzdHJ5LnJlZGhhdC5pbyI6IFsKICAgICAgICAgICAgewogICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvZXRjL3BraS9ycG0tZ3BnL1JQTS1HUEctS0VZLXJlZGhhdC1yZWxlYXNlIgogICAgICAgICAgICB9XSwKICAgICAgICAgICJyZWdpc3RyeS5hY2Nlc3MucmVkaGF0LmNvbSI6IFsKICAgICAgICAgICAgewogICAgICAgICAgICAgICJ0eXBlIjogInNpZ25lZEJ5IiwKICAgICAgICAgICAgICAia2V5VHlwZSI6ICJHUEdLZXlzIiwKICAgICAgICAgICAgICAia2V5UGF0aCI6ICIvZXRjL3BraS9ycG0tZ3BnL1JQTS1HUEctS0VZLXJlZGhhdC1yZWxlYXNlIgogICAgICAgICAgICB9XSwKICAgICAgICAgICJxdWF5LmlvL2pzdGFrdW4vaHR0cGQtc2lnbmVkIjogWwogICAgICAgICAgICB7CiAgICAgICAgICAgICAgInR5cGUiOiAic2lnbmVkQnkiLAogICAgICAgICAgICAgICJrZXlUeXBlIjogIkdQR0tleXMiLAogICAgICAgICAgICAgICJrZXlQYXRoIjogIi9ldGMvcGtpL3JwbS1ncGcvanN0YWt1bi1wdWIiCiAgICAgICAgICAgIH1dCiAgICAgICAgfQogICAgfQp9Cg==
filesystem: root
mode: 420
path: /etc/containers/policy.json
- contents:
source: data:text/plain;charset=utf-8;base64,ZG9ja2VyOgogIHJlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tOgogICAgc2lnc3RvcmU6IGh0dHBzOi8vYWNjZXNzLnJlZGhhdC5jb20vd2ViYXNzZXRzL2RvY2tlci9jb250ZW50L3NpZ3N0b3JlCiAgcmVnaXN0cnkucmVkaGF0LmlvOgogICAgc2lnc3RvcmU6IGh0dHBzOi8vYWNjZXNzLnJlZGhhdC5jb20vd2ViYXNzZXRzL2RvY2tlci9jb250ZW50L3NpZ3N0b3JlCg==
filesystem: root
mode: 420
path: /etc/containers/registries.d/redhat.yaml
- contents:
source: data:text/plain;charset=utf-8;base64,ZG9ja2VyOgogIHF1YXkuaW8vanN0YWt1bi9odHRwZC1zaWduZWQ6CiAgICBzaWdzdG9yZTogaHR0cDovL2VjMi0xMDctMjEtMjQ3LTQ3LmNvbXB1dGUtMS5hbWF6b25hd3MuY29tOjgwMDAK
filesystem: root
mode: 420
path: /etc/containers/registries.d/httpd-signed.jstakun.quay.io.yaml
- contents:
source: data:,-----BEGIN%20PGP%20PUBLIC%20KEY%20BLOCK-----%0AVersion%3A%20GnuPG%20v2.0.22%20%28GNU%2FLinux%29%0A%0AmQENBF86aMQBCACfv0qeej1rLW9wQKSmSjDcALqZW6wz23at6l%2FD2lLlMOuZSns2%0A4YwZL0mV61j5gfr5D7vk40KMhmcu0jfHeth9TeEMCptFkAXMoY%2Boec8Dz%2Bp0YBuj%0A53ff36VbUjpGa%2BocX32yfTtG9Ez8rc%2BjQxbe1ecZEgVhi41Z7xZmXxR4MkX1YThp%0A%2FofSnULtVhvk1jV43s1ZOwcloe1iNIM8mq185tP67ZBeaLvHIFKiXFOP0w%2F19Jjb%0AhkzUMlaw2ggXVylDA2GVVKw0QJ3iMdt4i%2Fx8DlRFqRsa7Vrrryg08n0fTB4ZyvxB%0AGGJarUliJaavFDORbkA58XougJsT8d5RIaDxABEBAAG0JEphcm9zbGF3IFN0YWt1%0AbiA8anN0YWt1bkByZWRoYXQuY29tPokBOQQTAQIAIwUCXzpoxAIbAwcLCQgHAwIB%0ABhUIAgkKCwQWAgMBAh4BAheAAAoJEP3YZiiMOKUTUwUH%2F3%2BiRlZk1idLk1tntGSg%0AaO4CvhSz8dlC%2Bt062ccPMYVXOgLv%2FCfI8gwpYmLKMieZLeJVlWN7gTuwsFSlAdqn%0AWKBm1JA2MsJ08b0jtYOG6xMKeScLgim0zX%2BdoK8ljrB4%2FvijsW7Vk5ykcyxDogK0%0AOyPAGD%2FNQUFfUsPFFdaMOGaxhpswh1VKZQ0NL67hAi2tASsufr3FdgF3%2B0ELSKQB%0A9gX4thaBN5wOYNUZlLXbGRipxi%2BrcksgaQj0DMUaqRMWpfRXrbTnimCrr0cNvr%2Bf%0AdFljfbkjoL4VXCUics%2BdpKpj3iEDOJBTBkAy84nQExNfh1nJJlrVsnfHVx%2FM9czF%0A3VC5AQ0EXzpoxAEIAK40kfShcTxrR7QljNBrAywaSflgrKOT9DXv22%2FXvo0wHSPc%0AfVkzWkaCwH7%2F4P4WOMpZfhr1QKw8GA3jvn7zJ1m4zVwe9UZsmPPQR8pCRtuelpb%2B%0A1O6LhOjNbqc58rgFsV95ZcSQoJV%2BSK3HLKjUyzzHgby%2BOPmOIuj5kNHg9juAcAwH%0A%2FAKrIhPV5Kvpxo334ZgmZAgAdEuPKtRcpsW62YU0i9nlaR82eWMj6mxk0KEpVxww%0AGleke2mFvroW1RegADJta78W6wvxZpQgi9D%2B9lZgr5jlm0Q%2F05egYcAHve4hB4vw%0AJNSuNMUtaxo7bpf4sNavSHLanWbeIkAhtgndrH0AEQEAAYkBHwQYAQIACQUCXzpo%0AxAIbDAAKCRD92GYojDilE%2BD2B%2FwJUkBfBlzlVgMZ1ahXnQzRzU%2B3h8pSlQcVeaD5%0AgNfdOmVHd0KYZUhHsoY4uJqa590Spl3JJL72%2BG5U2qFo4gO49TZS18dPExPIFFJ8%0AsHukXbjuHXfEVBOUuU9OkBna2d1A61RswigfNXFao39Hvzudg7zQLeN%2B59mZsnKO%0AAFuFxPdY9F9xMf2%2F5%2FLdK5M%2BwBfl%2BjS0ca0I5IV32vKJjEJpbnlKfyblULDuSjsC%0Ag4snOoO0JiqM4KnmFm0la7VjXQ4hU7NdCCT4pimDLyG4Q%2FbYPcYiMVFTZX%2F6uxeI%0AlHHwH%2FPenLgSxyeWSw6bdwy1zyav%2BxUPA22T49a7eLRAEGuG%0A%3D07nQ%0A-----END%20PGP%20PUBLIC%20KEY%20BLOCK-----
filesystem: root
mode: 420
path: /etc/pki/rpm-gpg/jstakun-pub
" | oc create -f - -n openshift-config
In order to test whether images signatures are verified I recommend to run podman on one of your OpenShift 4 cluster nodes:
$ podman --log-level debug run --rm --name test quay.io/jstakun/httpd-signed:2.4
In the output of this command you should see that before image is pulled image signature is downloaded and verified.
