Verify Cosign bring-your-own PKI signature on OpenShift

Cosign brings its BYO-PKI signing feature to OpenShift in a Developer Preview version. This feature, available from 4.19, allows users to validate container images using an existing X.509 certificate while aligning with Cosign’s BYO-PKI signing workflow. The article walks through this process, from generating the certificate chain to configuring the ClusterImagePolicy CRD and verifying the signature using the image’s Docker reference.

To enable the Developer Preview features, follow these steps:

1. Install the following packages:
– `openssl`: The OpenSSL library for generating and verifying certificates.
– `cosign`: The cosign library for supporting BYO-PKI signatures.

“`bash
# Generate Root CA
openssl req -x509 -newkey rsa:4096 -keyout root-ca-key.pem -sha256 -noenc -days 9999 -subj “/C=ES/L=Valencia/O=IT/OU=Security/CN=Linuxera Root Certificate Authority” -out root-ca.pem
# Intermediate CA
openssl req -noenc -newkey rsa:4096 -keyout intermediate-ca-key.pem \
-addext “subjectKeyIdentifier = hash” \
-addext

Red Hat OpenShift 4.16 introduced ClusterImagePolicy and ImagePolicy as a tech preview feature for sigstore verification through the ClusterImagePolicy and ImagePolicy Custom Resource Definitions (CRDs). These initial implementations supported two policy types:

• Fulcio CA with Rekor: Leverages Sigstore’s certificate authority and transparency log for verification.
• Public key: Uses Cosign-generated private and public key pairs.

In this article, we will introduce the bring-your-own PKI (BYO-PKI) signature verification through the ClusterImagePolicy and ImagePolicy API. This Developer Preview feature (available from 4.19) enables you to validate container images using an existing X.509 certificate while aligning with Cosign’s BYO-PKI signing workflow. 

Cosign bring-your-own PKI signing

The following example generates the certificate chain using OpenSSL commands. We then use Cosign BYO-PKI to sign the image and attach the signature to the quay.io registry.

ClusterImagePolicy requires a subject alternative name (SAN) to authenticate the user’s identity, which can be either a hostname or an email address. In this case, both a hostname and an email address were specified when generating the certificate.

# Generate Root CA
openssl req -x509 -newkey rsa:4096 -keyout root-ca-key.pem -sha256 -noenc -days 9999 -subj "/C=ES/L=Valencia/O=IT/OU=Security/CN=Linuxera Root Certificate Authority" -out root-ca.pem
# Intermediate CA
openssl req -noenc -newkey rsa:4096 -keyout intermediate-ca-key.pem \
-addext "subjectKeyIdentifier = hash" \
-addext "keyUsage = keyCertSign" \
-addext "basicConstraints = critical,CA:TRUE,pathlen:2"  \
-subj "/C=ES/L=Valencia/O=IT/OU=Security/CN=Linuxera Intermediate Certificate Authority" \
-out intermediate-ca.csr
openssl x509 -req -days 9999 -sha256 -in intermediate-ca.csr -CA root-ca.pem -CAkey root-ca-key.pem -copy_extensions copy -out intermediate-ca.pem
# Leaf CA
openssl req -noenc -newkey rsa:4096 -keyout leaf-key.pem \
-addext "subjectKeyIdentifier = hash" \
-addext "keyUsage = digitalSignature" \
-addext "subjectAltName = email:qiwan@redhat.com,DNS:myhost.example.com" \
-subj "/C=ES/L=Valencia/O=IT/OU=Security/CN=Team A Cosign Certificate" -out leaf.csr
openssl x509 -req -in leaf.csr -CA intermediate-ca.pem -CAkey intermediate-ca-key.pem -copy_extensions copy -days 9999 -sha256 -out leaf.pem
# Bundle CA chain (Intermediate + Root)
cat intermediate-ca.pem root-ca.pem > ca-bundle.pem
# Sign the image using cosign
podman pull quay.io/libpod/busybox
podman tag quay.io/libpod/busybox quay.io/qiwanredhat/byo:latest
podman push --tls-verify=false --creds=<username>:<password> quay.io/qiwanredhat/byo:latest
IMAGE=quay.io/qiwanredhat/byo
PAYLOAD=payload.json
cosign generate $IMAGE >$PAYLOAD
openssl dgst -sha256 -sign leaf-key.pem -out $PAYLOAD.sig $PAYLOAD
cat $PAYLOAD.sig | base64 >$PAYLOAD.base64.sig
cosign attach signature $IMAGE \
	--registry-password=<password> \
	--registry-username=<username> \
	--payload $PAYLOAD \
	--signature $PAYLOAD.base64.sig \
	--cert leaf.pem \
	--cert-chain ca-bundle.pem

The next section will show how to configure ClusterImagePolicy to verify this signature.

Configure OpenShift for PKI verification

This section will guide you through verifying the quay.io/qiwanredhat/byo image. This involves enabling DevPreviewNoUpgrade features and configuring the ClusterImagePolicy CRD.

Enable Developer Preview features

First we have to enable the required Developer Preview features for your cluster by editing the FeatureGate CR named cluster. 

$ oc edit featuregate cluster
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  name: cluster
spec:
  featureSet: DevPreviewNoUpgrade

Define ClusterImagePolicy

This section creates the following ClusterImagePolicy CR for image verification. In the CR spec, it specifies the image to be verified and the details of the PKI certificate. It also specifies the matchPolicy to MatchRepository because the image was signed with the repository (the value of docker-reference from payload.json) rather than a specific tag or digest. If not specified,  the default matchPolicy is MatchRepoDigestOrExact, which requires the signature docker-reference to match the image specified in the pod Spec.

apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy
metadata:
  name: pki-quay-policy
spec:
  scopes:
  - quay.io/qiwanredhat/byo
  policy:
    rootOfTrust:
      policyType: PKI
      pki:
    	 caRootsData: <base64-encoded-root-ca>
    	 caIntermediatesData: <base64-encoded-intermediate-ca>
    	 pkiCertificateSubject:
      	   email: qiwan@redhat.com
      	   hostname: myhost.example.com
    signedIdentity:
  	# set matchPolicy(default is MatchRepoDigestOrExact) since the above signature was signed on the repository, not a specific tag or digest
      matchPolicy: MatchRepository

This ClusterImagePolicy object will be rolled out to /etc/containers/policy.json, and update /etc/containers/registries.d/sigstore-registries.yaml to add an entry that enables sigstore verification on the quay.io/qiwanredhat/byo scope.

Validate signature requirements

Create the following test pod to confirm that CRI-O will verify the signature. To see the debug level log, follow this documentation to configure ContainerRuntimeConfig.

Create a test pod as follows:

kind: Pod
apiVersion: v1
metadata:
 generateName: img-test-pod-
spec:
 serviceAccount: default
 containers:
   - name: step-hello
 	command:
   	- sleep
   	- infinity
 	image: quay.io/qiwanredhat/byo:latest

Check CRI-O logs for verification.

sh-5.1# journalctl -u crio | grep -A 100 "Pulling image: quay.io/qiwanredhat"
Apr 21 08:09:07 ip-10-0-27-44 crio[2371]: time="2025-04-21T08:09:07.381322395Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/qiwanredhat/byo:latest" file="signature/policy_eval.go:274"
Apr 21 08:09:07 ip-10-0-27-44 crio[2371]: time="2025-04-21T08:09:07.381485828Z" level=debug msg=" Using transport \"docker\" specific policy section \"quay.io/qiwanredhat/byo\"" file="signature/policy_eval.go:150"

Policy enforcement failure modes and diagnostics

For an image to be accepted by CRI-O during container creation, all the signature requirements must be satisfied. Pod events should show SignatureValidationFailed from the kubelet on verification failures. The CRI-O log provides more details.

The following is the result of an attempt to deploy an unsigned image quay.io/qiwanredhat/byo:latest.

$ oc get pods
NAME                 READY   STATUS             RESTARTS   AGE
img-test-pod-sdk47   0/1     ImagePullBackOff   0          13m

Events:
  Type 	Reason      	Age               	From           	Message
  ---- 	------      	----              	----           	-------
  Normal   Scheduled   	13m               	default-scheduler  Successfully assigned default/img-test-pod-sdk47 to ip-10-0-56-56.us-east-2.compute.internal
  Normal   AddedInterface  13m               	multus         	Add eth0 [10.131.2.23/23] from ovn-kubernetes
  Normal   Pulling     	10m (x5 over 13m) 	kubelet        	Pulling image "quay.io/qiwanredhat/busybox-byo:latest"
  Warning  Failed      	10m (x5 over 13m) 	kubelet        	Failed to pull image "quay.io/qiwanredhat/busybox-byo:latest": SignatureValidationFailed: Source image rejected: A signature was required, but no signature exists
  Warning  Failed      	10m (x5 over 13m) 	kubelet        	Error: SignatureValidationFailed
  Normal   BackOff     	3m16s (x42 over 13m)  kubelet        	Back-off pulling image "quay.io/qiwanredhat/busybox-byo:latest"
  Warning  Failed      	3m16s (x42 over 13m)  kubelet        	Error: ImagePullBackOff

journalctl -u crio | grep "byo"
Apr 23 06:12:38 ip-10-0-56-56 crio[2366]: time="2025-04-23T06:12:38.141197504Z" level=debug msg="Fetching sigstore attachment manifest failed, assuming it does not exist: reading manifest sha256-8677cb90773f20fecd043e6754e548a2ea03a232264c92a17a5c77f1c4eda43e.sig in quay.io/qiwanredhat/byo: manifest unknown" file="docker/docker_client.go:1129"

Final thoughts

This article demonstrated how to perform signature verification on images signed with the Cosign’s bring-your-own PKI feature in OpenShift using the ClusterImagePolicy CRD. We walked through the end-to-end process of signing an image with Cosign and BYO-PKI, followed by configuring OpenShift to verify that signature. 

As we progress toward general availability (GA) for this feature, organizations can leverage their existing PKI infrastructure to enhance the security and integrity of container images running on OpenShift.

The post Verify Cosign bring-your-own PKI signature on OpenShift appeared first on Red Hat Developer.