Multi tenant metrics collection from OpenShift built in Prometheus

In one of my previous posts I've described 3 ways to collect metrics stored in OpenShift built-in Prometheus metrics database. Now I'd like to show you how you can limit access to metrics per tenant projects (namespaces). 

Built-in Thanos Queries contains dedicated tenancy port which requires namespace parameter to access metrics of objects belonging to the project. If you want to expose this port outside of the cluster you'll need to create custom route (ingress):

kind: Route

apiVersion: route.openshift.io/v1

metadata:

  name: multitenant-thanos-querier

  namespace: openshift-monitoring

spec:

  host: ROUTE_HOSTNAME

  path: /api

  to:

    kind: Service

    name: thanos-querier

    weight: 100

  port:

    targetPort: tenancy

  tls:

    termination: reencrypt

  wildcardPolicy: None

 

Next you can execute follwing commands to query metrics from selected namespace: 

 

PROJECT=sample-app-prod

SA=querier

oc project $PROJECT

oc create sa $SA

TOKEN=$(oc sa get-token $SA)

URL=$(oc get route -n openshift-monitoring | grep multitenant | awk '{print $2}')

 

You don't need to assign cluster-monitoring-view role to the service account but only view role in the project where you want to query the metrics:  

 

oc adm policy add-cluster-role-to-user view -z $SA

 

In the thanos querier query you must enter namespace parameter to specify which namespace metrics you want to get:  

 

curl -v -k -H "Authorization: Bearer $TOKEN" "https://$URL/api/v1/query?namespace=$PROJECT&query=kube_pod_status_ready"

 

In the results you will only see values of metrics which are related to your selected project:  

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"kube_pod_status_ready","condition":"false","container":"kube-rbac-proxy-main","endpoint":"https-main","job":"kube-state-metrics","namespace":"sample-app-prod","pod":"hello-quarkus-5859859f9f-vbhjh","prometheus":"openshift-monitoring/k8s","service":"kube-state-metrics"},"value":[1651068513.646,"0"]}]}} 

That's it. Now you only have access to metrics from the projects where you have the view role.