feature: Sources for scraping KCP related metrics

[vishnuchalla]

Feature Description

As the performance and scale managed services team, we are trying to do some performance analysis on KCP. To see how it performs on single vs multiple clusters in functionalities like syncing resources, managing them (especially latencies, cpu utilization and other useful metrics) etc.

But we couldn't find anything related to metrics and couldn't come across any resources for metrics to look at? The only things we see is the syncer pod cpu/memory utilization in our aws cluster which has prometheus capturing those metrics. Attaching the screenshot for your reference.

And also after increasing the verbosity of the syncer logs we were able to see the down-sync and up-sync events happening and some latencies being mentioned over there like in the below output.

[vchalla@vchalla ~]$ oc logs pod/kcp-syncer-cluster-a-2u2d8044-66bff97fb5-t8t2h | grep -i 'reverse-words' 
{"ts":1676217410470.9785,"caller":"status/status_controller.go:155","msg":"queueing GVR","syncTarget.workspace":"14tbo1g9yrz834mu","syncTarget.name":"cluster-a","syncTarget.key":"1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV","reconciler":"kcp-workload-syncer-status","key":"kcp-2g7c9urme8u7/reversewords","v":2,"gvr":"apps/v1, Resource=deployments"}
{"ts":1676217410471.0366,"caller":"status/status_controller.go:197","msg":"processing key","syncTarget.workspace":"14tbo1g9yrz834mu","syncTarget.name":"cluster-a","syncTarget.key":"1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV","reconciler":"kcp-workload-syncer-status","key":"kcp-2g7c9urme8u7/reversewords","gvr":"apps/v1, Resource=deployments","v":1}
{"ts":1676217410471.2617,"caller":"rest/request.go:1073","msg":"Request Body: {\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"kcp.io/cluster\":\"14tbo1g9yrz834mu\"},\"creationTimestamp\":\"2023-02-12T14:12:25Z\",\"finalizers\":[\"workload.kcp.io/syncer-1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\"],\"generation\":1,\"labels\":{\"app\":\"reversewords\",\"state.workload.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":\"Sync\"},\"managedFields\":[{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:app\":{}}},\"f:spec\":{\".\":{},\"f:replicas\":{},\"f:selector\":{\".\":{},\"f:matchLabels\":{\".\":{},\"f:app\":{}}},\"f:strategy\":{},\"f:template\":{\".\":{},\"f:metadata\":{\".\":{},\"f:creationTimestamp\":{},\"f:labels\":{\".\":{},\"f:app\":{}}},\"f:spec\":{\".\":{},\"f:containers\":{\".\":{},\"k:{\\\"name\\\":\\\"reversewords\\\"}\":{\".\":{},\"f:image\":{},\"f:name\":{},\"f:resources\":{}}}}}}},\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"time\":\"2023-02-12T14:12:25Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"workload.kcp.io/syncer-1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\\\"\":{}},\"f:labels\":{\"f:state.workload.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":{}}}},\"manager\":\"kcp\",\"operation\":\"Update\",\"time\":\"2023-02-12T15:56:00Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:diff.syncer.internal.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":{}}},\"f:status\":{\".\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:replicas\":{},\"f:unavailableReplicas\":{},\"f:updatedReplicas\":{}}},\"manager\":\"kcp\",\"operation\":\"Update\",\"subresource\":\"status\",\"time\":\"2023-02-12T15:56:48Z\"}],\"name\":\"reversewords\",\"namespace\":\"reverse-words\",\"resourceVersion\":\"439491\",\"uid\":\"d1a85ee8-c8e0-4a05-8fd3-946c5a1b780c\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"reversewords\"}},\"strategy\":{},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"reversewords\"}},\"spec\":{\"containers\":[{\"image\":\"quay.io/mavazque/reversewords:latest\",\"name\":\"reversewords\",\"resources\":{}}]}}},\"status\":{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2023-02-12T15:56:50Z\",\"lastUpdateTime\":\"2023-02-12T15:56:50Z\",\"message\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-02-12T15:56:48Z\",\"lastUpdateTime\":\"2023-02-12T15:56:50Z\",\"message\":\"ReplicaSet \\\"reversewords-6cfdb59556\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}}\n","v":8}
{"ts":1676217410471.3452,"caller":"transport/round_trippers.go:466","msg":"curl -v -XPUT  -H \"Content-Type: application/json\" -H \"Accept: application/json\" -H \"User-Agent: syncer/v0.0.0 (linux/amd64) kubernetes/$Format/kcp#syncing/v0.0.0-master+$Format:%H$\" -H \"Authorization: Bearer <masked>\" 'https://10.0.35.28:6443/services/syncer/14tbo1g9yrz834mu/cluster-a/06f5af0a-c7e2-4f60-ba1d-904d06804421/clusters/14tbo1g9yrz834mu/apis/apps/v1/namespaces/reverse-words/deployments/reversewords/status'\n","v":0}

{"ts":1676217410483.225,"caller":"transport/round_trippers.go:553","msg":"PUT https://10.0.35.28:6443/services/syncer/14tbo1g9yrz834mu/cluster-a/06f5af0a-c7e2-4f60-ba1d-904d06804421/clusters/14tbo1g9yrz834mu/apis/apps/v1/namespaces/reverse-words/deployments/reversewords/status 200 OK in 11 milliseconds\n","v":0}

{"ts":1676217410483.3745,"caller":"rest/request.go:1073","msg":"Response Body: {\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"kcp.io/cluster\":\"14tbo1g9yrz834mu\"},\"creationTimestamp\":\"2023-02-12T14:12:25Z\",\"finalizers\":[\"workload.kcp.io/syncer-1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\"],\"generation\":1,\"labels\":{\"app\":\"reversewords\",\"state.workload.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":\"Sync\"},\"managedFields\":[{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:app\":{}}},\"f:spec\":{\".\":{},\"f:replicas\":{},\"f:selector\":{\".\":{},\"f:matchLabels\":{\".\":{},\"f:app\":{}}},\"f:strategy\":{},\"f:template\":{\".\":{},\"f:metadata\":{\".\":{},\"f:creationTimestamp\":{},\"f:labels\":{\".\":{},\"f:app\":{}}},\"f:spec\":{\".\":{},\"f:containers\":{\".\":{},\"k:{\\\"name\\\":\\\"reversewords\\\"}\":{\".\":{},\"f:image\":{},\"f:name\":{},\"f:resources\":{}}}}}}},\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"time\":\"2023-02-12T14:12:25Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:finalizers\":{\".\":{},\"v:\\\"workload.kcp.io/syncer-1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\\\"\":{}},\"f:labels\":{\"f:state.workload.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":{}}}},\"manager\":\"kcp\",\"operation\":\"Update\",\"time\":\"2023-02-12T15:56:00Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:diff.syncer.internal.kcp.io/1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV\":{}}},\"f:status\":{\".\":{},\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"manager\":\"kcp\",\"operation\":\"Update\",\"subresource\":\"status\",\"time\":\"2023-02-12T15:56:50Z\"}],\"name\":\"reversewords\",\"namespace\":\"reverse-words\",\"resourceVersion\":\"439494\",\"uid\":\"d1a85ee8-c8e0-4a05-8fd3-946c5a1b780c\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"reversewords\"}},\"strategy\":{},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"reversewords\"}},\"spec\":{\"containers\":[{\"image\":\"quay.io/mavazque/reversewords:latest\",\"name\":\"reversewords\",\"resources\":{}}]}}},\"status\":{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2023-02-12T15:56:50Z\",\"lastUpdateTime\":\"2023-02-12T15:56:50Z\",\"message\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-02-12T15:56:48Z\",\"lastUpdateTime\":\"2023-02-12T15:56:50Z\",\"message\":\"ReplicaSet \\\"reversewords-6cfdb59556\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}}\n","v":8}
{"ts":1676217410483.5417,"caller":"status/status_process.go:246","msg":"Updated status of upstream resource","syncTarget.workspace":"14tbo1g9yrz834mu","syncTarget.name":"cluster-a","syncTarget.key":"1KTm6s7pmYlj7Svt8flQWZitwuiiIG2YP5XJCV","reconciler":"kcp-workload-syncer-status","key":"kcp-2g7c9urme8u7/reversewords","gvr":"apps/v1, Resource=deployments","downstream.namespace":"kcp-2g7c9urme8u7","downstream.name":"reversewords","workspace":"14tbo1g9yrz834mu","namespace":"reverse-words","name":"reversewords","v":0}

But It is very tedius task to scrape them as the logs are only renteded for a cretain period of time after the event has been created. And also they do not have any sort of requestId in common so that we can use that for scraping.

Apart from that based on some of the previous discussions we have tried to look at the below endpoint for scraping metrics. But it showed up with the below error. Is it the right endpoint to look at? or else please point us with right ones.

ubuntu@ip-10-0-35-28:~/kcp$ curl -k https://localhost:6443/metrics --capath .kcp/
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\": access denied",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Proposed Solution

Please provide us with any sources like endpoints to connect and scrape the metrics from.
Or else any other pointers/suggestions on gathering metrics are much appreciated.

Alternative Solutions

No response

Want to contribute?

• I would like to work on this issue.

Additional Context

No response