customresource GitRepo cannot be scraped

[ulikl]

What happened:

I try to get metrics for the following custom resources:

$ kubectl api-resources 
...
gitrepos                                                        fleet.cattle.io/v1alpha1                     true         GitRepo

During the execution the kind "GitRepo" is replaced for the kube api call with "gitrepoes", for which the api call fails as this kind does not exists.
There is an "e" too much in gitrepoes.

Here the logs

I0420 14:48:28.600525       1 reflector.go:257] Listing and watching fleet.cattle.io/v1alpha1, Kind=GitRepo from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169
I0420 14:48:28.600792       1 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json" -H "User-Agent: kube-state-metrics/v2.8.0 (linux/amd64) kubernetes/" -H "Authorization: Bearer <masked>" 'https://10.43.0.1:443/apis/fleet.cattle.io/v1alpha1/gitrepoes?limit=500&resourceVersion=0'
I0420 14:48:28.601511       1 round_trippers.go:553] GET https://10.43.0.1:443/apis/fleet.cattle.io/v1alpha1/gitrepoes?limit=500&resourceVersion=0 404 Not Found in 0 milliseconds

W0420 14:48:44.174514       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list fleet.cattle.io/v1alpha1, Kind=GitRepo: the server could not find the requested resource
E0420 14:48:44.174541       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch fleet.cattle.io/v1alpha1, Kind=GitRepo: failed to list fleet.cattle.io/v1alpha1, Kind=GitRepo: the server could not find the requested resource

and the scrape is empty.

with "Kind: gitrepos" I get

# initial read ok
I0420 14:52:56.193289       1 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json" -H "User-Agent: kube-state-metrics/v2.8.0 (linux/amd64) kubernetes/" -H "Authorization: Bearer <masked>" 'https://10.43.0.1:443/apis/fleet.cattle.io/v1alpha1/gitrepos?limit=500&resourceVersion=0'
I0420 14:52:56.194855       1 round_trippers.go:553] GET https://10.43.0.1:443/apis/fleet.cattle.io/v1alpha1/gitrepos?limit=500&resourceVersion=0 200 OK in 1 milliseconds

# but watch still not working:

E0420 13:56:12.707972       1 reflector.go:513] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: expected gvk fleet.cattle.io/v1alpha1, Kind=gitrepos, but watch event object had gvk fleet.cattle.io/v1alpha1, Kind=GitRepo

metrics scrape:

# HELP custom_fleet_gitrepo_desired_ready_clusters fleet gitrepository overall status summary
# TYPE custom_fleet_gitrepo_desired_ready_clusters gauge
custom_fleet_gitrepo_desired_ready_clusters{branch="main",customresource_group="fleet.cattle.io",customresource_kind="gitrepos",customresource_version="v1alpha1",name="...",namespace="fleet-default",repo="...git"} 1

But this will never be updated as the watch runs into an error, which skips the processing of the result according to the source code.

What you expected to happen:

kind "GitRepo" is replace for the kube api call with "gitrepos"

kubectl work correctly for GitRepo:

$ kubectl -v=10  get GitRepo -A

...
I0420 14:51:47.606674 2575333 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.21.0 (linux/amd64) kubernetes/cb303e6" -H "Authorization: Bearer <masked>" 'https://.../apis/fleet.cattle.io/v1alpha1/gitrepos?limit=500'
I0420 14:51:47.667377 2575333 round_trippers.go:454] GET https://.../apis/fleet.cattle.io/v1alpha1/gitrepos?limit=500 200 OK in 60 milliseconds

How to reproduce it (as minimally and precisely as possible):

Install fleet helm chart in Kubernetes from https://github.com/rancher/fleet/tree/master/charts/fleet-crd
The kube-state-metrics pod uses a serviceaccount with cluster-admin role.
Here is the custom-resource-state-config setting to use.

    spec:
      resources:
        - groupVersionKind:
            group: fleet.cattle.io
            version: "v1alpha1"
            kind: GitRepo
          metricNamePrefix: custom_fleet_gitrepo
          metrics:
            - name: desired_ready_clusters
              help: "fleet gitrepository overall status summary"
              each:
                type: Gauge
                gauge:
                  path: [status, "desiredReadyClusters"]
              labelsFromPath:
                # and specific fields are copied. these fields will always override values from *s
                name: [metadata, name]
                namespace: [metadata, namespace]
                repo: [spec, repo]
                branch: [spec, branch]

Anything else we need to know?:

Environment:

• kube-state-metrics version: 2.8.2
• Kubernetes version (use kubectl version): v1.24.8+rke2r1
• Cloud provider or hardware configuration: Rancher RKE2 cluster
• Other info:

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mrueg]

This may not be related to kube-state-metrics but might be an issue in your CRD.
Which version of the fleet-crd helm chart do you have deployed?
Can you check how this key looks like in your cluster's CRD? https://github.com/rancher/fleet/blob/1190e49c07e1e75528f19141f5b0c9924ec6bc31/charts/fleet-crd/templates/crds.yaml#L2653

[ulikl]

I am using rancher fleet based on version 0.4.0 and 0.5.3.
The names there look good to me.
e.g from 0.4.0

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: fleet-crd
    meta.helm.sh/release-namespace: cattle-fleet-system
  creationTimestamp: "2023-02-06T15:54:33Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gitrepos.fleet.cattle.io
  resourceVersion: "1726"
  uid: 8092caae-6d80-4cbb-864a-ffbede3db066
spec:
  conversion:
    strategy: None
  group: fleet.cattle.io
  names:
    categories:
    - fleet
    kind: GitRepo
    listKind: GitRepoList
    plural: gitrepos
    singular: gitrepo
  scope: Namespaced
 ... 

[mrueg]

It looks like client-go makes guesses at the plural kubernetes/client-go#1082

[ulikl]

ok. Thanks.

[mrueg]

Can you try the following CRM: (it's the same with resourcePlural: gitrepos added)

    spec:
      resources:
        - groupVersionKind:
            group: fleet.cattle.io
            version: "v1alpha1"
            kind: GitRepo
          metricNamePrefix: custom_fleet_gitrepo
          resourcePlural: gitrepos
          metrics:
            - name: desired_ready_clusters
              help: "fleet gitrepository overall status summary"
              each:
                type: Gauge
                gauge:
                  path: [status, "desiredReadyClusters"]
              labelsFromPath:
                # and specific fields are copied. these fields will always override values from *s
                name: [metadata, name]
                namespace: [metadata, namespace]
                repo: [spec, repo]
                branch: [spec, branch]

[ulikl]

Thanks, this work.