Operator is unable to create secret for vmagent when compressed config "vmagent.yaml.gz" is larger than 1MB

[prasadrajesh]

I am using latest build of the operator (v0.46.4) and vmagent (v1.102.0). I have lots of various endpoints to scrape. So Operator is trying to create secret for config "vmagent.yaml.gz" but it is larger than 1MB and managed AKS Kubernetes won't allow to create secret larger than 1MB (Ref-https://kubernetes.io/docs/concepts/configuration/secret/#working-with-secrets)

Error
Warning ReconcilationError <unknown> victoria-metrics-operator Secret "vmagent-vmagent-0" is invalid: data: Too long: must have at most 1048576 bytes

NAME        SHARDS COUNT   REPLICA COUNT   STATUS
vmagent-0                  1               failed
vmagent-1                  1               failed
vmagent-2                  1               failed

I also tried by enabling shards but no luck.

It is similar issue like prometheus-operator/prometheus-operator#4702

I think fix will be like prometheus-operator/prometheus-operator#4427 where operator creating shardSecret.

[f41gh7]

This problem is related to any kubernetes version. It should be possible to split generated config into multiple secrets and combine it with config-reloader container.

I think it should be a problem, since we're going to remove prometheus-operator config-reloader and use own implementation instead.

[prasadrajesh]

@f41gh7 how can I split generated config into multiple secrets? I believe it should be done by operator no?

[f41gh7]

It's not possible for now. It should be addressed at one of the next releases.

Current workaround - split scrape objects (vmservicescrape,vmpodscrape) between multiple vmagents with object selectors. An easiest way to me, split it by namespace. https://docs.victoriametrics.com/operator/resources/vmagent/#scraping

[prasadrajesh]

@f41gh7 , Thanks for quick reply. 90-95% my objects are vmservicescrape.

One more observation I found that generated config "vmagent.yaml.gz" has old podscrape objects which are not exist now (Including existing). Prometheus has only available object in config. So, generated compressed config from Prometheus-Operator is taking very less size compare to generated vm-operator.

[f41gh7]

One more observation I found that generated config "vmagent.yaml.gz" has old podscrape objects which are not exist now (Including existing). Prometheus has only available object in config. So, generated compressed config from Prometheus-Operator is taking very less size compare to generated vm-operator.

Hm, looks like kinda of misconfiguration for operator. By default, it doesn't deleted objects created from prometheus APIs. See this issue for details https://docs.victoriametrics.com/operator/migration/#deletion-synchronization

Looks like current solution is:

1. manually delete all objects converted from prometheus APIs ( operator will re-create it on start-up with proper links).
2. change operator configuration to add references for prometheus objects.