Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MySQL Scaled job now working with TriggerAuthentication #5416

Closed
ferrandinand opened this issue Jan 19, 2024 · 5 comments
Closed

MySQL Scaled job now working with TriggerAuthentication #5416

ferrandinand opened this issue Jan 19, 2024 · 5 comments
Labels
bug Something isn't working

Comments

@ferrandinand
Copy link

Report

MySQL Scaled job is not able to get variables from a TriggerAuthentication

Expected Behavior

I expect that password and other configs are configured from a TriggerAuthentication CR grabbing the config from a secret

Actual Behavior

Keda complains about variables required not present if configured in a TriggerAuthentication

Steps to Reproduce the Problem

  1. Run the following code Keda waits for a required missing host parameter
---
apiVersion: v1
data:
  mysql_conn_str: WVdSdGFXNDZjMlJtWkhOb...tZWFJoWW1GelpTMHg=
kind: Secret
metadata:
  name: mysql-secrets
  namespace: module-cwh-test3
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: mysql-auth
  namespace: module-cwh-test3
spec:
  secretTargetRef:
  - key: mysql_conn_str
    name: mysql-secrets
    parameter: connectionString
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: myscaled-job 
  namespace: module-cwh-test3    
spec:
  jobTargetRef:
    parallelism: 1                            
    completions: 1                            
    activeDeadlineSeconds: 600              
    backoffLimit: 6 
    template:
      spec:
        restartPolicy: Never
        containers:
        - image: public.ecr.aws/unicc/ubuntu-tooling:stable
          name: tooling
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: mysql-secrets
  triggers:
    - type: mysql
      authenticationRef:
        name: mysql-auth
      metadata: 
        queryValue: "1"
        query: "SELECT COUNT(*) FROM notifications WHERE state='QUEUED'"

  Warning  KEDAScalerFailed      1s (x11 over 6s)  scale-handler  error parsing MySQL metadata: missing required field in scaler config: no host given
  Warning  ScaledJobCheckFailed  1s (x11 over 6s)  keda-operator  Failed to ensure ScaledJob is correctly created
  1. Introducing the host in the secret and in the triggerauth does not make any difference
---
apiVersion: v1
data:
  mysql_conn_str: WVdSdGFXNDZjMlJtWkhOb...tZWFJoWW1GelpTMHg=
  host: ZGF0Y...cy5hbWF6b25hd3MuY29t
kind: Secret
metadata:
  name: mysql-secrets
  namespace: module-cwh-test3
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: mysql-auth
  namespace: module-cwh-test3
spec:
  secretTargetRef:
  - key: mysql_conn_str
    name: mysql-secrets
    parameter: connectionString
  - key: host
    name: mysql-secrets
    parameter: host
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: myscaled-job 
  namespace: module-cwh-test3    
spec:
  jobTargetRef:
    parallelism: 1                            
    completions: 1                            
    activeDeadlineSeconds: 600              
    backoffLimit: 6 
    template:
      metadata:
        labels:
          run: ubuntu-tooling-lap
      spec:
        restartPolicy: Never
        containers:
        - image: public.ecr.aws/unicc/ubuntu-tooling:stable
          name: tooling
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: mysql-secrets
  triggers:
    - type: mysql
      authenticationRef:
        name: mysql-auth
      metadata: 
        queryValue: "1"
        query: "SELECT COUNT(*) FROM notifications WHERE state='QUEUED'"

  Warning  KEDAScalerFailed      104s (x18 over 12m)  scale-handler  error parsing MySQL metadata: missing required field in scaler config: no host given
  Warning  ScaledJobCheckFailed  104s (x18 over 12m)  keda-operator  Failed to ensure ScaledJob is correctly created

3.- Configuring the host as a var works for host but waits for password parameter

... omitting secret

---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: mysql-auth
  namespace: module-cwh-test3
spec:
  secretTargetRef:
  - key: mysql_conn_str
    name: mysql-secrets
    parameter: connectionString
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: myscaled-job 
  namespace: module-cwh-test3    
spec:
  jobTargetRef:
    parallelism: 1                            
    completions: 1                            
    activeDeadlineSeconds: 600              
    backoffLimit: 6 
    template:
      metadata:
        labels:
          run: ubuntu-tooling-lap
      spec:
        restartPolicy: Never
        containers:
        - image: public.ecr.aws/unicc/ubuntu-tooling:stable
          name: tooling
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: mysql-secrets
          resources:
            limits:
              cpu: "1"
              memory: "512Mi"
            requests:
              cpu: "300m"
              memory: "256Mi"
  triggers:
    - type: mysql
      authenticationRef:
        name: mysql-auth
      metadata: 
        host: database-1.c3hwsnntysui.eu-west-1.rds.amazonaws.com
        dbName: database-1
        port: "3306"
        username: admin
        queryValue: "1"
        query: "SELECT COUNT(*) FROM notifications WHERE state='QUEUED'"

  Warning  ScaledJobCheckFailed  3s (x20 over 12m)   keda-operator  Failed to ensure ScaledJob is correctly created
  Warning  KEDAScalerFailed      3s (x6 over 10m)    scale-handler  error parsing MySQL metadata: no password given
  1. Configuring the password as env var works as expected

Logs from KEDA operator

Error no host given

2024-01-19T15:43:38Z	ERROR	Reconciler error	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"myscaled-job","namespace":"module-cwh-test3"}, "namespace": "module-cwh-test3", "name": "myscaled-job", "reconcileID": "68dc9b5f-3c24-4772-a810-516cc73b6f73", "error": "error parsing MySQL metadata: missing required field in scaler config: no host given"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-01-19T15:44:14Z	INFO	Reconciling ScaledJob	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"myscaled-job","namespace":"modu

Error no password given

2024-01-19T15:49:33Z	ERROR	Failed to ensure ScaledJob is correctly created	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"myscaled-job","namespace":"module-cwh-test3"}, "namespace": "module-cwh-test3", "name": "myscaled-job", "reconcileID": "eb6a6edd-fccd-498f-b1ca-39d4ec238fb4", "error": "error parsing MySQL metadata: no password given"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
	/workspace/controllers/keda/scaledjob_controller.go:149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-01-19T15:49:33Z	ERROR	Reconciler error	{"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "ScaledJob": {"name":"myscaled-job","namespace":"module-cwh-test3"}, "namespace": "module-cwh-test3", "name": "myscaled-job", "reconcileID": "eb6a6edd-fccd-498f-b1ca-39d4ec238fb4", "error": "error parsing MySQL metadata: no password given"}

KEDA Version

2.12.1

Kubernetes Version

Other

Platform

Amazon Web Services

Scaler Details

MySQL

Anything else?

No response
@ferrandinand ferrandinand added the bug Something isn't working label Jan 19, 2024
@JorTurFer
Copy link
Member

Hello,
Could you confirm that your connectionString follows this format? user:password@tcp(mysql:3306)/stats_db
If your connection doesn't follow that format, you will have to provide ALL the connection parameters. I mean, connection string and auth parameters are mutually exclusive, so if you set connection string other authentication parameters aren't read. You can see the logic here:

keda/pkg/scalers/mysql_scaler.go

Lines 98 to 106 in 072fcf0

switch {
case config.AuthParams["connectionString"] != "":
meta.connectionString = config.AuthParams["connectionString"]
case config.TriggerMetadata["connectionStringFromEnv"] != "":
meta.connectionString = config.ResolvedEnv[config.TriggerMetadata["connectionStringFromEnv"]]
default:
meta.connectionString = ""
var err error
host, err := GetFromAuthOrMeta(config, "host")

@ferrandinand
Copy link
Author

ferrandinand commented Jan 22, 2024 •
edited
Loading

Hi @JorTurFer thanks for your help.
Yes the connection string in the secret follows that format. Even I face a similar error (not able to parse variable from a TriggerAuthentication) using the connectionString example present in the mysql scaler documentation.

If I modify the secret to use exactly the same base64 secret. I get the following error:

  Warning  KEDAScalerFailed      8s (x12 over 18s)  scale-handler  error parsing MySQL metadata: missing required field in scaler config: no host given
  Warning  ScaledJobCheckFailed  8s (x12 over 18s)  keda-operator  Failed to ensure ScaledJob is correctly created

I would expect something like: not able to connect to mysql

This is the k8s config I am trying to make work. Am I missing something?

---
apiVersion: v1
data:
  # base64 encoded value of mysql connectionString of format user:password@tcp(mysql:3306)/stats_db
  mysql_conn_str: dXNlckB0Y3AobXlzcWw6MzMwNikvc3RhdHNfZGI=
kind: Secret
metadata:
  name: mysql-secrets
  namespace: module-cwh-test3
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: mysql-auth
  namespace: module-cwh-test3
spec:
  secretTargetRef:
  - key: mysql_conn_str
    name: mysql-secrets
    parameter: connectionString
---
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: myscaled-job 
  namespace: module-cwh-test3    
spec:
  jobTargetRef:
    parallelism: 1                            
    completions: 1                            
    activeDeadlineSeconds: 600              
    backoffLimit: 6 
    template:
      spec:
        restartPolicy: Never
        containers:
        - image: public.ecr.aws/unicc/ubuntu-tooling:stable
          name: tooling
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: mysql-secrets
  triggers:
    - type: mysql
      authenticationRef:
        name: mysql-auth
      metadata: 
        queryValue: "1"
        query: "SELECT COUNT(*) FROM notifications WHERE state='QUEUED'"

@JorTurFer
Copy link
Member

That's weird because we use connectionString as the authentication option for e2e tests:

keda/tests/scalers/mysql/mysql_test.go

Line 299 in a815f06

base64MySQLConnectionString := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s@tcp(mysql.%s.svc.cluster.local:3306)/%s", mySQLUsername, mySQLPassword, testNamespace, mySQLDatabase)))

Let me check your config, just in case

@JorTurFer
Copy link
Member

I've testes the files you sent and I have this error:

2024-01-22T16:59:44Z	ERROR	scale_handler	error resolving auth params	{"type": "ScaledJob", "namespace": "module-cwh-test3", "name": "myscaled-job", "triggerIndex": 0, "error": "error establishing MySQL connection: dial tcp: lookup mysql on 10.0.0.10:53: no such host"}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
	/workspace/pkg/scaling/scalers_builder.go:98
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).performGetScalersCache
	/workspace/pkg/scaling/scale_handler.go:356
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).GetScalersCache
	/workspace/pkg/scaling/scale_handler.go:281
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).reconcileScaledJob
	/workspace/controllers/keda/scaledjob_controller.go:192
github.com/kedacore/keda/v2/controllers/keda.(*ScaledJobReconciler).Reconcile
	/workspace/controllers/keda/scaledjob_controller.go:148
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227

Basically, an error trying to connect to an invalid host

@ferrandinand
Copy link
Author

let me close the issue.
After your message, I redeployed in a different clean cluster and worked as you refered.
I removed the objects, uninstalled keda and its CRD and then it started to work.

Thanks for your help

@github-project-automation github-project-automation bot moved this from To Triage to Ready To Ship in Roadmap - KEDA Core Jan 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ferrandinand @JorTurFer