wait-for-db-migrations Timeout error

[conchaox]

Hello,

I've been trying to upgrade from chart version 8.5.2 to 8.6.0 but I'm getting a timeout error in the web pod, these are the error logs in the wait-for-db-migrations init container:

wait-for-db-migrations Traceback (most recent call last):                                                                                                                 │
│ wait-for-db-migrations   File "/home/airflow/.local/bin/airflow", line 8, in <module>                                                                                     │
│ wait-for-db-migrations     sys.exit(main())                                                                                                                               │
│ wait-for-db-migrations   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main                                                    │
│ wait-for-db-migrations     args.func(args)                                                                                                                                │
│ wait-for-db-migrations   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command                                           │
│ wait-for-db-migrations     return func(*args, **kwargs)                                                                                                                   │
│ wait-for-db-migrations   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 54, in check_migrations                         │
│ wait-for-db-migrations     db.check_migrations(timeout=args.migration_wait_timeout)                                                                                       │
│ wait-for-db-migrations   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line 613, in check_migrations                                       │
│ wait-for-db-migrations     raise TimeoutError(                                                                                                                            │
│ wait-for-db-migrations TimeoutError: There are still unapplied migrations after 60 seconds. Migration Head(s) in DB: {'587bdf053233'} | Migration Head(s) in Source Code: │
│ Stream closed EOF for bdi-dev/bdi-dev-scheduler-78979559b5-6rqx4 (wait-for-db-migrations)\

I also looke the db-migrations pod either running as a deployment or as a pod and I can see this:

db-migrations /home/airflow/.local/lib/python3.8/site-packages/azure/storage/common/_connection.py:82 SyntaxWarning: "is" with a literal. Did you mean "=="?              │
│ db-migrations [2022-06-13 23:17:37,289] {db_migrations.py:53} WARNING - there are unapplied db migrations, triggering apply...                                            │
│ db-migrations [2022-06-13 23:17:37,289] {db_migrations.py:45} INFO - -------- START - APPLY DB MIGRATIONS --------                                                        │
│ db-migrations [2022-06-13 23:17:37,401] {db.py:919} INFO - Creating tables                                                                                                │
│ db-migrations INFO  [alembic.runtime.migration] Context impl MySQLImpl.                                                                                                   │
│ db-migrations INFO  [alembic.runtime.migration] Will assume non-transactional DDL.                                                                                        │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade a13f7613ad25 -> 97cdd93827b8, Add queued_at column to dagrun table                                        │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 97cdd93827b8 -> 092435bf5d12, Add max_active_runs column to dagmodel table                                │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 092435bf5d12 -> ccde3e26fe78, Add index on state, dag_id for queued dagrun                                │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade ccde3e26fe78 -> 83f031fd9f1c, improve mssql compatibility                                                 │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 83f031fd9f1c -> e9304a3141f0, make xcom pkey columns non-nullable                                         │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade e9304a3141f0 -> 30867afad44a, Rename concurrency column in dag table to max_active_tasks                  │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 30867afad44a -> 54bebd308c5f, Add trigger table and task info                                             │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 54bebd308c5f -> 142555e44c17, Add data_interval_[start|end] to DagModel and DagRun.                       │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 142555e44c17 -> 7b2661a43ba3, TaskInstance keyed to DagRun                                                │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade 7b2661a43ba3 -> be2bfac3da23, Add has_import_errors column to DagModel                                    │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade be2bfac3da23 -> c381b21cb7e4, add session table to db                                                     │
│ db-migrations INFO  [alembic.runtime.migration] Running upgrade c381b21cb7e4 -> 587bdf053233, adding index for dag_id in job                                              │
│ db-migrations INFO  [/mnt/scripts/db_migrations.py] -------- FINISH - APPLY DB MIGRATIONS --------

I'm using the Terraform helm_release resource to create the the deployment and using a Cloud SQL/MySQL instance as an external DB, I'm also passing some specific configs to the web server pods in order to integrate okta, config examples here:

External DB:

externalDatabase:
  type: "${var.external_db_type}"
  host: "${google_sql_database_instance.instance.private_ip_address}"
  port: "${var.external_db_port}"
  database: "${google_sql_database.database.name}"
  user: "${google_sql_user.users.name}"
  passwordSecret: airflow-mysql-password
  passwordSecretKey: mysql-password

Web server config for Okta:

webserverConfig:
    stringOverride: |-
      from flask_appbuilder.security.manager import AUTH_OAUTH
      import os
      AUTH_TYPE = AUTH_OAUTH
      AUTH_USER_REGISTRATION = True
      AUTH_USER_REGISTRATION_ROLE = "Public"
      OAUTH_PROVIDERS = [
          {
              'name': 'okta',
              'icon': 'fa-circle-o',
              'token_key': 'access_token',
              'remote_app': {
                  'client_id': os.getenv('OKTA_CLIENT_ID'),
                  'client_secret': os.getenv('OKTA_CLIENT_SECRET'),
                  'api_base_url': 'REPLACE_ME',
                  'client_kwargs': {
                      'scope': 'openid profile email groups'
                  },
                  'access_token_url': 'REPLACE_ME',
                  'authorize_url': 'REPLACE_ME',
              }
          }
      ]

I found this GitHub issue that seems to explain the issue that we are having, but I also tried what is stated there (passing wait = false) and it didn't make any difference.

Is there something specific that needs to be configured for the web pod? Is the DB upgrade happening correctly since we are using the Terraform resource?

Your help will be much appreciated.

Thanks!

[thesuperzapper]

@conchaox for context, the wait-for-db-migrations sidecar containers are meant to fail (after 60 seconds) if the database is not migrated to the correct schema (which sometimes changes between airflow releases).

In our case, we actually fixed the helm wait issue by using a Deployment for db-migrations rather than a post-install Job.

Some questions to help me understand your situation:

1. What version of airflow are you trying to run?
2. If you wait a while, does the cluster stabilize, and does the wait-for-db-migrations start succeeding (once the migrations are finished)?
3. Is your db-migrations deployment showing success/running or is it crashing?
4. Can you share your custom values (please hide any passwords, etc)?

[thesuperzapper]

Your issues about a Missing "jwks_uri" in metadata is because of authlib 1.0.0+ now requiring jwks_uri to be set.
(NOTE: jwks is REALLY important, as it's used in the OAUTH flow to verify the token is from who you expect!)

I have now raised an issue in Flask-AppBuilder to update their docs and pinned authlib version (but I will also update the chart's OAUTH/LDAP docs).

To fix this in your case, you can either:

1. Set jwks_uri to https://DOMAIN_NAME/oauth2/v1/keys (under "remote_app")
2. Use the OAUTH2 .well-known/openid-configuration with server_metadata_url, like:

OAUTH_PROVIDERS = [
  {
      "name": "okta",
      "icon": "fa-circle-o",
      "token_key": "access_token",
      "remote_app": {
          "client_id": os.getenv("OKTA_CLIENT_ID"),
          "client_secret": os.getenv("OKTA_CLIENT_SECRET"),
          "server_metadata_url": "https://OKTA_DOMAIN.okta.com/.well-known/openid-configuration",
          "client_kwargs": {"scope": "openid profile email groups"},
      },
  },
]

Can you try option 2 and report back?
(If it's all good I will update our OAUTH docs to suggest using server_metadata_url)

[conchaox]

@thesuperzapper thanks for the explanation! I tired option two but now I'm getting a different error message when clicking the button:

This is how the my OAUTH_PROVIDERS config looks like:

 OAUTH_PROVIDERS = [
          {
              "name": "okta",
              "icon": "fa-circle-o",
              "token_key": "access_token",
              "remote_app": {
                  "client_id": os.getenv("OKTA_CLIENT_ID"),
                  "client_secret": os.getenv("OKTA_CLIENT_SECRET"),
                  "server_metadata_url": "https://DOMAIN.okta.com/.well-known/openid-configuration",
                  "client_kwargs": {
                      "scope": "openid profile email groups"
                  },
                  "access_token_url": "https://DOMAIN.okta.com/oauth2/v1/token",
                  "authorize_url": "https://DOMAIN.okta.com/oauth2/v1/authorize",
              }
          }
      ]

The logs that I'm seeing now in the web pod:

airflow-web [2022-06-17 16:45:12,391] {views.py:671} ERROR - Error returning OAuth user info: Invalid URL 'userinfo': No scheme supplied. Perhaps you meant http://userin │
│ airflow-web [2022-06-17 16:45:12,391] {views.py:671} ERROR - Error returning OAuth user info: Invalid URL 'userinfo': No scheme supplied. Perhaps you meant http://userin │

is it still something missing in the config? could you please provide an example for option 1 as well?

Thanks!

[conchaox]

@thesuperzapper I finally got a working config:

webserverConfig:
    stringOverride: |-
      from flask_appbuilder.security.manager import AUTH_OAUTH
      import os

      AUTH_TYPE = AUTH_OAUTH

      AUTH_USER_REGISTRATION = True

      AUTH_USER_REGISTRATION_ROLE = "Public"

      OAUTH_PROVIDERS = [
          {
              "name": "okta",
              "icon": "fa-circle-o",
              "token_key": "access_token",
              "remote_app": {
                  "client_id": os.getenv("OKTA_CLIENT_ID"),
                  "client_secret": os.getenv("OKTA_CLIENT_SECRET"),
                  "api_base_url": "https://DOMAIN.okta.com/oauth2/v1/",
                  "server_metadata_url": "https://DOMAIN.okta.com/.well-known/openid-configuration",
                  "client_kwargs": {
                      "scope": "openid profile email groups"
                  },
                  "access_token_url": "https://DOMAIN.okta.com/oauth2/v1/token",
                  "userinfo_url": "https://DOMAIN.okta.com/oauth2/default/userinfo",
                  "authorize_url": "https://DOMAIN.okta.com/oauth2/v1/authorize",
              }
          }
      ]

added back "api_base_url" and added "userinfo_url": "https://DOMAIN.okta.com/oauth2/default/userinfo",

Hope this helps, thanks a lot for your advice!

[thesuperzapper]

@conchaox after checking, api_base_url needs to be set because Flask-AppBuilder is not actually using the in-built userInfo support in Authlib, it's manually running an HTTP GET against the URI (which requires api_base_url to be set).

This probably means that you can set api_base_url and server_metadata_url only, and it will work:

OAUTH_PROVIDERS = [
  {
      "name": "okta",
      "icon": "fa-circle-o",
      "token_key": "access_token",
      "remote_app": {
          "client_id": os.getenv("OKTA_CLIENT_ID"),
          "client_secret": os.getenv("OKTA_CLIENT_SECRET"),
          "api_base_url": "https://OKTA_DOMAIN.okta.com/oauth2/v1/",
          "server_metadata_url": "https://OKTA_DOMAIN.okta.com/.well-known/openid-configuration",
          "client_kwargs": {"scope": "openid profile email groups"},
      },
  },
]

Can you confirm that this works?

[conchaox]

@thesuperzapper removing "userinfo_url" worked as well.

Thanks a lot!