Skip to content

Commit

Permalink
Do not reset DB in CI tests if not needed
Browse files Browse the repository at this point in the history
The apache#43979 refactoring of tests caused unnecessary database reset
attempts in tests that did not require it or had no database set.

This caused unnecessary `airflow db reset` in collection-only tests
with removed non-ARM packages, but also it caused the error printed
in non-DB tests:

```
Resetting the DB

[2024-11-16T03:50:37.812+0000] {cli_parser.py:67} ERROR - Failed to load CLI commands from executor: LocalExecutor
Traceback (most recent call last):
  File "/opt/airflow/airflow/cli/cli_parser.py", line 64, in <module>
    executor, _ = ExecutorLoader.import_executor_cls(executor_name)
  File "/opt/airflow/airflow/executors/executor_loader.py", line 285, in import_executor_cls
    return _import_and_validate(executor_name.module_path), executor_name.connector_source
  File "/opt/airflow/airflow/executors/executor_loader.py", line 282, in _import_and_validate
    cls.validate_database_executor_compatibility(executor)
  File "/opt/airflow/airflow/executors/executor_loader.py", line 327, in validate_database_executor_compatibility
    if engine.dialect.name == "sqlite":
AttributeError: 'NoneType' object has no attribute 'dialect'
[2024-11-16T03:50:37.813+0000] {cli_parser.py:68} ERROR - Ensure all dependencies are met and try again. If using a Celery based executor install a 3.3.0+ version of the Celery provider. If using a Kubernetes executor, install a 7.4.0+ version of the CNCF provider
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/opt/airflow/airflow/__main__.py", line 62, in main
    args.func(args)
  File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
    return func(*args, **kwargs)
  File "/opt/airflow/airflow/cli/commands/db_command.py", line 63, in resetdb
    print(f"DB: {settings.engine.url!r}")
AttributeError: 'NoneType' object has no attribute 'url'

Database has been reset
```

The fix is to add `--no-db-reset` in collection tests and force
db_reset = False in case `skip_db_tests` is set to True.
  • Loading branch information
potiuk committed Nov 16, 2024
1 parent d8bff00 commit 6bf3be4
Show file tree
Hide file tree
Showing 18 changed files with 391 additions and 369 deletions.
16 changes: 8 additions & 8 deletions dev/breeze/doc/images/output-commands.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
16 changes: 8 additions & 8 deletions dev/breeze/doc/images/output_shell.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_shell.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
706a617a798c2d5a0f64d356cab3ef96
369acf181f53ebddbaf928f4743e0014
16 changes: 8 additions & 8 deletions dev/breeze/doc/images/output_start-airflow.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_start-airflow.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
6821f495c57914229fd83e5ab850e38f
3c833010b2dc87dd333e69700a1b7073
102 changes: 53 additions & 49 deletions dev/breeze/doc/images/output_testing_core-integration-tests.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1 +1 @@
7eb5ce3a9b1e3aa95bfe0bc66f2ba06a
5894cf887269f4cc5da2b0c5fb015685
182 changes: 93 additions & 89 deletions dev/breeze/doc/images/output_testing_core-tests.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_testing_core-tests.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
08246b54e85155d59d8a2626cea6e093
802a9c84c375b0c45706af3586080783
104 changes: 54 additions & 50 deletions dev/breeze/doc/images/output_testing_providers-integration-tests.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1 +1 @@
aee7e27c7ce9648aec22cb696da4afb6
b49b90e4e681c9a0d66187bc086437e0
206 changes: 105 additions & 101 deletions dev/breeze/doc/images/output_testing_providers-tests.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_testing_providers-tests.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
52adde67bdd7a3a4c9c62eaa941bd317
7e368768c7b33789e242b4fc1bd4dfe9
94 changes: 49 additions & 45 deletions dev/breeze/doc/images/output_testing_system-tests.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion dev/breeze/doc/images/output_testing_system-tests.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
e9585ade0c8a09928c98991c55579621
7fc5b82d9be11a1262c99f0b58a31fa7
4 changes: 3 additions & 1 deletion dev/breeze/src/airflow_breeze/commands/common_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,11 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
)
option_db_reset = click.option(
"-d",
"--db-reset",
"--db-reset/--no-db-reset",
help="Reset DB when entering the container.",
is_flag=True,
default=False,
show_default=True,
envvar="DB_RESET",
)
option_debug_resources = click.option(
Expand Down
2 changes: 1 addition & 1 deletion dev/breeze/src/airflow_breeze/commands/testing_commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -1113,7 +1113,7 @@ def _run_test_command(
use_packages_from_dist=use_packages_from_dist,
use_xdist=use_xdist,
run_tests=True,
db_reset=db_reset,
db_reset=db_reset if not skip_db_tests else False,
)
rebuild_or_pull_ci_image_if_needed(command_params=shell_params)
fix_ownership_using_docker()
Expand Down
4 changes: 2 additions & 2 deletions scripts/ci/testing/run_unit_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ function core_tests() {
set +x
elif [[ "${TEST_SCOPE}" == "ARM collection" ]]; then
set -x
breeze testing core-tests --collect-only --remove-arm-packages --test-type "All"
breeze testing core-tests --collect-only --remove-arm-packages --test-type "All" --no-reset-db
set +x
elif [[ "${TEST_SCOPE}" == "System" ]]; then
set -x
Expand Down Expand Up @@ -82,7 +82,7 @@ function providers_tests() {
set +x
elif [[ "${TEST_SCOPE}" == "ARM collection" ]]; then
set -x
breeze testing providers-tests --collect-only --remove-arm-packages --test-type "All"
breeze testing providers-tests --collect-only --remove-arm-packages --test-type "All" --no-reset-db
set +x
elif [[ "${TEST_SCOPE}" == "System" ]]; then
set -x
Expand Down

0 comments on commit 6bf3be4

Please sign in to comment.