Fixed the behavior of the incremental schema change ignore option to …

…properly handle the scenario when columns are dropped (#580)

Co-authored-by: Ben Cassell <[email protected]>

CHANGELOG.md

@@ -1,15 +1,22 @@
-## dbt-databricks 1.7.7 (TBD)
+## dbt-databricks 1.7.8 (TBD)
 
 ### Features
 
 - Support `on_config_change` for materialized views, expand the supported config options ([536](https://github.com/databricks/dbt-databricks/pull/536)))
 
-## dbt-databricks 1.7.6 (Feb 6, 2024)
+### Fixes
+- Fixed the behavior of the incremental schema change ignore option to properly handle the scenario when columns are dropped (thanks @case-k-git!) ([580](https://github.com/databricks/dbt-databricks/pull/580))
+
+## dbt-databricks 1.7.7 (Feb 6, 2024)
 
 ### Fixes
 
 - Rollback databricks-sql-connector to 2.9.3 to actually fix connection timeout issue ([578](https://github.com/databricks/dbt-databricks/pull/578))
 
+## dbt-databricks 1.7.6 (Feb 6, 2024)
+
+Skipped due to incorrect files in deployed package
+
 ## dbt-databricks 1.7.5 (Jan 30, 2024) (Pulled due to poor SQL Warehouse connection behavior)
 
 ### Fixes

dbt/include/databricks/macros/materializations/incremental/strategies.sql

@@ -50,9 +50,15 @@
 
 {% macro get_insert_into_sql(source_relation, target_relation) %}
 
-    {%- set dest_columns = adapter.get_columns_in_relation(target_relation) -%}
-    {%- set dest_cols_csv = dest_columns | map(attribute='quoted') | join(', ') -%}
-    insert into table {{ target_relation }}
+    {%- set common_columns = [] -%}
+    {%- set source_columns = adapter.get_columns_in_relation(source_relation) | map(attribute="name") -%}
+    {%- for dest_col in adapter.get_columns_in_relation(target_relation) -%}
+      {% if dest_col.name in source_columns %}
+          {%- if common_columns.append(dest_col) -%}{%- endif -%}
+      {%- endif -%}
+    {%- endfor -%}
+    {%- set dest_cols_csv = common_columns | map(attribute='quoted') | join(', ') -%}
+    insert into table {{ target_relation }} ({{ dest_cols_csv }})
     select {{dest_cols_csv}} from {{ source_relation }}
 
 {% endmacro %}