Explicitly set Delta table props to accommodate for different defaults [databricks]

[gerashegalov]

Addresses #11541

Table properties should be set unconditionally to accommodate diverging defaults in different Databricks versions

Standardize table creation to be via SQL

Improves

env TEST_PARALLEL=0 \
    TEST_MODE=DELTA_LAKE_ONLY TESTS=delta_lake_update_test.py \
    PYSP_TEST_spark_rapids_sql_detectDeltaLogQueries=false \
    PYSP_TEST_spark_rapids_sql_format_parquet_reader_type=PERFILE \
./jenkins/databricks/test.sh 

[gerashegalov]

build

[gerashegalov]

build

[revans2]

What happens if we enable deletion_vectors and use_cdf is False, but we are not on databrick 122 or later? I know that this didn't really change from the previous code. It is just confusing to me and I would like to understand better what is happening.

nit: Also why is writer.mode("append") in a separate if clause right after this one?

[gerashegalov]

We cannot enable_deleltion_vectors before 12.2, hence all tests where enable_deletion_vectors may be True are

@pytest.mark.skipif(not supports_delta_lake_deletion_vectors(), reason="Deletion vectors are new in Spark 3.4.0 / DBR 12.2")

re: write.mode("append"). I did not want to introduce changes beyond the absolute minimum.

[revans2]

OK I understand it now.

[gerashegalov]

build

[gerashegalov]

build

[mythrocks]

This is much cleaner.

[gerashegalov]

Fixing DBR 12.2 failures
test_delta_delete_partitions

  pyspark.errors.exceptions.AnalysisException: Cannot write to already existent path file:/tmp/pyspark_tests/0122-204345-nozez35d-10-59-175-195-gw1-24259-1671166402/DELTA_DATA/CPU without setting OVERWRITE = 'true'.

This appears to be related to the parametrization for use_cdf when the value is False. It indicates an issue with the test on 12.2 where the default is probably True. So the test was executing the same callpaths for either parameter prior to this PR fixing exactly this kind of problems.

[gerashegalov]

build

[razajafri]

Fixing DBR 12.2 failures test_delta_delete_partitions

  pyspark.errors.exceptions.AnalysisException: Cannot write to already existent path file:/tmp/pyspark_tests/0122-204345-nozez35d-10-59-175-195-gw1-24259-1671166402/DELTA_DATA/CPU without setting OVERWRITE = 'true'.

This appears to be related to the parametrization for use_cdf when the value is False. It indicates an issue with the test on 12.2 where the default is probably True. So the test was executing the same callpaths for either parameter prior to this PR fixing exactly this kind of problems.

I believe it's happening because you are still saving the dataframe right after you are using SQL to write the table.

[razajafri]

Saving right after you have already created a table will cause a DELTA_PATH_EXISTS error to be thrown

[gerashegalov]

This is handled by the append mode

[razajafri]

Nevermind, I see that you are setting the mode to append

[gerashegalov]

build

[revans2]

Can you please coordinate with @razajafri on how we want to do this? #12048 is doing something similar, but very differently.

I think I prefer this method because it is using the front door for setting up the table properties instead of using a config that is not well documented.

But at the same time I think we need what @razajafri has with the parametrization of the tests so that we don't run with deletion vectors in situations we don't support them and/or situations where delta does not support them. Unless we are planning to have multiple distinct tests for each case.

1. run with deletion vectors if they are supported by the plugin and delta lake and verify the result is correct
2. run with deletion vectors if delta lake supports them, but we do not and verify that we fall back to the CPU
3. run with deletion vectors disabled and verify that the result is correct.

It is more complex if when the test involves multiple delta lake operations i.e. read + write.

[revans2]

Never mind about coordinating with @razajafri I see you two have already done it and I am the one out of sync.