Field "path" of type Path in DatabendRelation has invalid value

[hantmac]

Hi , I am a developer from databend and I am writing the dbt adapter for it. But When I running the BaseSnapshotCheckCols I got the error message: Field "path" of type Path in DatabendRelation has invalid value. I have pass the BaseIncremental and BaseSimpleMaterializations. More error messages are as follows:

Looking forward to your reply, thanks!

[jtcohen6]

@hantmac Very cool!

I'm guessing the issue here is with 'database': Undefined. dbt expects to know about three:

• top level: database
• second level: schema
• object level: identifier (view, table, ...)

Does Databend only support two namespace levels for its tables? If so, you can look at how other adapters have managed this in the past, by setting 'database': None (rather than Undefined):

• within custom Relation object
• within credentials for connection

[hantmac]

@jtcohen6 Thanks for your reply!

The dbt model database.schema.table is not compatible with Databend because Databend does not support a schema. So I need to use a simple model schema.table, where schema is the Databend's database.

I will try to fix it according your answer. Thanks again!

[hantmac]

I have check my code and I write the same code with https://github.com/dbt-labs/dbt-spark/blob/b6cee1dbeb817c6e51d60b2a0c2342c5fb9c1835/dbt/adapters/spark/relation.py#L18 which you added in the comment.

class DatabendQuotePolicy(Policy):
    database: bool = False
    schema: bool = False
    identifier: bool = False


@dataclass
class DatabendIncludePolicy(Policy):
    database: bool = False
    schema: bool = True
    identifier: bool = True


@dataclass(frozen=True, eq=False, repr=False)
class DatabendRelation(BaseRelation):
    quote_policy: DatabendQuotePolicy = DatabendQuotePolicy()
    include_policy: DatabendIncludePolicy = DatabendIncludePolicy()
    quote_character: str = ""

But still has the error 🤔

[jtcohen6]

Do you have the same logic within connections.py to set database (meaning third-level thing) to None by default? I suppose you could also try that as a __post_init__ on DatabendRelation. It should be supported to have a Relation with 'database': None, just not Undefined.

[hantmac]

Yeah, I have the following code within connections.py:

 @classmethod
    def __pre_deserialize__(cls, data):
        data = super().__pre_deserialize__(data)
        if "database" not in data:
            data["database"] = None
        return data

    def __post_init__(self):
        # databend classifies database and schema as the same thing
        if (
                self.database is not None and
                self.database != self.schema
        ):
            raise dbt.exceptions.RuntimeException(
                f"    schema: {self.schema} \n"
                f"    database: {self.database} \n"
                f"On Databend, database must be omitted or have the same value as"
                f" schema."
            )
        self.database = None

And the same logic within DatabendRelation :

 def __post_init__(self):
        if self.database != self.schema and self.database:
            raise dbt.exceptions.RuntimeException(
                f"    schema: {self.schema} \n"
                f"    database: {self.database} \n"
                f"On Databend, database must be omitted or have the same value as"
                f" schema."
            )

But still has the Field "path" of type Path in DatabendRelation has invalid value {'database': Undefined, 'schema': 'test16687346852554920559_test_basic', 'identifier': 'ts_snapshot'} error 😭

[hantmac]

@jtcohen6 BTW, this is the processing data in debug and we can see that the database is None. So I don't konw where it was set to undefine 🤔.

[hantmac]

@jtcohen6 Need I overwirte the snapshoot.sql in macro directory?

[hantmac]

I have solved this problem by overwriting create method in relation.py. Thanks for your help!

[hantmac]

I have solved this problem by overwriting create method in relation.py. Thanks for your help!