dbt-maxcompute v1.9.0-a6

Features:

• Added support for dbt source freshness functionality.

Fixes:

• Fixed an issue where using the incremental materialization with the insert_overwrite strategy would cause errors for auto-partitioned tables.
• Fixed an issue where temporary tables created using the incremental materialization were not automatically deleted.

dbt-maxcompute v1.9.0-a5

Features:

• Adaptation to the latest MaxCompute SQL syntax:
  • DATE_TRUNC now supports datepart values: 'quarter', 'isoweek', 'week', and 'weekday'
  • DATEADD now supports datepart values: 'millisecond' and 'microsecond'
  • DATEDIFF now supports datepart values: 'microsecond', 'isoweek', 'week', and 'weekday'
• persist_doc now supports view materialization
• New calculate_freshness_from_metadata function for calculating data freshness
• lifecycle parameter support for table and incremental materialization to specify table lifecycle

Fixes:

• Resolved an issue where specifying a partition column as the first column during materialization would add an extra comma.

dbt-maxcompute v1.9.0-a4

BugFix

• Fixed an issue where the drop_relation function failed to properly delete views.

dbt-maxcompute v1.9.0-a3

Important Changes

In dbt, there are two styles of incremental materialization logic for insert_overwrite:

• Original Logic: Maintained by dbt-adapters, it requires specifying a unique_key to deduplicate data based on the unique key.
• Enhanced Logic: Used by dbt-bigquery, it requires specifying a partition_by field and overwrites only the new partition each time.

In the v1.9.0-a0 release, dbt-maxcompute supported both logics and named them insert_overwrite (original logic) and bq_insert_overwrite (enhanced logic). However, considering the following reasons:

• The majority of MaxCompute tables are partitioned tables, and users utilize the enhanced logic (bq_insert_overwrite) far more frequently than the original logic.
• Maintaining two sets of logic incurs higher costs.

We have decided to adopt the enhanced logic as the default implementation and made the following adjustments:

• Renamed bq_insert_overwrite to insert_overwrite.
• Renamed bq_microbatch to microbatch.
• Removed the original insert_overwrite and microbatch logic.

Note: This change may impact existing project configurations. Please update your code according to the new naming conventions.

New Features

View Creation Optimization

• Added the OR REPLACE keyword when creating views to ensure successful updates if the view already exists, avoiding conflicts caused by duplicate views.

Enhanced Authentication Methods

• Introduced the alibabacloud_credential module, supporting multiple authentication methods, including:
  • STSToken
  • RAM Role
  • credential provider chain
  • etc.
• For more information, refer to the documentation.

Bug Fixes

• Resolved compatibility issues with pyodps 0.12.2, ensuring smooth operation with the latest dependency libraries.
• Fixed an issue where the default schema specified in raw materialization mode was inconsistent with the actual runtime schema, improving data consistency.

dbt-maxcompute v1.9.0-a1

New Features

Raw Materialization Enhancements

• Added new configuration options:

  • schema: Specifies the default schema for the current query
  • sql_hints: Allows users to specify SQL hints for the current query

  Note: Other materialization methods only support schema specification in the dbt.yaml file

Example Usage:

{{ config(
    materialized="raw",
    schema="new_schema",
    sql_hints={"odps.sql.submit.mode": "script"}
) }}

create schema new_schema;
create table table_in_new_schema(c1 bigint);

Bug Fixes

• Fixed logical issues in the bq_insert_overwrite incremental materialization strategy and improved execution efficiency

v1.9.0-a0

dbt-maxcompute 1.9.0a0

New Features

• Compatibility with dbt-core 1.9.x: This version has been adapted to work seamlessly with dbt-core version 1.9.x.

• New Incremental Materialization Strategies:

  • microbatch
  • bq_insert_overwrite
  • bq_microbatch

• New Materialization Method:

  • raw: A new materialization method that supports manual SQL execution by users.

Note

Due to the distinct nature of the incremental materialization methods offered by dbt-bigquery, the micro provided in the dbt-adapters has been deeply customized to reduce the costs associated with using incremental materialization on partitioned tables. As a result, bq_insert_overwrite and bq_microbatch have been added to mimic the materialization method found in dbt-bigquery.

Key Differences

• insert_overwrite: This method removes duplicates based on specified unique_keys.
• bq_insert_overwrite: This method removes duplicates based on the partition_by condition and allows for specifying which partitions to work with.

Usage Examples

bq_insert_overwrite (Static)

{{ config(
    materialized='incremental',
    partition_by={"fields": "some_date", "data_types": "timestamp"},
    partitions=["TIMESTAMP'2024-10-10 00:00:00'", "TIMESTAMP'2025-01-01 00:00:00'"],
    incremental_strategy='bq_insert_overwrite'
) }}
select * from {{ source('raw', 'seed') }}
{% if is_incremental() %}
   where {{ dbt.datediff("some_date", "TIMESTAMP'2000-01-01 00:00:00'", 'day') }} > 0
{% endif %}

bq_insert_overwrite (Dynamic)

{{ config(
    materialized='incremental',
    partition_by={"fields": "some_date", "data_types": "timestamp"},
    incremental_strategy='bq_insert_overwrite'
) }}
select * from {{ source('raw', 'seed') }}
{% if is_incremental() %}
   where {{ dbt.datediff("some_date", "TIMESTAMP'2000-01-01 00:00:00'", 'day') }} > 0
{% endif %}

Raw Materialization Example

{{ config(
    materialized='raw'
) }}
create table test(c1 bigint) lifecycle 1;

This release enhances the functionality of dbt-maxcompute significantly by providing improved incremental strategies and a new raw materialization method, making it easier for users to manage their data transformations effectively.

dbt-maxcompute v1.8.0-a13

New Features

1. Support for Creating Ordinary Partition Tables via partition_py

Users can now utilize partition_py to create ordinary partition tables for better data management and querying.

2. Support for Creating Materialized Views

Support for materialized views has been introduced, enabling users to query and manage data more efficiently.

Usage

Creating an Ordinary Partition Table

To create an ordinary partition table, you can use the following configuration:

{{ config(
    materialized='table',
    partition_by={"fields": "name,some_date", "data_types": "string,string"}
) }}
select id, name, some_date from {{ source('raw', 'seed') }}

Schema of the materialized table like:

create table model(id bigint) partitioned by(name string, some_date string)

Creating an Auto-Partitioned Table

Here is an example configuration for creating an auto-partitioned table:

{{ config(
  materialized='table',
  partition_by={"fields": "some_date", "data_types": "timestamp", "granularity": "day"}
) }}
select id, name, some_date from {{ source('raw', 'seed') }}

Schema of the materialized table like:

create table model(id bigint, name string, some_date timestamp) auto partitioned by trunc_time(some_date, "day");

Creating a Materialized View

The configuration for creating a materialized view is as follows:

{{ config(
    materialized='materialized_view',
    lifecycle=1,
    build_deferred=True,
    columns=["id", "name", "some_date"],
    column_comment={"id": "this is id.", "name": "this is name."},
    disable_rewrite=True,
    table_comment="this is a materialized view.",
    tblProperties={"compressionstrategy":"normal"}
) }}
select * from {{ source('raw', 'seed') }}

For the meanings of each field, please refer to the Alibaba Cloud MaxCompute User Guide.

Limitations

1. The current version does not support specifying partitions when refreshing materialized views.
2. Materialized views do not support the rename operation.
3. It is not possible to switch to another materialization method without dropping the materialized table.

dbt-maxcompute v1.8.0-a12

New Features

• Support for specifying partition_py during table materialization to automatically create auto-partitioned tables.
• Introduced a new date_spine macro for generating date dimension tables.
• Modified the dateadd macro to add support for hour and week partitions.
• Refactored the datediff macro to simplify it and support more date intervals.

Full Changelog: v1.8.0-a11...v1.8.0-a12

dbt-maxcompute v1.8.0-a11

What's Changed

• feat: 增加 insert_overwrite strategy 的支持，支持创建 auto-partition 表 by @dingxin-tech in #4

Full Changelog: v1.8.0-a10...v1.8.0-a11

dbt-maxcompute v1.8.0-a10

Version: v1.8.0-a10
Release Date: 2024.12.04

New Features

• Micro Related to apply_grants.sql:

  • Added micros that supports MaxCompute permission operations, enabling flexible management of data permissions through SQL statements for granting and revoking specific user or group permissions.

• create_or_replace_clone:

  • Implemented a new micro that supports the dbt clone operation, allowing users to quickly create modified or variant models from existing ones.

• persist_doc:

  • Introducing a new micro that supports persisting descriptions into a comments table in the database, allowing for the permanent storage of relevant document information for future reference.

• query_header:

  • Added support for query_header, enabling users to include custom header information during SQL query execution for additional context and improved debugging.

• Various Bug Fixes:

  • Addressed multiple known issues to enhance system stability and reliability, including error handling improvements and performance optimizations.