Pass output values from Dagster asset to @dlt_assets decorator

[bruk0]

Hi, I am looking for a solution to pass value dynamically from Dagster asset to @dlt_asset decorator

@asset
def pipeline_config_asset(context:AssetExecutionContext, config: PipelineConfig) -> int:
    context.log.info(config.start_date)
    context.log.info(config.queries)
    context.log.info(config.property_id)
    context.log.info(config.organization_id)
    return config.property_id, config.start_date


@dlt_assets(
    dlt_source=google_analytics(
        property_id=pipeline_config_asset[0],
        start_date='',
        queries='',
    ),
    dlt_pipeline=pipeline(
        pipeline_name='',
        destination='bigquery',
        dataset_name='',
    ),
    name='',
    group_name='Google_Analytics',
    dlt_dagster_translator=CustomDagsterDltTranslator()
)

Is there a way to do that? Because, when I try to call pipeline_config_asset[0] I get an error message that AssetDefinition is not subscriptable.

The AI assistant didn't provide a solution for that.
Thanks in advance for your time, effort and help!

[mlarose]

Hi @bruk0,

What did you expect pipeline_config_asset[0] to return? The error about the AssetDefinition not being subscriptable is that the object doesn't support the [i] operator.

What is it you want to share between the two assets?

[bruk0]

Hi, @mlarose . I want to pass data to @dlt_assets decorator from Dagster asset. The output of Dagster asset is tuple. In @dlt_assets I want to pass those values. I use accessor to get value by index

[mlarose]

No, you can't pass the output of an asset to define another asset in this way. But I am wondering, given that your example is passing config through an asset, if you are really using the right concepts and constructs. That's why I am inquiring about your use case and what information needs to be shared between these assets definitions?

The output of the asset will be available at run time when you are materializing it. However, asset definitions are consumed when your code is initially loaded but before anything is executed. Therefore, your asset definitions cannot depend of compute results from other assets.

Also, just to point out the python error there, if we ignore the fact that this is an asset, and just any function, you would have to invoke the function to get the return value. Here you're passing the function object. ex:

def my_tuple() -> Tuple[str, str]:
    return "hi", "hello"
    
my_tuple
<function my_tuple at 0x1031bb240>

my_tuple()
('hi', 'hello') 

[bruk0]

Thank you, @mlarose for your response. In this case, how can i build Dagster job to pass necessary parameters to @dlt_assets? Because I retrieve information about Source GA and BigQuery destination secrets and other information using API calls from our platform and Dagster wants to be them already initialized, in case of dlt asset?

[bruk0]

Maybe, I misunderstood you, but I have a Dagster job where I have a dependency between assets. I could use value of upstream assets in downstream one

[mlarose]

Because I retrieve information about Source GA and BigQuery destination secrets and other information using API calls from our platform and Dagster wants to be them already initialized, in case of dlt asset?

To manage configuring and reusing connection to external systems such as database, what's the Dagster way to configure your pipelines is to use resources. And you'll probably need to pass some credentials to setup those connections, so I am also adding a reference to env vars and secrets.

• https://docs.dagster.io/concepts/resources
• https://docs.dagster.io/dagster-plus/managing-deployments/environment-variables-and-secrets

Maybe, I misunderstood you, but I have a Dagster job where I have a dependency between assets. I could use value of upstream assets in downstream one

Seems that's a different task altogether. There's a built-in concept of passing i/o from one asset to another leveraging IOManagers. See this page for more information and examples:

• https://docs.dagster.io/concepts/io-management/io-managers