Structured / json logging by default in Dagster+

[zyd14]

What is the recommended route to enable Dagster to emit structured logs in Dagster+ by default? If you add a logger to the loggers argument of your Definitions object, the logger becomes available to your jobs, but must be opted-into by adding some configuration to the launchpad. This can't really be pre-configured with the config= argument of jobs if your configuration is dynamically set at runtime, which is the case for most of my jobs. It seems like the user needs to configure the settings in dagster.yaml - but for Dagster+, this is configured as part of the Dagster Agent container (at least for ECS hybrid deployment). So it appears that to get a custom logger to be used by default for Dagster jobs and event logs one needs to make a custom Dagster Agent container with their custom logger included and change the dagster.yaml config to point to the custom log handler with my formatter already attached to it. Is this correct? Or is there a class argument to the formatter block that I could use to point to a custom formatter? Do I even need to add the custom logging code to the Dagster Agent container since it's mostly just launching the user code container? Or is there another way entirely to extend logging in Dagster+?

I've had a few discussions with Dagster team members who have mentioned there are some vague plans for improving this, but no one has been able to give me guidance on how exactly to set up custom loggers with the current Dagster implementation. Even if it's suboptimal, I'd really appreciate some guidance on how this is meant to be configured right now. There are multiple issues and discussions up right now with no actionable input from Dagster team members on how to solve this problem, non-standard logging feels very neglected right now.

Of note, I'm using the ECS hybrid deployment right now. Willing to switch to k8s if it can get me custom logging, this is really killing a lot of plans I have to introduce much better visibility into our Dagster deployments and better tie Dagster logs to external compute logs and metrics. It sounds like any sort of first-class structured logging support is still a long ways out with no concrete plans in place right now, so even if someone can just provide a workaround while improvements are made that would be incredibly helpful. Unless of course it's just plain impossible right now, which would also be good to know so people don't spend tons of time on it.

[gibsondan]

Hi @zyd14 - firstly apologies for the bad experiences here. I'd like to see if we can fix the confusing loggers override behavior that you and many others have run into and sent out a PR to attempt to start that process here: #24338. I suspect the main blocker there is how to enable that behavior (which seems much more intuitive to me) without it being a breaking change for existing loggers.

To get a custom logger to be used by default in your Dagster+ jobs, you don't need to build a custom agent container. It's a little confusing because you specify it in the agent's dagster.yaml, but the configuration there is only read in your user code container so it should only need to be defined there.

And you can use a custom formatter class - I sent out #24331 to see if we can make the Dagster-compatible JSON logging formatter that the json_console_logger exportable as a top-level class. Once that is live, you could put something this in your agent dagster.yaml in ECS:

python_logs:
  dagster_handler_config:
    handlers:
      console:
        class: logging.StreamHandler
        formatter: json
    formatters:
      json:
       '()': dagster.JsonFormatter

(or a different custom formatter class of your choosing). When I tried that, it added JSON formatted logs like this to the job output (in addition to the default system console logger for the jobs).

One additional benefit of that approach (setting it in the dagster.yaml rather than in python code) is that it will still use that formatter for the initial events that happen before we have had a chance to load the job code yet.

[zyd14]

Hi @gibsondan - thanks for your response. We ended up with a similar approach where we installed a structlog handler in the Dagster Agent, then added it as an additional handler to the dagster.yaml. This seems to work for our needs and gets most of the Dagster logs written out as json-formatted, while still maintaining normal logs in the Dagster console through the standard handler. This does mean that currently our logs are being duplicated - we're considering having the structlog handler write out to a file and then just sucking up that file with Datadog. The next step is to figure out how we can add bound attributes to log lines at runtime, so we can add things like the Dagster run ID as log line attributes, which would make filtering and grouping logs in an aggregation service like Datadog much much simpler.

Looking forward to those PRs landing, thanks for putting in some work on them. Hopefully the path for this can become a bit more clear and customizable soon.