Support Loki structured metadata

[Preisschild]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Structured metadata offers a way to attach metadata labels to logs without indexing them.

Currently, we only save some metadata and send it to Loki by using labels. We drop most of the other metadata because they have a high cardinality, which should not be used as labels according to the Loki best practices, because indexing them leads to bad performance.

Loki now offers a way to add labels without indexes, which would make using labels with a high cardinality possible.

If this feature is implemented, Loki users could then query based on structured metadata like this:

{job="example"} | trace_id="0242ac120002"

Where job is using the existing labels and trace_id is an attached structured metadata label.

Here is an excerpt from the above-linked Loki Labels Best Practices:

If you often parse a label from a log line at query time, the label has a high cardinality, and extracting that label is expensive in terms of performance; consider extracting the label on the client side attaching it as structured metadata to log lines .

Attempted Solutions

No response

Proposal

Vector's Loki sink should have additionally to labels an optional structured_metadata field.

Here is what this looks like in Promtail: https://grafana.com/docs/loki/latest/send-data/promtail/stages/structured_metadata/

The Loki API used to send those metadata fields is just an extension of the already implemented push API, but instead of the events looking like this:

{
    [ "<timestamp>", "<log line>" ]
}

it works like this (trace_id and user_id are the structured metadata fields)

{
    [ "<timestamp>", "<log line>", {"trace_id": "0242ac120002", "user_id": "superUser123"}]
}

References

No response

Version

v0.33.0

[emikitas]

Hi, I can take this (unless we don't want it)

[mzupan]

this would help out a lot for people that want better search performance with high cardinality

[StephenWakely]

The concern I have before we embark on this is:

Warning: Structured metadata is an experimental feature and is subject to change in future releases of Grafana Loki. This feature is not yet available for Cloud Logs users.

Do we have any insight as to how likely this is to change in future and how much it may change? I would prefer not to take on the additional maintenance if we are going to have to update this feature regularly until it becomes stable.

[precisionconage]

According to a Grafana blog post today, structured metadata will be GA in Loki 3.0, which they mention later in the post will be "coming soon". I imagine the feature won't change drastically after that.

[jszwedko]

Agreed, it seems unlikely to change after Loki 3.0. We'd be happy to see a contribution to support it then.

[tai-nd]

Structured metadata: The Structured Metadata feature, which was introduced as experimental in release 2.9.0, is generally available as of release 2.9.4.

As of Loki 2.9.4, the Structured metadata feature was marked as generally available.
https://grafana.com/docs/loki/latest/release-notes/v2-9/#features-and-enhancements

[dmitryvakulenko]

Loki 3.0 is out :)

[jszwedko]

Indeed it is! I'm not sure when we (the core team) would get to this ourselves, but happy to see a PR if someone wants to take a shot.

[maxboone]

@jszwedko I would like to pick this up in the coming week, should I just start working on a PR or ask to have the issue assigned?

[jszwedko]

@maxboone nice! That would be fantastic. I assigned you to this issue just so people are aware you are working on it. Let us know if you have any questions or need any pointers! We'd be happy to provide feedback on the PR.

[slim-bean]

Hey folks! I work on the Loki project and would like to offer my assistance in answering questions here or helping with this implementation, please let me know if I can help! I'm not sure I can help writing rust but I can help answer questions from the Loki side.

[maxboone]

"In the coming week" didn't end up happening, but I've started work on it just now and expect to send in a PR by the end of the week!

@jszwedko the main thing that I am wondering about is how to stay backwards compatible with older Loki versions, as the protobuf definition (that includes structured metadata) is different - but I guess we can better work that out from the PR, just a heads-up that I'll be asking for comments on that (:

[miadabrin]

Thank you for this much needed feature. Any updates on the ETA of merge/release?

[maxboone]

Thank you for this much needed feature. Any updates on the ETA of merge/release?

I am currently vacationing (with laptop, but less responsive) but have the PR with the implementation as ready as I could get it - awaiting review. I'll poke the reviewer there again!

If you really need it you can test a build of the PR branch, and maybe see whether it works as expected?

[miadabrin]

@maxboone sounds good thank you

[jszwedko]

Closed by #20576 (comment)

[miadabrin]

@jszwedko do you think we will have a release for this feature or should we use a nightly build for the time being?

[jszwedko]

@jszwedko do you think we will have a release for this feature or should we use a nightly build for the time being?

It will be included in v0.42.0 which is scheduled for next week.