Patterns/rules for applying function level instrumentation

[ctron]

We've been applying instrumentation (like ``#[instrument]`) on functions when necessary. And that's fine.

However, I am not sure what the right approach is when it comes to handling fields and return/errors. This also touches a bit on the log levels. Also, I do it wrong myself. So I'd appreciate a guideline I could look a too.

Checking back with syslog (RFC5424) there are severities defined like this:

Emergency: system is unusable
Alert: action must be taken immediately
Critical: critical conditions
Error: error conditions
Warning: warning conditions
Notice: normal but significant condition
Informational: informational messages
Debug: debug-level messages

I know, we don't have all of them in the log levels, but close enough.

I guess everything of error and above is the same for us. Above error might be a panic on the console. But that's not part of the log system of the application.

When taking a look at what we do, and what tracing does by default, then that's spamming a lot of information on the INFO level. Which is ok, if we assume that the application will be run on WARN. Because other than a developer debugging, only WARN and above would be important for an operator of the software.

For instrumenting we mostly have #[instrument(…, ret)] or #[instrument(…, err)]. Which logs the span with the result on INFO, of the error on ERROR. However, an error on a function isn't necessary a log worthy ERROR for the person operating this software.

So I think we need a set of rules, maybe with examples, that explain how to consistently apply instrumentation and log levels.