CBG-4187: Add stat to track number of assertion failures

[bbrks]

CBG-4187

Add stat to track number of assertion failures

Integration Tests

• n/a

[torcolvin]

Should we say what an assertion failure is? Something like:

Assertions count is already included in the warning stat and mark problems that are not caused by users and are a software precondition.

This message isn't good, and but for these failures there's no meaningful action a user could take. That's the difference I see between these and other warnings, which could be things that a user could act upon.

Calling AssertfCtx will call WarnfCtx directly and this value already increments the WarnCount metric. We also have no way to determine that a message was an assertion failure vs a standard warning from a log file.

I don't know how useful this stat is without being easily able to ID which are assertion failures from the log files without doing a search for the messages.

I don't know if it makes sense to just add a prefix like "[WRN] Assertion Error: msg" or change AssertfCtx -> WarnfCtx -> logTo to put a new log level? I don't think that a log level change is right because I think that will break the loose log compatibility we have.

Finally, does it actually make sense to log these as errors?

I'm not opposed to the stat, I just want to have it be clear from the description how dev, support, end users are intended to use it. Should this be specially flagged from nutshell, etc as something that needs immediate dev attention? Do we expect a special alerting from capella?