Block builder: update alerts and add critical alerts

[codesome]

What this PR does

Updated some warnings and add critical alerts to block builders.

• BlockBuilderNoCycleProcessing is a little sensitive. Meaning if a consumption ran a little longer than 5m compared to the earlier one (for duration is 5m), this alert will fire. Made the for period to be 20 mins
• BlockBuilderLagging current warning alert fires when the lag is over 4M for 75mins (basically lag went above 4M, but it stayed above 4M in the next cycle as well). We have a critical alert with for duration of 2h20m. Meaning once it went above 4M in a cycle, it remained about 4M in the next two cycles as well. If the cycle run was not successful, then BlockBuilderNoCycleProcessing will already fire much the lagging alert will fire. BlockBuilderLagging will catch other issues that is not just "not able to consume", like block builders are not scaled enough for example.
• If BlockBuilderCompactAndUploadFailed fired, it should be a critical alert immediately. Removed the for period.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[narqo]

I'm not sure about having two of each kind of alerts.

E.g. the "no cycles" two: to have only 10 minutes gap between a warning and a critical alert is a fairly small window for a person, who'll be paged, to do any meaningful investigations.

I suggest we only keep one but make sure the for is long enough to be critical.

[codesome]

Fair point! We can probably just keep a critical alert for 20 mins and remove the warning for 10 mins.

[codesome]

I updated the PR and the description.

I removed the BlockBuilderNoCycleProcessing warning and made for period to be 20m for the critical alert.

Remove for period for BlockBuilderCompactAndUploadFailed to fire immediately. If we have for period of 5m, it will never fire because in the query we lookback only 1 min.

[seizethedave]

LG2M.