alerts for weird things

[nikhiljha]

if something like that happens, there should be an alert ^

I don't know if there's a tool that makes alert for patterns like that

[jvperrin]

Do we need an alert for something like this? IMO it only matters once we actually start getting close to running out of usable memory/disk/whatever or there's some performance problem for an application for instance. Could we alert on those instead?

[ja5087]

You can probably write a Prometheus query for that pattern, but I think it's fine since this isn't necessarily abnormal, just a large job.

[dkess]

These kinds of alerts are currently configured in Prometheus puppet: https://github.com/ocf/puppet/blob/master/modules/ocf_prometheus/files/rules.d/node.rules.yaml

You can alert on most things, including a memory threshold or a "sustained rate of change" type thing (the DiskWillFillIn3Hours alert tries to do this). There's lots of room to experiment here, though a simple threshold is probably fine. There are probably best practices for this around on the internet too.