4.15 automation [OCS-5576]: Fail prometheus and verify that the alert still exists after recovering it.

[nagendra202]

Automated below test case:

RHSTOR-4798 Fail prometheus and verify that the alert still exists after recovering it.

1. Deploy cluster and run workloads.
2. Run IO pods to fill MDS memory to 95%.
3. Now, the alert is triggered to increase memory limit. Don't take any action on the alert.
4. Fail Prometheus, and recover it.
5. Make sure that the alert is still firing after recovering Prometheus.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nagendra202

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ocs-ci]

PR validation on existing cluster

Cluster Name: nagreddy-n11-2
Cluster Configuration:
PR Test Suite: tier2
PR Test Path: tests/functional/monitoring/prometheus/alerts/test_alert_mds_cache_high_usage.py
Additional Test Params:
OCP VERSION: 4.17
OCS VERSION: 4.17
tested against branch: master

Job PASSED.