Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CNF-14862: Configure AlertManager to send alerts #306

Merged
merged 1 commit into from
Nov 20, 2024
Merged

CNF-14862: Configure AlertManager to send alerts #306

merged 1 commit into from
Nov 20, 2024
+196 −6

Conversation

mlguerrero12
Copy link
Contributor

@mlguerrero12 mlguerrero12 commented Nov 8, 2024 •
edited
Loading

Add a webhook receiver to AlertManager configuration so that alarm server is notified to the internal endpoint.

@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Nov 8, 2024 •
edited by openshift-ci bot
Loading

@mlguerrero12: This pull request references CNF-14862 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Add a webhook receiver to AlertManager configuration so that alarm server is notified to the internal endpoint.

Alerts are grouped by alertname and cluster_managed labels. Default timeout values are used. Alerts from the hub cluster are skipped, as well as, Watchdog alerts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mlguerrero12 mlguerrero12 marked this pull request as draft November 8, 2024 13:09
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 8, 2024
@openshift-ci-robot
Copy link
Collaborator

openshift-ci-robot commented Nov 14, 2024 •
edited by openshift-ci bot
Loading

@mlguerrero12: This pull request references CNF-14862 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Add a webhook receiver to AlertManager configuration so that alarm server is notified to the internal endpoint.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mlguerrero12 mlguerrero12 force-pushed the alertmanager branch 4 times, most recently from 5fb0c9b to 9a8e56c Compare November 18, 2024 11:33
@mlguerrero12 mlguerrero12 marked this pull request as ready for review November 18, 2024 11:33
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 18, 2024
@mlguerrero12 mlguerrero12 force-pushed the alertmanager branch 2 times, most recently from 5b528e2 to a8e193b Compare November 18, 2024 16:40
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 19, 2024
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 19, 2024
pixelsoccupied
pixelsoccupied reviewed Nov 19, 2024
View reviewed changes
Copy link
Collaborator

@pixelsoccupied pixelsoccupied left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

internal/service/alarms/internal/alertmanager/alertmanager.tmpl Outdated Show resolved Hide resolved
internal/service/alarms/internal/alertmanager/alertmanager.go Outdated Show resolved Hide resolved
internal/service/alarms/serve.go Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
internal/service/kustomize/alertmanager/README.md Outdated Show resolved Hide resolved
@pixelsoccupied
Copy link
Collaborator

/cc @browsell

@openshift-ci openshift-ci bot requested a review from browsell November 19, 2024 16:10
@mlguerrero12 mlguerrero12 force-pushed the alertmanager branch 4 times, most recently from 4c8e8b9 to ff194a3 Compare November 20, 2024 11:29
pixelsoccupied
pixelsoccupied reviewed Nov 20, 2024
View reviewed changes
Copy link
Collaborator

@pixelsoccupied pixelsoccupied left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are ready to merge! just a few tiny comments

internal/service/alarms/k8s/base/endpoints.yaml Outdated Show resolved Hide resolved
internal/service/alarms/internal/alertmanager/alertmanager.yaml
Comment on lines +12 to +15
- receiver: "null"
matchers:
# Always firing alert to verify alertmanager is working
- Watchdog
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think skipping Watchdog is fine but just want to make sure @browsell is aware of this since we initially sort of agreed to collect all the alerts regardless of it being a noise.

internal/service/alarms/internal/alertmanager/alertmanager.go Show resolved Hide resolved
@pixelsoccupied
Copy link
Collaborator

/approve

will make any follow up changes as needed to AM config but I think we can get started with building on top of this!

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 20, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pixelsoccupied

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2024
@mlguerrero12
Configure AlertManager to send alerts
8915a54 
Add a webhook receiver to AlertManager configuration
so that alarm server is notified to the internal endpoint.

Signed-off-by: Marcelo Guerrero <[email protected]>
@pixelsoccupied
Copy link
Collaborator

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 20, 2024
@openshift-merge-bot openshift-merge-bot bot merged commit e4e47de into openshift-kni:main Nov 20, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pixelsoccupied pixelsoccupied pixelsoccupied left review comments

@alegacy alegacy alegacy left review comments

@tliu2021 tliu2021 Awaiting requested review from tliu2021

@Missxiaoguo Missxiaoguo Awaiting requested review from Missxiaoguo

@browsell browsell Awaiting requested review from browsell

Assignees

@pixelsoccupied pixelsoccupied

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@mlguerrero12 @openshift-ci-robot @pixelsoccupied @alegacy @browsell @openshift-merge-robot