Metrics investigation for CSI driver and dashboard creation on Grafana #274

prajwalvathreya · 2024-10-01T17:20:04Z

General: This PR investigates the metrics provided by the side cars of the csi driver and uses prometheus to scrape them. It also includes implementation of make targets to automate the creation of a grafana dashboard for a user to inspect the metrics.

Have you removed all sensitive information, including but not limited to access keys and passwords?
Have you checked to ensure there aren't other open or closed Pull Requests for the same bug/feature/question?

Pull Request Guidelines:

Does your submission pass tests?
Have you added tests?
Are you addressing a single feature in this PR?
Are your commits atomic, addressing one change per commit?
Are you following the conventions of the language?
Have you saved your large formatting changes for a different PR, so we can focus on your work?
Have you explained your rationale for why this feature is needed?
Have you linked your PR to an [open issue](https://blog.github.com/2013-05-14-closing-issues-via-pull-requests/

added legend to explain the graphs

…directory

- Added shell script to install prometheus and grafana

- added custom dashboard JSON file

- moved files into observability directory - updated readme and documentation in observability.md

# Conflicts: # Makefile

codecov · 2024-10-01T17:26:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.32%. Comparing base (0297748) to head (62e806d).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #274   +/-   ##
=======================================
  Coverage   76.32%   76.32%           
=======================================
  Files          21       21           
  Lines        1622     1622           
=======================================
  Hits         1238     1238           
  Misses        286      286           
  Partials       98       98

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hack/install-monitoring-tools.sh

Co-authored-by: Rahul Sharma <[email protected]>

Makefile

Co-authored-by: amold1 <[email protected]>

docs/dashboard.json

docs/observability.md

hack/install-monitoring-tools.sh

Co-authored-by: Rahul Sharma <[email protected]>

hack/install-monitoring-tools.sh

…period, port - updated observability.md to find the PID incase a user doesn't have access to script output - moved dashboard.json to observability dir

eljohnson92 · 2024-10-02T02:58:56Z

deploy/kubernetes/base/ss-csi-linode-controller.yaml

@@ -44,6 +44,11 @@ spec:
            - "--volume-name-uuid-length=16"
            - "--csi-address=$(ADDRESS)"
            - "--v=2"
+            - "--metrics-address=0.0.0.0:10248"


Could we potentially move all of the metrics related yaml updates to the helm chart on an opt in basis instead of here by default?

Agree with Evan. If we can do this, that would be a better approach.

- added loadbalancer service to avoid port-forwarding

…and not deploy

…e_metrics`

…e metrics

eljohnson92 · 2024-10-07T20:37:39Z

internal/driver/deploy/releases/linode-blockstorage-csi-driver.yaml

@@ -365,7 +365,7 @@ spec:
            secretKeyRef:
              key: token
              name: linode
-        image: linode/linode-blockstorage-csi-driver:latest
+        image: linode/linode-blockstorage-csi-driver:metrics-endpoint


I think we want to rever this to latest

eljohnson92 · 2024-10-07T20:39:20Z

observability/metrics/csi-linode-controller-metrics-service.yaml

+apiVersion: v1
+kind: Service
+metadata:
+  name: csi-linode-controller-metrics-10248


can we rename these services to represent what they are going to(attacher/provisoiner/resizer)

eljohnson92 · 2024-10-07T20:40:06Z

observability/metrics/csi-linode-controller-metrics-service.yaml

@@ -0,0 +1,62 @@
+apiVersion: v1


can this be moved into the helm chart templates and conditionally created if the related metrics endpoints are enabled?

docs/observability.md

observability/metrics/loadBalancer.yaml

- added new line - removed unused files

komer3

LGTM!

prajwalvathreya added 13 commits September 23, 2024 12:15

Added ports for the sidecars to allow prometheus to scrape the metrics

8387728

Fixed error in linode-csi-plugin container due to incorrect metrics port

0c39d75

Added documentation and example graphs for metrics in the csi-driver.

68b9b98

updated graphs and documentation.

361c077

added legend to explain the graphs

added line break after future scope to keep the doc consistent

0a4609e

added additional node metrics

4df77e3

added clarification on the unit of measurement of time

06d04b3

fixed typo

fcf133b

Moved metrics-documentation.md and example-images folder to the docs …

aedc721

…directory

Merge branch 'refs/heads/main' into metrics-endpoint

1db1a39

- Created make target for creating a grafana-dashboard

28691b8

- Added shell script to install prometheus and grafana

- created services to expose metrics to prometheus

7f133cc

- added custom dashboard JSON file

- updated install script to run process in the background

cb43eff

- moved files into observability directory - updated readme and documentation in observability.md

prajwalvathreya requested review from a team as code owners October 1, 2024 17:20

prajwalvathreya added 2 commits October 1, 2024 13:22

Merge branch 'main' into metrics-endpoint

4b09fbf

# Conflicts: # Makefile

- fixed conflict in Makefile

c10ee59

prajwalvathreya changed the title ~~Metrics investation for csi driver and dashboard creation on grafana~~ Metrics investigation for CSI driver and dashboard creation on Grafana Oct 1, 2024

rahulait reviewed Oct 1, 2024

View reviewed changes

hack/install-monitoring-tools.sh Outdated Show resolved Hide resolved

Update hack/install-monitoring-tools.sh

960c574

Co-authored-by: Rahul Sharma <[email protected]>

amold1 reviewed Oct 1, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Updated the syntax of passing the CLUSTER_NAME variable

016edfa

Co-authored-by: amold1 <[email protected]>

komer3 reviewed Oct 1, 2024

View reviewed changes

rahulait reviewed Oct 1, 2024

View reviewed changes

hack/install-monitoring-tools.sh Outdated Show resolved Hide resolved

rahulait reviewed Oct 1, 2024

View reviewed changes

hack/install-monitoring-tools.sh Outdated Show resolved Hide resolved

rahulait reviewed Oct 1, 2024

View reviewed changes

hack/install-monitoring-tools.sh Outdated Show resolved Hide resolved

prajwalvathreya and others added 3 commits October 1, 2024 14:55

Update hack/install-monitoring-tools.sh, namespace creation

efeea6e

Co-authored-by: Rahul Sharma <[email protected]>

Update hack/install-monitoring-tools.sh Grafana helm chart update

9370d03

Co-authored-by: Rahul Sharma <[email protected]>

Update hack/install-monitoring-tools.sh Prometheus helm chart update

5d92fbb

Co-authored-by: Rahul Sharma <[email protected]>

rahulait reviewed Oct 1, 2024

View reviewed changes

hack/install-monitoring-tools.sh Outdated Show resolved Hide resolved

prajwalvathreya added 2 commits October 1, 2024 15:18

- added environment variables for username, password, data retention …

c0a1c56

…period, port - updated observability.md to find the PID incase a user doesn't have access to script output - moved dashboard.json to observability dir

- removed echo used for debugging

069b55d

eljohnson92 reviewed Oct 2, 2024

View reviewed changes

prajwalvathreya and others added 9 commits October 2, 2024 10:51

Merge branch 'main' into metrics-endpoint

802245f

- updated the script to 3 make targets

9bc3beb

- added loadbalancer service to avoid port-forwarding

- updated templates to opt in to install using helm

c120fa7

- fixed container port mapping which was causing containers to crash …

3276b46

…and not deploy

- resolving Makefile conflict

9e18c05

Merge branch 'main' into metrics-endpoint

ceac735

- updated to helm chart to expose drivers based on passed flag `enabl…

2977e89

…e_metrics`

- updated documentation to explain how to use the helm chart to enabl…

9380f5d

…e metrics

Merge branch 'main' into metrics-endpoint

6fe3e49

eljohnson92 reviewed Oct 7, 2024

View reviewed changes

prajwalvathreya added 2 commits October 7, 2024 21:28

- made changes to install metrics services through helm chart

9d97d6e

- reverted csi-driver image to latest

79750ca

komer3 reviewed Oct 8, 2024

View reviewed changes

docs/observability.md Show resolved Hide resolved

docs/observability.md Show resolved Hide resolved

docs/observability.md Show resolved Hide resolved

docs/observability.md Show resolved Hide resolved

observability/metrics/loadBalancer.yaml Outdated Show resolved Hide resolved

prajwalvathreya added 3 commits October 8, 2024 11:59

- updated documentation to explain modifications to make targets

07c63df

- added new line - removed unused files

- updated comment to a more sensible one

6ac52d6

- updated documentation to be less verbose

62e806d

komer3 approved these changes Oct 8, 2024

View reviewed changes

eljohnson92 approved these changes Oct 9, 2024

View reviewed changes

prajwalvathreya merged commit 1748a1b into main Oct 9, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metrics investigation for CSI driver and dashboard creation on Grafana #274

Metrics investigation for CSI driver and dashboard creation on Grafana #274

prajwalvathreya commented Oct 1, 2024 •

edited

Loading

codecov bot commented Oct 1, 2024 •

edited

Loading

eljohnson92 Oct 2, 2024

komer3 Oct 2, 2024

eljohnson92 Oct 7, 2024

eljohnson92 Oct 7, 2024

eljohnson92 Oct 7, 2024

komer3 left a comment

Metrics investigation for CSI driver and dashboard creation on Grafana #274

Metrics investigation for CSI driver and dashboard creation on Grafana #274

Conversation

prajwalvathreya commented Oct 1, 2024 • edited Loading

General: This PR investigates the metrics provided by the side cars of the csi driver and uses prometheus to scrape them. It also includes implementation of make targets to automate the creation of a grafana dashboard for a user to inspect the metrics.

Pull Request Guidelines:

codecov bot commented Oct 1, 2024 • edited Loading

Codecov Report

eljohnson92 Oct 2, 2024

Choose a reason for hiding this comment

komer3 Oct 2, 2024

Choose a reason for hiding this comment

eljohnson92 Oct 7, 2024

Choose a reason for hiding this comment

eljohnson92 Oct 7, 2024

Choose a reason for hiding this comment

eljohnson92 Oct 7, 2024

Choose a reason for hiding this comment

komer3 left a comment

Choose a reason for hiding this comment

prajwalvathreya commented Oct 1, 2024 •

edited

Loading

codecov bot commented Oct 1, 2024 •

edited

Loading