-
Notifications
You must be signed in to change notification settings - Fork 168
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
profiling: add tips for kubernetes self-managed deployment #3932
Changes from 1 commit
02c2ac9
ddaea3a
5e1f612
6d5c8a1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -12,6 +12,7 @@ This page outlines operating the backend when running Universal Profiling on a s | |
* <<profiling-self-managed-ops-monitoring, Monitoring your collector and symbolizer>> | ||
* <<profiling-scaling-backend-resources, Scaling your resources>> | ||
* <<profiling-self-managed-upgrade, Upgrading backend binaries>> | ||
* <<profiling-self-managed-kubernetes-tips, Kubernetes tips>> | ||
|
||
[discrete] | ||
[[profiling-self-managed-ops-sizing-guidance]] | ||
|
@@ -262,3 +263,85 @@ Download the corresponding binary version and replace the existing one, using th | |
Replace the old binary and restart the services. | ||
|
||
You will find the links to the new binaries in the "Add Data" page, under the "Binary" tab. | ||
|
||
[discrete] | ||
[[profiling-self-managed-kubernetes-tips]] | ||
== Kubernetes tips | ||
|
||
When deploying the Universal Profiling backend on Kubernetes, there are some best practices to follow. | ||
|
||
[discrete] | ||
=== Ingress configuration | ||
|
||
If you are using an ingress controller, the connection routing to the collector Service should be configured to use the gRPC protocol. | ||
|
||
We provide an `Ingress` resource as part of the Helm chart, and because the ingress can be any implementation, | ||
you should provide to the chart values the configuration of the ingress, adding the class name and the | ||
necessary annotations using the `ingress.annotations` field. | ||
inge4pres marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
For example, when using an NGINX ingress controller, | ||
set the annotation `nginx.ingress.kubernetes.io/backend-protocol: "GRPC"`, as shown in the following example: | ||
|
||
[source,yaml] | ||
---- | ||
ingress: | ||
create: true | ||
ingressClassName: "nginx" | ||
annotations: | ||
nginx.ingress.kubernetes.io/backend-protocol: "GRPC" | ||
---- | ||
|
||
For symbolizer, the connection routing should be configured to use the HTTP protocol. | ||
There is usually no need to customize annotations for this type of service, but the chart provides similar configuration options. | ||
|
||
[discrete] | ||
=== Output TLS configuration | ||
|
||
You can secure the communication between the Universal Profiling backend and the Elasticsearch cluster by enabling TLS | ||
in the `output.elasticsearch` section of the collector and symbolizer configuration files. | ||
|
||
To do so, Kubernetes secrets containing the TLS key pairs should be provisioned in the namespace where the backend is installed. | ||
In case of self-signed certificates, the CA bundle used to validate Elasticsearch's certificates should also be part of the secret. | ||
|
||
Create two secrets, one for the collector and one for the symbolizer, with the names `pf-symbolizer-tls-certificate` and `pf-collector-tls-certificate`. | ||
The secrets should contain the following keys: | ||
|
||
- `tls.key`: the certificate private key | ||
- `tls.cert`: the certificate public key | ||
- `ca.cert` (optional): the certificate CA bundle | ||
|
||
Follow these steps to enable TLS connection from collector/symbolizer to Elasticsearch: | ||
|
||
1. Create secrets with the TLS key pairs (omit the `ca.cert` field if you are not using a self-signed CA): | ||
|
||
kubectl create secret generic pf-collector-tls-certificate --from-file=tls.key=/path/to/key.pem \ | ||
--from-file=tls.cert=/path/to/cert.pem --from-file=ca.cert=/path/to/ca.crt | ||
|
||
kubectl create secret generic pf-symbolizer-tls-certificate --from-file=tls.key=/path/to/key.pem \ | ||
--from-file=tls.cert=/path/to/cert.pem --from-file=ca.cert=/path/to/ca.crt | ||
inge4pres marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
2. Update the collector and symbolizer Helm values files to enable the use of TLS configuration: | ||
+ | ||
[source, yaml, highlight=3..4] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unfortunately this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bummer... is there anything else we can use to make the diff more evident? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Our docs support code callouts. You could add one of those? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added a small text to clarify what needs to be done 👍🏼 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. perfect |
||
---- | ||
output: | ||
elasticsearch: | ||
ssl: | ||
enabled: true | ||
---- | ||
|
||
3. Upgrade the charts using the `helm upgrade` command, providing the updated values file. | ||
|
||
[discrete] | ||
=== Horizontal scaling | ||
|
||
When scaling the Universal Profiling backend on Kubernetes, you can increase the number of replicas for the collector, or | ||
enable Horizontal Pod Autoscaling V2. | ||
|
||
To enable HPAv2 for the collector or symbolizer, you can set the `autoscalingV2` dictionary in each Helm values file. | ||
|
||
At the moment, **it is not recommended to enable an autoscaler for symbolizer**. | ||
Due to a current limitation on how symbolizer replicas can synchronize their workloads, it is best | ||
to only use a single replica for the symbolizer. | ||
Scale the symbolizer vertically first. | ||
Only in case of high latency in symbolizing native frames (10+ minutes) you can evaluate adding more replicas. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what you're trying to say here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Ingress is managed by the chart, but its configurations need to be passed through from the users, as they are implementation-specific.
This practically means that all the times an ingress is enabled, it must be configured with some values passed in Helm by users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, that's very helpful. I made a small recommendation based off of this comment: https://github.com/elastic/observability-docs/pull/3932/files#r1621192289