Skip to content

Latest commit

 

History

History
232 lines (172 loc) · 9.07 KB

README.md

File metadata and controls

232 lines (172 loc) · 9.07 KB

Installing a custom Loki stack in Kyma

Overview

The following instructions outline how to use Loki as a logging backend with Kyma's LogPipeline or with Promtail.

CAUTION: This example uses the Grafana Loki version, which is distributed under AGPL-3.0 only and might not be free of charge for commercial usage.

Table of Content

Prerequisites

Preparation

  1. Export your Namespace as a variable. Replace the {NAMESPACE} placeholder in the following command and run it:

    export KYMA_NS="{NAMESPACE}"

  2. Export the Helm release names that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Run the following command:

    export HELM_LOKI_RELEASE="loki"

  3. Update your Helm installation with the required Helm repository:

    helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Loki installation

Depending on your scalability needs and storage requirements, you can install Loki in different Deployment modes. The following instructions install Loki in a lightweight in-cluster solution that does not fulfil production-grade qualities. Consider using a scalable setup based on an object storage backend instead (see Simple scalable deployment of Grafana Loki with Helm).

Install Loki

You install the Loki stack with a Helm upgrade command, which installs the chart if not present yet.

helm upgrade --install --create-namespace -n ${KYMA_NS} ${HELM_LOKI_RELEASE} grafana/loki -f https://raw.githubusercontent.com/kyma-project/examples/main/loki/loki-values.yaml

In any case, you can either create your own values.yaml file, or use the loki-values.yaml provided in this loki folder, which contains customized settings deviating from the default settings: The provided values.yaml file activates the singleBinary mode and disables additional components that are usually used when running Loki as a central backend.

Verify Loki installation

Check that the loki Pod has been created in the Namespace and is in the Running state:

kubectl -n ${KYMA_NS} get pod -l app.kubernetes.io/name=loki

Log agent installation

Install the log agent

To ingest the application logs from within your cluster to Loki, you can either choose an installation based on Promtail, which is the log collector recommended by Loki and provides a ready-to-use setup. Alternatively, you can use Kyma's LogPipeline feature based on Fluent Bit.

Install Promtail

To install Promtail pointing it to the previously installed Loki instance, run:

helm upgrade --install --create-namespace -n ${KYMA_NS} promtail grafana/promtail -f https://raw.githubusercontent.com/kyma-project/examples/main/loki/promtail-values.yaml --set "config.clients[0].url=https://${HELM_LOKI_RELEASE}.${KYMA_NS}.svc.cluster.local:3100/loki/api/v1/push"
Install Fluent Bit with Kyma's LogPipeline

CAUTION: This setup uses an unsupported output plugin for the LogPipline.

Apply the LogPipeline:

cat <<EOF | kubectl apply -f -
apiVersion: telemetry.kyma-project.io/v1alpha1
kind: LogPipeline
metadata:
  name: custom-loki
spec:
   input:
      application:
         namespaces:
           system: true
   output:
      custom: |
         name   loki
         host   ${HELM_LOKI_RELEASE}-headless.${KYMA_NS}.svc.cluster.local
         port   3100
         auto_kubernetes_labels off
         labels job=fluentbit, container=\$kubernetes['container_name'], namespace=\$kubernetes['namespace_name'], pod=\$kubernetes['pod_name'], node=\$kubernetes['host'], app=\$kubernetes['labels']['app'],app=\$kubernetes['labels']['app.kubernetes.io/name']
EOF

When the status of the applied LogPipeline resource turns into Running, the underlying Fluent Bit is reconfigured and log shipment to your Loki instance is active.

NOTE: The used output plugin configuration uses a static label map to assign labels of a Pod to Loki log streams. It's not recommended to activate the auto_kubernetes_labels feature for using all labels of a Pod because this lowers the performance. Follow Loki's labelling best practices for a tailor-made setup that fits your workload configuration.

Verify the setup by accessing logs using the Loki API

  1. To access the Loki API, use kubectl port forwarding. Run:

    kubectl -n ${KYMA_NS} port-forward svc/$(kubectl  get svc -n ${KYMA_NS} -l app.kubernetes.io/name=loki -ojsonpath='{.items[0].metadata.name}') 3100

  2. Loki queries need a query parameter time, provided in nanoseconds. To get the current nanoseconds in Linux or macOS, run:

    date +%s

  3. To get the latest logs from Loki, replace the {NANOSECONDS} placeholder with the result of the previous command, and run:

    curl -G -s  "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query={job="fluentbit"}' \
  --data-urlencode \
  'time={NANOSECONDS}'

Grafana installation

Because Grafana provides a very good Loki integration, you might want to install it as well.

Install Grafana

  1. To deploy Grafana, run:

    helm upgrade --install --create-namespace -n ${KYMA_NS} grafana grafana/grafana -f https://raw.githubusercontent.com/kyma-project/examples/main/loki/grafana-values.yaml

  2. To enable Loki as Grafana data source, run:

    cat <<EOF | kubectl apply -n ${KYMA_NS} -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-loki-datasource
  labels:
    grafana_datasource: "1"
data:
   loki-datasource.yaml: |-
      apiVersion: 1
      datasources:
      - name: Loki
        type: loki
        access: proxy
        url: "http://${HELM_LOKI_RELEASE}:3100"
        version: 1
        isDefault: false
        jsonData: {}
EOF

  3. Get the password to access the Grafana UI:

    kubectl get secret --namespace ${KYMA_NS} grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

  4. To access the Grafana UI with kubectl port forwarding, run:

kubectl -n ${KYMA_NS} port-forward svc/grafana 3000:80
  1. In your browser, open Grafana under http://localhost:3000 and log in with user admin and the password you retrieved before.

Grafana Exposure

Expose Grafana

  1. To expose Grafana using the Kyma API Gateway, create an APIRule: 
    kubectl -n ${KYMA_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/loki/apirule.yaml
  2. Get the public URL of your Loki instance: 
    kubectl -n ${KYMA_NS} get virtualservice -l apirule.gateway.kyma-project.io/v1beta1=grafana.${KYMA_NS} -ojsonpath='{.items[*].spec.hosts[*]}'

Add a link for Grafana to the Kyma Dashboard

  1. Download the dashboard-configmap.yaml file and change {GRAFANA_LINK} to the public URL of your Grafana instance.

       curl https://raw.githubusercontent.com/kyma-project/examples/main/loki/dashboard-configmap.yaml -o dashboard-configmap.yaml

  2. Optionally, adjust the ConfigMap: You can change the label field to change the name of the tab. If you want to move it to another category, change the category tab.

  3. Apply the ConfigMap, and go to Kyma Dashboard. Under the Observability section, you should see a link to the newly exposed Grafana. If you already have a busola-config, merge it with the existing one:

    kubectl apply -f dashboard-configmap.yaml

Cleanup

  1. To remove the installation from the cluster, run:

    helm delete -n ${KYMA_NS} ${HELM_LOKI_RELEASE}
helm delete -n ${KYMA_NS} promtail
helm delete -n ${KYMA_NS} grafana

  2. To remove the deployed LogPipeline instance from cluster, run:

    kubectl delete LogPipeline custom-loki