Where to see the logs of the different service ?

[AMontagu]

Hello,

I was previously working with loki that was reading stdout from docker socket. And in grafana I was able to search in the explore panel for log stored in loki.

When using the demo I see that OpenSearch is used. I am not familiar with it so maybe it doesn't do what I am looking for.
But I see in the docker compose that the port is not accesible so I was guessing that I need to use grafana to see them but when going in the explore menu and selecting opensearch as source I do not find any way to look for content as Loki did.

Also when trying a lucene query "*" it show me the error:
Query error
An error occurred within the plugin

with the log in grafana:

grafana  | logger=context traceID=afb802a8eee80946d5be75f8277ac0fe userId=0 orgId=1 uname= t=2024-10-29T17:48:07.598885475Z level=error msg=InternalError error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Unknown desc = version is required, err=error reading opensearch version" remote_addr=10.10.6.26 traceID=afb802a8eee80946d5be75f8277ac0fe
grafana  | logger=context traceID=afb802a8eee80946d5be75f8277ac0fe userId=0 orgId=1 uname= t=2024-10-29T17:48:07.59900605Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=10.10.6.26 time_ms=139 duration=139.461292ms size=148 referer="http://localhost:8080/grafana/explore?orgId=1&panes=%7B%228te%22%3A%7B%22datasource%22%3A%22P9744FCCEAAFBD98F%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22datasource%22%3A%7B%22type%22%3A%22grafana-opensearch-datasource%22%2C%22uid%22%3A%22P9744FCCEAAFBD98F%22%7D%2C%22query%22%3A%22%2A%22%2C%22queryType%22%3A%22lucene%22%2C%22alias%22%3A%22%22%2C%22metrics%22%3A%5B%7B%22type%22%3A%22count%22%2C%22id%22%3A%221%22%7D%5D%2C%22bucketAggs%22%3A%5B%7B%22type%22%3A%22date_histogram%22%2C%22id%22%3A%222%22%2C%22settings%22%3A%7B%22interval%22%3A%22auto%22%7D%2C%22field%22%3A%22observedTimestamp%22%7D%5D%2C%22format%22%3A%22table%22%2C%22timeField%22%3A%22observedTimestamp%22%2C%22luceneQueryType%22%3A%22Traces%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-1h%22%2C%22to%22%3A%22now%22%7D%7D%7D&schemaVersion=1" handler=/api/ds/query status_source=downstream

Also the opensearch is not documented in the architecture documentation: https://opentelemetry.io/docs/demo/architecture/

[julianocosta89]

hello @AMontagu 👋🏽

We do use opensearch, but not to view the logs.
The Collector sends the logs via OTLP to opensearch and we use Grafana to view those logs.
You can navigate to http://localhost:8080/grafana -> select demo dashboard, select a service that emits logs and you will see them below the Spanmetrics panel:

[julianocosta89]

Do you have any ideas on how to fix this without the clicking around?
That would be a great addition to the demo

[AMontagu]

I think that can be done by modifying: src/grafana/provisioning/datasources/opensearch.yaml and adding a version key into jsonData with the value od the current version of opensearch (in the demo it's 2.17.1):

apiVersion: 1

datasources:
  - name: OpenSearch
    type: grafana-opensearch-datasource
    url: http://opensearch:9200/
    access: proxy
    editable: true
    isDefault: false
    jsonData:
      database: otel
      flavor: opensearch
      version: "2.17.1"
      logLevelField: severity
      logMessageField: body
      pplEnabled: true
      timeField: observedTimestamp

But the version is a dynamic number coming from the .env OPENSEARCH_IMAGE_VERSION value and I don't think the demo support env variable in this yaml files.

[julianocosta89]

We were doing that before, but I recently changed that to use the following env var:

"GF_INSTALL_PLUGINS=grafana-opensearch-datasource ${OPENSEARCH_IMAGE_VERSION}"

According to their docs, that should work.
Here is the PR:
https://github.com/open-telemetry/opentelemetry-demo/pull/1743/files

If this is a problem, maybe we should revert the PR

[AMontagu]

I think it install the plugin with the correct version but did not set the version correctly to the datasource. At least for me.

[julianocosta89]

Thanks @AMontagu!

Reverting it here: #1760