Unable to query backend S3

[NewArk117]

Hi all,

My intention:
I am trying to use the grafana search function to get past traces from s3 bucket.

Issue:
I am unable to search traces in my grafana after 30mins(For testing) I'm not sure if i missed out anything in my configuration. For the first 30 mins I am able to get traces and i can see the blocks in my container at /storage/tempo/wal/blocks/ but after 30 mins or so it is removed and I was not able to search for traces on grafana however the block on my s3 is still there but I was not able to get the traces from the blocks in s3(storage backend).

Configuration:
This is the file for the tempo configmap file with the tempo configuration. I am running my grafana stack on an eks cluster node. From what i observed from my current configuration the compacted_block_retention: 3h retains the compacted blocks in s3. I am not sure what block_retention is pertaining to.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-config
  namespace: otel-cloud
  labels:
    app: tempo
data:
  tempo.yaml: |
    server:
      http_listen_port: 3200

    ingester:
      max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally

    compactor:
      compaction:
        block_retention: 5m # overall Tempo trace retention. set for demo purposes
        compacted_block_retention: 3h 

    query_frontend:
      search:
        max_duration: 0
        query_backend_after: 15m
        query_ingesters_until: 30m   
        
    metrics_generator:
      storage:
        path: /storage/generator/wal
        remote_write:
          - url: http://prometheus-svc.${namespace}.svc.cluster.local:9090/api/v1/write
            send_exemplars: true
    
    storage:
      trace:
        backend: s3
        s3:
          bucket: my-bucket 
          endpoint: s3.${region}.amazonaws.com
          access_key: XXXX
          secret_key: XXXX
        blocklist_poll: 5m
        wal:
          path: /storage/tempo/wal       

    overrides:
      metrics_generator_processors: [service-graphs, span-metrics] # enables metrics generator

[mapno]

Hi! The problem is block_retention set to 5min. block_retention configures how long to keep data in long-term storage for (S3 in your case). compacted_block_retention is a more advanced parameter to configure how long to keep compacted blocks for. These are blocks that have been merged into a larger one. I suggest you run the default value for compacted_block_retention and increase block_retention for however long you want to keep your data available.

You can read about Tempo config parameters in our docs.

[NewArk117]

@mapno, thank you for the reply. If that's the case what does the compacted_block_retention applying to?

Based on your comments does the block_retention refer to the object in the image below? (Image taken from my s3 storage) or is the image a compacted_block. Because I am trying to figure out where the trace inside grafana is taken from.

[mapno]

The block you're sharing is a block whose lifecycle is managed by block_retention.

Compaction is a process by which blocks are taken from object storage and merged into a bigger block. This is done to make search more efficient (+less overhead, -duplication). These blocks that are merged are marked as compacted in object storage. You can find them by searching for meta.compacted.json files. That indicates that the block has been merged into another one.

compacted_block_retention configures how long to keep compacted blocks before deleting them permanently. This is because the read path can take some time until it discovers the new blocks, so we need to give it time before pruning the old ones to be safe that queries are correct. I recommend you use the default value for this parameter.

Because I am trying to figure out where the trace inside grafana is taken from.

If the read path has discovered the new block, then it'll take it from there. If the read path still has an outdated view of object storage and only knows about compacted blocks, then it'll use those.

[NewArk117]

Okay I understand. Thanks for your help, much appreciated.