Loki - Old Data Not available after Upgrade from Simple Scalable to Distributed

[mdnouman-t]

Hi Community,

We tried to upgrade loki from Simple Scalable to Distributed. Post Upgrade we are unable to view the older logs.

Logs Storage - S3 Bucket.

Deployment is via helmchart.

• Char Version: loki-6.12.0
• App Version - 3.1.1

Simple Scalable Yaml:

loki:
  schemaConfig:
    configs:
      - from: 2024-07-15
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  useTestSchema: false
  auth_enabled: false
  analytics:
    reporting_enabled: false
  storage:
    type: "s3"
    s3:
      region: ap-south-1
    bucketNames:
      chunks: xxxx-chunks
      ruler: xxx-ruler
serviceAccount:
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::xxxxxxxx:role/xxx-chunks-ruler"
minio:
  enabled: false
chunksCache:
  enabled: false
resultsCache:
  enabled: false
test:
  enabled: false
monitoring:
  lokiCanary:
    enabled: false
queryScheduler:
  replicas: 1
gateway:
  service:
    type: NodePort
    nodePort: 32438
lokiCanary:
  enabled: false
test:
  enabled: false
write:
  replicas: 3
read:
  replicas: 3
backend:
  replicas: 3

Distributed Yaml:

loki:
   schemaConfig:
     configs:
       - from: "2024-07-15"
         store: tsdb
         object_store: s3
         schema: v13
         index:
           prefix: loki_index_
           period: 24h
   storage:
    type: "s3"
    s3:
      region: ap-south-1
    bucketNames:
      chunks: xxx-chunks
      ruler: xxx-ruler
   ingester:
     chunk_encoding: snappy
   querier:
     max_concurrent: 4
   pattern_ingester:
     enabled: true
   limits_config:
     allow_structured_metadata: true
     volume_enabled: true
     retention_period: 720h
   compactor:
     retention_enabled: false
     delete_request_store: s3

serviceAccount:
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::xxxxx:role/xxx-chunks-ruler"

deploymentMode: Distributed

ingester:
  replicas: 3
querier:
  replicas: 3 
  maxUnavailable: 1
queryFrontend:
  replicas: 3
  maxUnavailable: 1
queryScheduler:
  replicas: 3
distributor:
  replicas: 3
  maxUnavailable: 1
compactor:
  replicas: 1
indexGateway:
  replicas: 3
  maxUnavailable: 1

bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0

backend:
   replicas: 0
read:
   replicas: 0
write:
   replicas: 0

singleBinary:
   replicas: 0

gateway:
  service:
    type: NodePort
    nodePort: 32438

minio:
  enabled: false

We tried to validate the data in s3 bucket and found bucket size is increasing from past 2 weeks which we understand as data is being write to the bucket. The folders found in s3 buckets are 1/,index/,fake/, self-monitoring/.

In another instance of loki post deployment we are not seeing fake/ folder but the bucket size is increasing.

Please help us understand the folder structure and a way to see the old logs.

Thanks,
Nouman