Fluent-bit is unable to read logs and send it to data prepper

[jayeshjeh]

attaching docker compose for fluentbit, opensearch & opensearch dashboard

version: ‘3’
services:
fluent-bit:
container_name: fluent-bit
image: fluent/fluent-bit
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- ./test.log:/var/log/test.log
networks:
- dscnet
opensearch:
container_name: opensearch
image: opensearchproject/opensearch:latest
environment:
- discovery.type=single-node
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- “OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m” # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- OPENSEARCH_INITIAL_ADMIN_PASSWORD=Developer@123
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- dscnet
dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- “5601”
environment:
OPENSEARCH_HOSTS: ‘[“https://10.177.164.51:9200/”]’
depends_on:
- opensearch
networks:
- dscnet

networks:
dscnet:
external: true
driver: overlay
name: test-net

fluent-bit.conf

[INPUT]
name tail
refresh_interval 5
path ./test.log
read_from_head true

[OUTPUT]
Name http
Match *
Host data-prepper
Port 2021
URI /log/ingest
Format json

docker compose for data prepper

version: ‘3.7’
services:
data-prepper:
image: opensearchproject/data-prepper:2.0.0
container_name: data-prepper
volumes:
- ./log_pipeline.yaml:/usr/share/data-prepper/pipelines/log_pipeline.yaml
ports:
- 2021:2021
networks:
- test-net

networks:
test-net:
external: true

[if i use depends upon i get error]

log-pipeline:
source:
http:
ssl: false
processor:
- grok:
match:
log: [ “%{COMMONAPACHELOG}” ]
sink:
- opensearch:
hosts: [“https://10.177.164.51:9200/”]
insecure: true
username: admin
password: Developer@123
index: test_logs

test.log file has necessary permissions.

Fluent-bit
Opensearch
Opensearch-dashboards
dataprepper

all are running without any error but fluent-bit is not able to read the test.log file

[KarstenSchnitter]

Thanks for providing this issue. Can you share the error message from Fluent Bit, that you are receiving. In your configuration, I would have expected the path in the fluent-bit.conf to match the mount path of the log file:

[INPUT]
name tail
refresh_interval 5
path /var/log/test.log

Maybe Fluent BIt is just unable to find your log file.

[jayeshjeh]

Thanks for providing this issue. Can you share the error message from Fluent Bit, that you are receiving. In your configuration, I would have expected the path in the fluent-bit.conf to match the mount path of the log file:

[INPUT]
name tail
refresh_interval 5
path /var/log/test.log

Maybe Fluent BIt is just unable to find your log file.

Path is right, actually test.log is in same folder as docker files.

[dlvenable]

@jayeshjeh , Thank you for this issue. Including the logs from FluentBit can help us understand the error as it may have some additional information such as the response status.

[jayeshjeh]

@jayeshjeh , Thank you for this issue. Including the logs from FluentBit can help us understand the error as it may have some additional information such as the response status.

All i was getting was:

worker #0 started
worker #1 started

no errors were displayed, also the path is right with necessary permissions to the file.

[KarstenSchnitter]

Maybe it would be helpful to see the metrics reported by your DataPrepper instance. After the ingestion attempt, could you query the metrics endpoint as documented here and provide the result?

[KarstenSchnitter]

I am still not convinced, that the path is right. My understanding of your configuration is the following:

You mounted the file ./test.log on the Docker host to the file /var/log/test.log inside the Fluent Bit container. This is where Fluent Bit needs to be configured to find the file. But in your Fluent Bit config, you configure path: ./test.log. This would be correct, if Fluent Bit was running on the Docker host. But since it is running in its own container, it needs to read from the target of your bind-mount, which is /var/log/test.log.

[jayeshjeh]

I am still not convinced, that the path is right. My understanding of your configuration is the following:

You mounted the file ./test.log on the Docker host to the file /var/log/test.log inside the Fluent Bit container. This is where Fluent Bit needs to be configured to find the file. But in your Fluent Bit config, you configure path: ./test.log. This would be correct, if Fluent Bit was running on the Docker host. But since it is running in its own container, it needs to read from the target of your bind-mount, which is /var/log/test.log.

Hello Sir, alot thanks.. it was very very silly mistake.. i was checking path in fluent docker compose.
My POC is done. Thanks alot