reloader: config file stops being watched in case of atomic updates

[machine424]

Thanos, Prometheus and Golang version used:
reloader used by prometheus-operator

Object Storage Provider:

What happened:

When the CfgFile is updated atomically which results in the file being deleted and recreated (through temp file and rename or other) the watcher will stop receiving events for the as the original file was deleted.

fsnotify warns about that here https://github.com/fsnotify/fsnotify/blob/a9bc2e01792f868516acf80817f7d7d7b3315409/README.md?plain=1#L128

As a workaround the parent dir of can be added to WatchedDirs (assuming it doesn't contain noisy files)

Related prometheus-operator PR prometheus-operator/prometheus-operator#7366

What you expected to happen:

watcher should handle atomic updates of CfgFile

How to reproduce it (as minimally and precisely as possible):

run a reloader for a CfgFile, delete CfgFile, recreate CfgFile, change its content, reload will not be triggered immediately, but at the next WatchInterval tick.

Full logs to relevant components:

Anything else we need to know:

[dosubot]

Hey @machine424, I'm here to help you out with any bugs, questions, or contributions you have. Let's squash those bugs together! 🤖

The issue you're experiencing with the CfgFile not being watched after atomic updates is a known limitation of the fsnotify library. When a file is deleted and recreated, the watcher stops receiving events for that file. A common workaround is to watch the parent directory instead and filter events based on Event.Name. This approach helps ensure that updates are detected even when the file is replaced ^[1].

In the context of the thanos reloader, the ConfigWatcher uses the fsnotify package to handle file system events. It processes these events and reloads the configuration if any changes are detected, ensuring that the watcher continues to receive events even after the file is deleted and recreated ^[1][2].

The Prometheus Operator PR #7366 proposes explicitly watching the parent directory of the CfgFile to address this issue, provided that the directory does not contain other noisy files. This change aims to improve the reloader's responsiveness to configuration changes ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}