You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I had searched in the issues and found no similar issues.
Description
Occasionally, we found that there have been cases of disk cache data escaping from the management of Doris file cache, causing disk space leaks. To make it easier for debugging, we need a checking tool that compares the contents in the Doris file cache memory management structure with the current disk contents to identify the differences between the two (which are potential problematic data).
Search before asking
Description
Occasionally, we found that there have been cases of disk cache data escaping from the management of Doris file cache, causing disk space leaks. To make it easier for debugging, we need a checking tool that compares the contents in the Doris file cache memory management structure with the current disk contents to identify the differences between the two (which are potential problematic data).
To better understand how file cache works, please refer to: https://doris.apache.org/zh-CN/docs/dev/compute-storage-decoupled/file-cache/ and https://www.bilibili.com/video/BV1ath9eGEqL
Basic Ideas
Coz the cache is changing rapidly, we should freeze the cache (via lock) to get a snapshot of current status.
Then parse the status to get which data should be cached.
And scan the disk (also during the freeze) to see which data indeed exists.
Finally compare the above two and print the diff in logs.
Implementation Tips
We could use Restful API to trigger the check. FYI, check
be/src/http/action/file_cache_action.cpp
for more details of Restful API support in Doris.If you get in any trouble ...
Do not hesitate to contact me by WeChat 15811301868
Related issues
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: