Delete corrupt file on the Isilon

[hackartisan]

figgy-web-staging1 still / again cannot get a listing or interact with the directory `/mnt/hydra_sources/

replicate by trying to ls that location.

It definitely worked for a minute after the box was rebuilt. But now it does not.

previous ticket: #6593

[hackartisan]

It looks like a bunch of clean:expired_local_files jobs are starting and never finishing, then eventually you can't get a file listing at that location.

[tpendragon]

We killed the tasks with sudo kill $(ps aux | grep 'expired_local_files' | awk '{print $2}') -9

We do need that task to run, though.

[tpendragon]

What we've found:

We can't read or delete that file in SMB mounts.

If we SSH to the Isilon, we can't read the file still.

The file in question (on the isilon): /ifs/hydra/binaries/ingest_scratch/local_uploads_broken/1435651a747a13f36d59cb498ce70b04

On the Figgy boxes this is /mnt/hydra_sources/ingest_scratch/local_uploads_broken/1435651a747a13f36d59cb498ce70b04

From what we've been able to tell the other files are fine, but this one is holding up the cleanup process.

Edit: The paths were local_uploads, but we were able to mv local_uploads local_uploads_broken; mkdir local_uploads to prevent our cleanup task from never completing and freezing Figgy.

[aruiz1789]

@tpendragon we have an ongoing conversation with Dell support team as we are having an issue replacing a drive in the storage device. We will provide more information once we are done with Dell support as we believe this issue is not related to the figgy problem but needs to be discarded.

[aruiz1789]

@tpendragon can you try deleting the file once again? All cifs shares look normal and I am able to list the file in question.

pulsys@figgy-web-staging1:~$ ls -al /mnt/hydra_sources/ingest_scratch/local_uploads_broken/1435651a747a13f36d59cb498ce70b04
-rwxr-xr-x 1 deploy www-data 6269599 Dec 18 21:39 /mnt/hydra_sources/ingest_scratch/local_uploads_broken/1435651a747a13f36d59cb498ce70b04

[tpendragon]

@aruiz1789 Still broken: cat /mnt/hydra_sources/ingest_scratch/local_uploads_broken/1435651a747a13f36d59cb498ce70b04 just hangs.

It's also broken ON the isilon, when SSH'd into diglibdata1. I don't think it has anything to do with smb, or if it does it's some sort of file lock or something.

[aruiz1789]

Hey @tpendragon I unmounted and remounted the share but still can't see the content of the file.... but I can list inside the dir path which I was not able to before.

I also mounted the share on a "Rocky Linux 9.5" VM and got the same results, the system hangs while reading the file. So we can eliminate the client OS version being the issue.

When you tried deleting the file from the Storage Device did you get a busy error or it just hangs like from the client? If you get a busy output we can try rebooting the client to release the connection to the file.

I will also get in touch with the storage admin to see if we can pinpoint the cause as I do not have access to it.