Sisyphus Autoclean with Symlinks

[Atticus1806]

When importing jobs from other work directories as symlink sisyphus tries to autoclean them causing a lot of warnings, spamming the console. Is there a workarround or fix besides turning off autoclean?

[critias]

I guess you are talking about this warning here, right?
https://github.com/rwth-i6/sisyphus/blob/master/sisyphus/job.py#L518
Looking at the code I guess you keep getting the same warnings for the same directories over and over again. At least I don't see any check to stop that. That would be an undesired behavior and needs to be fixed anyway, Sisyphus should stop trying to clean it up after the first try.

Or is your problem that you keep getting warnings for different directories? The only way I see right now is to disable the warning there in the code until I find a nicer solution.

[Atticus1806]

Looking at the code I guess you keep getting the same warnings for the same directories over and over again.

Yes exactly. For now I turned off autocleaning because it was spamming the console every few minutes.

[critias]

This branch here should fix your issue: https://github.com/rwth-i6/sisyphus/tree/cleanup_once
Could you give it a try and let me know if it works for you?

[Atticus1806]

Hey, first of all sorry for the later reply.
So I switched to the branch (and also pulled the master which was quite behind in my sis version) and there is some other issue right now (visual I think):
It loads the config, then the usual appears
Print verbose overview (v), update aliases and outputs (u), start manager (y), or exit (n)? [2021-09-24 09:29:20,640] INFO: clean up: [SOME PATH] but as you can see there instantly also comes the first cleanup massage in the same line. I can just type y or the other commands but its already few lines behind because cleaning instantly starts. Besides that from what I can tell it is also not working, it still tries to clean up Jobs multiple times (In the output I found at least one job which is tried twice)

[critias]

Good point, maybe the starting the cleaner should be delayed until the manger is started.
I'll take a look at this and why this didn't solve the double cleaning problem next week.

[critias]

Sorry for the late reply. I added a bugfix to the cleanup_once branch and delayed the startup of the cleaner till the manager loop is started. Please try it again.

[Atticus1806]

No worries, I was on vacation and only came back today so the timing was fitting :)

From what I see (like 10 minute test run) it seems to be working. Thanks again for your help!

[Atticus1806]

Not sure why this is still open, but this is working now.

[Marvin84]

Not sure why this is still open, but this is working now.

@Atticus1806 Is this actually under master? I am still getting these warnings

[critias]

These changes are under master. They don't suppress all warning, but make sure that Sisyphus only tries to clean a directory once instead trying it each time the cleaner is run. So you should only get each warning once, instead of getting repeatedly all the time.

[Marvin84]

Thanks for the answer. I am getting this warning every time I run without exception. Not sure if something is misconfigured. I will check this with Benedikt.

…

On Sat, Oct 14, 2023, 09:55 Jan-Thorsten Peter ***@***.***> wrote: These changes are under master. They don't suppress all warning, but make sure that Sisyphus only tries to clean a directory once instead trying it each time the cleaner is run. So you should only get each warning once, instead of getting repeatedly all the time. — Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEQ6G5NO76WKA22VX7PC3MTX7JANVANCNFSM5EOBBH5Q> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[critias]

You get each warning only once per start up! So if you restart Sisyphus it's expected to see the warnings again. Before that fix the same warning kept showing up over and over again every time Sisyphus tried to cleanup the directory.

I think you could just add a check if the directory which should be cleaned actually belongs the user running Sisyphus inside
_sis_cleanable. I guess that would remove these warnings for you and would probably be a good idea anyway.

[albertz]

Isn't this also fixed now via PR #154 (fixing issue #105)?