attic watch-store crashes after a while: "sending into a closed channel"

[antifuchs]

I just recently started self-hosting attic on my machine, and overall it works really well! I'm also running a buildbot-nix setup on there, with an attic-watch-store systemd unit that the buildbot-nix devs recommend. Unfortunately, that service regularly crashes (2x in the past 2 days, I think):

Feb 17 03:47:31 gloria attic-watch-store-start[1859873]: thread 'main' panicked at client/src/command/watch_store.rs:116:51:
Feb 17 03:47:31 gloria attic-watch-store-start[1859873]: called `Result::unwrap()` on an `Err` value: sending into a closed channel
Feb 17 03:47:31 gloria attic-watch-store-start[1859873]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'll activate RUST_BACKTRACE to get a more detailed trace, but my suspicion is that there's something on the other side that's dying silently.

[antifuchs]

As expected, the full backtrace sadly doesn't reveal much:

Feb 18 03:46:38 gloria attic-watch-store-start[492544]: thread 'main' panicked at client/src/command/watch_store.rs:116:51:
Feb 18 03:46:38 gloria attic-watch-store-start[492544]: called `Result::unwrap()` on an `Err` value: sending into a closed channel
Feb 18 03:46:38 gloria attic-watch-store-start[492544]: Stack backtrace:
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    1: tokio::runtime::park::CachedParkThread::block_on
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    2: tokio::runtime::context::runtime::enter_runtime
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    3: attic::main
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    4: std::sys::backtrace::__rust_begin_short_backtrace
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    5: std::rt::lang_start::{{closure}}
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    6: std::rt::lang_start_internal
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    7: main
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    8: __libc_start_call_main
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    9: __libc_start_main_alias_2
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:   10: _start
Feb 18 03:46:38 gloria attic-watch-store-start[492544]: stack backtrace:
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    0: rust_begin_unwind
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    1: core::panicking::panic_fmt
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    2: core::result::unwrap_failed
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    3: tokio::runtime::park::CachedParkThread::block_on
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    4: tokio::runtime::context::runtime::enter_runtime
Feb 18 03:46:38 gloria attic-watch-store-start[492544]:    5: attic::main
Feb 18 03:46:38 gloria attic-watch-store-start[492544]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[antifuchs]

I think I have a suspicion, which is that long-lived push sessions might not be a good idea. I've attempted at updating the way this is handled in https://github.com/zhaofengli/attic/compare/main...antifuchs:attic:watch-renew-sessions?expand=1, which may or may not work: Give it a day or two and I'll file a PR if it prevents crashes (:

(Besides preventing that crash, it might also result in less memory being used over time as the pusher keeps a hash of all store paths it ever pushed and the results from those pushes...)