Terminators disappearing when round-robin restarting controllers

[plorenz]

Notes from @emoscardini

I'm trying & somewhat replicating an issue when ZET is hosting a service & I do a rolling restart of the controllers. The terminator for the for the service get's deleted somehow & it's not restored until I restart the hosting ZET. It seems to only happen if I do a rolling restart that follows how ZET is connecting to the controllers. Meaning, I see the fabric circuits being created on one of the three, so I restart that one, see the circuits switch to another & restart that one. Once I've restarted all three of the NCs, the terminator seems to disappear.

it's a simple http service with a curl going though every second. ZET is only hosting, I'm using ERT as the intercept.

I tested again this morning & I think it's not the sequence as much as the amount of time between each NC restart. I seem to be able to consistently replicate it, if I restart all NCs withing a few seconds, regardless of order. If I do a slower rolling restart, give ZET a chance to fully switch over to the new NC, it seems to work Ok, even if I follow the circuit pattern.

I say that...& this time I replicated the issue again when slowing switching. The ER don't show any logs at all once this happens & controller shows circuit request with NO_TERMINATOR

Notes from @plorenz

I was able to reproduce as well. The ZET is closing the channel to the ER, which is causing the terminator to be removed. I don't know why this is happening. The ER is not going down, only the controllers.

I tried to reproduce this with ziti tunnel and was unable to. This seems to be ZET specific, so I'm moving this to the ZET project.