Xandra crashes when a newly added node sends :up message

[harunzengin]

Recently, we added a new node to Xandra which caused some issues. The trigger was this:

** (stop) exited in: :gen_statem.call(#PID<0.152339.0>, :checkout, :infinity)
** (EXIT) an exception was raised:
    ** (FunctionClauseError) no function clause matching in anonymous fn/1 in Xandra.Cluster.Pool.handle_event/4
        (xandra 0.19.0) lib/xandra/cluster/pool.ex:245: anonymous fn(nil) in Xandra.Cluster.Pool.handle_event/4
        (elixir 1.16.3) lib/map.ex:957: Map.get_and_update/3
        (elixir 1.16.3) lib/map.ex:999: Map.get_and_update!/3
        (xandra 0.19.0) lib/xandra/cluster/pool.ex:245: Xandra.Cluster.Pool.handle_event/4
        (stdlib 5.2) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
        (stdlib 5.2) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
(stdlib 5.2) gen.erl:246: :gen.do_call/4
(stdlib 5.2) gen_statem.erl:923: :gen_statem.call/3
(xandra 0.19.0) lib/xandra/cluster.ex:548: Xandra.Cluster.with_conn_and_retrying/3

Looking closer, this is where we get the error: https://github.com/whatyouhide/xandra/blob/main/lib/xandra/cluster/pool.ex#L245

So since we get a HOST_UP event for a machine that we don't know about, Xandra crashes. Adding nodes to the cassandra cluster shouldn't cause crashes though.

[whatyouhide]

@harunzengin yep totally. Wanna work on fixing this? I won't have bandwidth for a while 😢

[harunzengin]

Looking closer at this, this looks more like a failure in Cassandra Gossip rather than Xandra. We added nodes before and didn't have any problems like this.

When a new host is added, we are supposed to get a NEW_NODE event in Xandra.Cluster.ControlConnection and consequently refresh the topology. But in this case, it seems like there was an error in Cassandra, so we didn't get that event, and somehow got a HOST_UP event instead, which made Xandra crash. Not sure whether we should actually be fixing this or let it crash? @whatyouhide

[whatyouhide]

@harunzengin the C* gossip protocol seems quite messy with this kind of events. It seems to happen really often that we don't get events we're supposed to get. I think a safer course of action is to refresh the topology if we get a HOST_UP for a host we've never seen before, effectively treating it like a NEW_NODE event?