Janusgraph.open() call fails when backend Cassandra cluster uses RF=3 and 2 nodes are down

[versaurabh]

• Janusgraph.open() call fails when backend Cassandra cluster uses RF=3 and 2 of the 3 nodes are down with below exception

org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:5135) at io.vavr.API$Match.of(API.java:5092) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:120) at io.vavr.control.Try.getOrElseThrow(Try.java:748) at org.janusgraph.diskstorage.cql.function.slice.CQLExecutorServiceSliceFunction.getSlice(CQLExecutorServiceSliceFunction.java:49) at org.janusgraph.diskstorage.cql.function.slice.AbstractCQLSliceFunction.getSlice(AbstractCQLSliceFunction.java:48) at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:358) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:99) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$1.call(KCVSConfiguration.java:96) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:106) at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:120) at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52) at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:117) at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration.get(KCVSConfiguration.java:96) at org.janusgraph.diskstorage.configuration.BasicConfiguration.isFrozen(BasicConfiguration.java:105) at org.janusgraph.diskstorage.configuration.builder.ReadConfigurationBuilder.buildGlobalConfiguration(ReadConfigurationBuilder.java:81) at org.janusgraph.graphdb.configuration.builder.GraphDatabaseConfigurationBuilder.build(GraphDatabaseConfigurationBuilder.java:67) at **org.janusgraph.core.JanusGraphFactory.open**(JanusGraphFactory.java:176) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:147) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:127)

Caused by: **com.datastax.oss.driver.api.core.AllNodesFailedException**: All 1 node(s) tried for the query failed (showing first 1 nodes, use getAllErrors() for more): Node(endPoint=atlas-cassandra/10.192.113.102:9042, hostId=356697f0-c36e-413e-8c12-175b7f2b9de4, hashCode=1f4a9e2a): [com.datastax.oss.driver.api.core.servererrors.UnavailableException: Not enough replicas available for query at consistency QUORUM (2 required but only 1 alive)]

• Can we open the JG graph only for reading in this cluster mode?

• Essentially, storage.cql.read-consistency-level=ONE is no longer usable, because the graph cannot be opened in the first place.

[hadoopmarc]

I believe JanusGraph maintains a list of all open graphs in its system tables, see https://docs.janusgraph.org/operations/recovery/#janusgraph-instance-failure Therefore, JanusGraph cannot open a graph if the write consistency level is not fulfilled.

[versaurabh]

I believe, we should add a "read-only" graph impl., that allows the graph to be read when consistency level is not fulfilled, something on the lines of (WRITE=ALL, READ=ONE), wdyt @hadoopmarc?

[hadoopmarc]

It feels to me that this would be very difficult to get right, unless:

• you first load the graph
• close all other janusgraph instances
• mark the graph (not the instance) as read-only. After that the graph is read-only for all instances until the mark is removed again.