Minor fixes for local execution #464

sundargates · 2023-06-18T22:16:19Z

Context

Explain context and other details for this pull request.

Checklist

./gradlew build compiles code correctly
Added new tests where applicable
./gradlew test passes all tests
Extended README or added javadocs where applicable

sundargates · 2023-06-18T22:16:42Z

mantis-server/mantis-server-agent/src/main/resources/agent.properties

-mantis.taskexecutor.blob-store.storage-dir=file:///localstore_dir
-mantis.taskexecutor.blob-store.local-cache=/apps/mantis/mantis-server-agent/mantis-artifacts
+mantis.taskexecutor.blob-store.storage-dir=file:///Users/[email protected]/Downloads/
+mantis.taskexecutor.blob-store.local-cache=/tmp/mantis/mantis-artifacts


Rewrite this.

sundargates · 2023-06-18T22:17:03Z

mantis-server/mantis-server-agent/build.gradle

 run {
-    standardInput = System.in
+//    standardInput = System.in
 }


Get rid of this block completely

github-actions · 2023-06-18T22:27:12Z

Test Results

126 files - 1 126 suites - 1 6m 46s ⏱️ +32s
536 tests - 2 526 ✔️ - 3 8 💤 ±0 2 ❌ +1
538 runs - 1 528 ✔️ - 2 8 💤 ±0 2 ❌ +1

For more details on these failures, see this check.

Results for commit 7e6ca0a. ± Comparison against base commit b78be8a.

This pull request removes 2 tests.

io.mantisrx.master.resourcecluster.resourceprovider.SimpleFileResourceStorageProviderTests ‑ testResourceClusterRules
io.mantisrx.master.resourcecluster.resourceprovider.SimpleFileResourceStorageProviderTests ‑ testUpdateClusterSpec

♻️ This comment has been updated with latest results.

sundargates · 2023-06-22T22:47:49Z

...ntrol-plane-client/src/main/java/io/mantisrx/server/master/client/MantisMasterClientApi.java

@@ -668,6 +668,7 @@ private HttpClient<ByteBuf, ServerSentEvent> getRxnettySseClient(String hostname

    private WebSocketClient<TextWebSocketFrame, TextWebSocketFrame> getRxnettyWebSocketClient(String host,
                                                                                              int port, String uri) {
+        logger.info("Creating websocket client for " + host + ":" + port + " uri " + uri + " ...");


Get rid of this change

sundargates · 2023-06-22T22:51:11Z

...-server/src/main/java/io/mantisrx/server/master/persistence/InMemoryPersistenceProvider.java

+import io.mantisrx.master.events.LifecycleEventPublisher;
+import io.mantisrx.server.master.store.KeyValueStore;
+
+public class InMemoryPersistenceProvider extends KeyValueBasedPersistenceProvider {


Move this to tests.

sundargates · 2023-06-22T22:51:29Z

...er/src/main/java/io/mantisrx/server/master/persistence/KeyValueBasedPersistenceProvider.java

+        return null;
+    }
+
+    @Override
+    public ResourceClusterSpecWritable getResourceClusterSpecWritable(ClusterID id)
+        throws IOException {
+        return null;
+    }
+
+    @Override
+    public ResourceClusterScaleRulesWritable getResourceClusterScaleRules(ClusterID clusterId)
+        throws IOException {
+        return null;
+    }
+
+    @Override
+    public ResourceClusterScaleRulesWritable registerResourceClusterScaleRule(
+        ResourceClusterScaleRulesWritable ruleSpec) throws IOException {
+        return null;
+    }
+
+    @Override
+    public ResourceClusterScaleRulesWritable registerResourceClusterScaleRule(
+        ResourceClusterScaleSpec rule) throws IOException {
+        return null;


These need to be filled

sundargates · 2023-06-22T22:56:42Z

...est/java/io/mantisrx/master/api/akka/route/v1/ResourceClusterNonLeaderRedirectRouteTest.java

@@ -89,15 +90,16 @@ public class ResourceClusterNonLeaderRedirectRouteTest extends JUnitRouteTest {
    private final ActorSystem system =
        ActorSystem.create(ResourceClusterNonLeaderRedirectRouteTest.class.getSimpleName());

+    private final IMantisPersistenceProvider storageProvider = new FileBasedPersistenceProvider(true);


Change this to InMemoryPersistenceProvider

sundargates · 2023-06-22T22:57:32Z

...src/test/java/io/mantisrx/master/resourcecluster/ResourceClusterAwareSchedulerActorTest.java

+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class ResourceClusterAwareSchedulerActorTest {


Delete this file

sundargates · 2023-06-22T22:59:07Z

.../src/test/java/io/mantisrx/master/resourcecluster/ResourceClustersHostManagerActorTests.java

        when(resProvider.getResponseHandler()).thenReturn(responseHandler);

        ActorRef resourceClusterActor = system.actorOf(
-                ResourceClustersHostManagerActor.props(resProvider, resStorageProvider));
+                ResourceClustersHostManagerActor.props(resProvider, new FileBasedPersistenceProvider(false)));


replace this with something else.

sundargates · 2023-06-26T18:23:43Z

...mantis-control-plane-server/src/main/java/io/mantisrx/server/master/store/KeyValueStore.java

+    class InMemoryStore implements KeyValueStore {
+
+        // table -> partitionKey -> secondaryKey -> data
+        private final Map<String, Map<String, Map<String, String>>> store = new ConcurrentHashMap<>();
+
+        @Override
+        public List<String> getAllPartitionKeys(String tableName) throws IOException {
+            try {
+                return store.get(tableName).keySet().stream().collect(Collectors.toList());
+            } catch (Exception e) {
+                throw new IOException(e);
+            }
+        }
+
+        @Override
+        public Map<String, String> getAll(String tableName, String partitionKey)
+            throws IOException {
+            if (store.get(tableName) == null) {
+                return Collections.emptyMap();
+            } else if (store.get(tableName).get(partitionKey) == null) {
+                return Collections.emptyMap();
+            } else {
+                return store.get(tableName).get(partitionKey);
+            }
+        }
+
+        @Override
+        public boolean upsertAll(String tableName, String partitionKey, Map<String, String> all,
+            Duration ttl) throws IOException {
+            store.putIfAbsent(tableName, new ConcurrentHashMap<>());
+            store.get(tableName).put(partitionKey, new ConcurrentHashMap<>(all));
+            return true;
+        }
+
+        @Override
+        public boolean delete(String tableName, String partitionKey, String secondaryKey)
+            throws IOException {
+            if (store.containsKey(tableName) && // table exists
+                store.get(tableName).containsKey(partitionKey) && // partitionKey exists
+                store.get(tableName).get(partitionKey).containsKey(secondaryKey)) { // secondaryKey exists
+                store.get(tableName).get(partitionKey).remove(secondaryKey);
+                return true;
+            }
+            return false;
+        }
+
+        @Override
+        public boolean deleteAll(String tableName, String partitionKey) throws IOException {
+            if (store.containsKey(tableName) && // table exists
+                store.get(tableName).containsKey(partitionKey)) { // partitionKey exists
+                store.get(tableName).remove(partitionKey);
+                return true;


@hmit Can you check if the semantics of this is consistent with the KeyValueStore assumptions

hmitnflx

looks good, thanks for this change!

sundargates requested review from nickmahilani, jeffchao, piygoyal, calvin681, Andyz26, hmitnflx, markcho, liuml07 and fdc-ntflx as code owners June 18, 2023 22:16

sundargates had a problem deploying to Integrate Pull Request June 18, 2023 22:16 — with GitHub Actions Failure

sundargates commented Jun 18, 2023

View reviewed changes

sundargates had a problem deploying to Integrate Pull Request June 22, 2023 22:47 — with GitHub Actions Failure

sundargates commented Jun 22, 2023

View reviewed changes

sundargates had a problem deploying to Integrate Pull Request June 22, 2023 22:50 — with GitHub Actions Failure

sundargates commented Jun 22, 2023

View reviewed changes

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 05:21 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 18:22 — with GitHub Actions Failure

sundargates commented Jun 26, 2023

View reviewed changes

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 18:25 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 19:09 — with GitHub Actions Failure

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 19:10 — with GitHub Actions Failure

sundargates added 3 commits June 26, 2023 12:11

Intermediate commit

bf7bf61

Cleaning up

c01d612

Intermediate commit

d43cb2b

sundargates added 7 commits June 26, 2023 12:22

[WIP] Intermediate commit

498db05

More changes

2d42e69

Cleaning up

b045fc0

More changes

5b9e34f

fixing the semantics for InMemoryStore::getAllPartitionKeys

a984a97

Fixing tests

1d7b3f5

Gradle changes

6a5b0e7

hmitnflx approved these changes Jun 26, 2023

View reviewed changes

Fixing the tests

b1610fe

sundargates force-pushed the sundaram/local branch from 1f769bf to b1610fe Compare June 26, 2023 19:28

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 19:28 — with GitHub Actions Failure

Spotless

7e6ca0a

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 20:36 — with GitHub Actions Failure

Updating properties

68bd3de

sundargates had a problem deploying to Integrate Pull Request June 26, 2023 20:51 — with GitHub Actions Failure

sundargates merged commit 91791b2 into Netflix:master Jun 26, 2023
1 of 2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minor fixes for local execution #464

Minor fixes for local execution #464

sundargates commented Jun 18, 2023

sundargates Jun 18, 2023

sundargates Jun 18, 2023

github-actions bot commented Jun 18, 2023 •

edited

Loading

sundargates Jun 22, 2023

sundargates Jun 22, 2023

sundargates Jun 22, 2023

sundargates Jun 22, 2023

sundargates Jun 22, 2023

sundargates Jun 22, 2023

sundargates Jun 26, 2023

hmitnflx left a comment

Minor fixes for local execution #464

Minor fixes for local execution #464

Conversation

sundargates commented Jun 18, 2023

Context

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 18, 2023 • edited Loading

Test Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmitnflx left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 18, 2023 •

edited

Loading