Add multicluster heal examples (#9689)

* add multicluster heal examples

Signed-off-by: Nikita Skrynnik <[email protected]>

* add all tests

Signed-off-by: Nikita Skrynnik <[email protected]>

* add interdomain tests + add description to all tests + cleanup

Signed-off-by: Nikita Skrynnik <[email protected]>

* cleanup

Signed-off-by: Nikita Skrynnik <[email protected]>

* fix all path issues

Signed-off-by: Nikita Skrynnik <[email protected]>

* fix yaml linter

Signed-off-by: Nikita Skrynnik <[email protected]>

* delete multicluster configuration for heal examples and use configuration for basic multicluster scenarios

Signed-off-by: Nikita Skrynnik <[email protected]>

---------

Signed-off-by: Nikita Skrynnik <[email protected]>

examples/multicluster_heal/README.md

@@ -0,0 +1,27 @@
+# Healing floating interdomain examples
+
+This document contains links for multicluster and interdomain heal examples of NSM.
+
+## Requires
+
+To run any heal example follow steps for [Multicluster NSM setup](../multicluster)
+
+## Requires
+
+- [Load balancer](../multicluster/loadbalancer)
+- [Interdomain DNS](../multicluster/dns)
+- Interdomain spire
+    - [Spire on first cluster](../spire/cluster1)
+    - [Spire on second cluster](../spire/cluster2)
+    - [Spire on third cluster](../spire/cluster3)
+    - [Spiffe Federation](../multicluster/spiffe_federation)
+
+## Includes
+
+- [Forwarders death in floating interdomain scenario](./floating-forwarder-death)
+- [NSE death in floating interdomain scenario](./floating-nse-death)
+- [NSM systems death in floating interdomain scenario](./floating-nsm-system-death)
+- [Proxy nsmgrs death in interdomain scenario](./interdomain-proxy-nsmgr-death)
+- [NSMGRs death in interdomain scenario](./interdomain-nsmgr-death)
+- [Registry death in interdomain scenario](./interdomain-registry-death)
+

examples/multicluster_heal/floating-forwarder-death/README.md

@@ -0,0 +1,114 @@
+# Forwarders death in floating interdomain scenario
+
+This example shows that NSM keeps working after forwarders on the first and the second clusters are deleted.
+
+NSC and NSE use the `kernel` mechanism to connect to their local forwarders.
+Forwarders from the first and the second cluster use the `vxlan` mechanism to connect to each other.
+
+NSE registers itself in the floating registry.
+
+## Requires
+
+Make sure that you have completed steps from [multicluster_heal](../)
+
+## Run
+
+**1. Deploy network service on cluster3**
+
+Deploy NS:
+```bash
+kubectl --kubeconfig=$KUBECONFIG3 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-forwarder-death/cluster3?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+**2. Deploy endpoint on cluster2**
+
+Deploy NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-forwarder-death/cluster2?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+Wait for applications ready:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ns-floating-forwarder-death
+```
+
+**3. Deploy client on cluster1**
+
+Deploy NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-forwarder-death/cluster1?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+Wait for applications ready:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 wait --for=condition=ready --timeout=5m pod -l app=alpine -n ns-floating-forwarder-death
+```
+
+**4. Check connectivity**
+
+Ping from NSC to NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 exec pods/alpine -n ns-floating-forwarder-death -- ping -c 4 172.16.1.2
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 exec deployments/nse-kernel -n ns-floating-forwarder-death -- ping -c 4 172.16.1.3
+```
+
+**5. Find forwarders on the first and the second clusters**
+
+```bash
+LOCALFWD=$(kubectl --kubeconfig=$KUBECONFIG1 get pods -l app=forwarder-vpp -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+REMOTEFWD=$(kubectl --kubeconfig=$KUBECONFIG2 get pods -l app=forwarder-vpp -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+**6. Delete forwarders from both clusters**
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 delete pod ${LOCALFWD} -n nsm-system
+```
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 delete pod ${REMOTEFWD} -n nsm-system
+```
+
+**7. Wait until newly created forwarders are ready**
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 wait --for=condition=ready --timeout=1m pod -l app=forwarder-vpp -n nsm-system
+```
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 wait --for=condition=ready --timeout=1m pod -l app=forwarder-vpp -n nsm-system
+```
+
+**8. Check connectivity with newly created forwarders**
+
+Ping from NSC to NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 exec pods/alpine -n ns-floating-forwarder-death -- ping -c 4 172.16.1.2
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 exec deployments/nse-kernel -n ns-floating-forwarder-death -- ping -c 4 172.16.1.3
+```
+
+
+## Cleanup
+
+Cleanup resources for *cluster1*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 delete ns ns-floating-forwarder-death
+```
+
+Cleanup resources for *cluster2*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 delete ns ns-floating-forwarder-death
+```
+
+3. Cleanup resources for *cluster3*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG3 delete ns ns-floating-forwarder-death
+```

examples/multicluster_heal/floating-forwarder-death/cluster1/client.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: alpine
+  labels:
+    app: alpine
+  annotations:
+    networkservicemesh.io: kernel://[email protected]/nsm-1
+spec:
+  containers:
+  - name: alpine
+    image: alpine:3.15.0
+    imagePullPolicy: IfNotPresent
+    # simple `sleep` command would work
+    # but we need `trap` to be able to delete pods quckly
+    command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]

examples/multicluster_heal/floating-forwarder-death/cluster1/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-floating-forwarder-death
+
+resources:
+- ns-floating-forwarder-death.yaml
+- client.yaml

examples/multicluster_heal/floating-forwarder-death/cluster1/ns-floating-forwarder-death.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ns-floating-forwarder-death

examples/multicluster_heal/floating-forwarder-death/cluster2/kustomization.yaml

@@ -0,0 +1,12 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-floating-forwarder-death
+
+resources:
+- ns-floating-forwarder-death.yaml
+- ../../../../apps/nse-kernel
+
+patchesStrategicMerge:
+- patch-nse.yaml

examples/multicluster_heal/floating-forwarder-death/cluster2/ns-floating-forwarder-death.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ns-floating-forwarder-death

examples/multicluster_heal/floating-forwarder-death/cluster2/patch-nse.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nse-kernel
+spec:
+  template:
+    spec:
+      containers:
+        - name: nse
+          env:
+          - name: POD_NAME
+            valueFrom:
+              fieldRef:
+                fieldPath: metadata.name
+          - name: NSM_NAME
+            value: "$(POD_NAME)@my.cluster3"
+          - name: NSM_CIDR_PREFIX
+            value: 172.16.1.2/31
+          - name: NSM_SERVICE_NAMES
+            value: "[email protected]"
+          - name: NSM_REGISTER_SERVICE
+            value: "false"

examples/multicluster_heal/floating-forwarder-death/cluster3/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-floating-forwarder-death
+
+resources:
+- ns-floating-forwarder-death.yaml
+- netsvc.yaml

examples/multicluster_heal/floating-forwarder-death/cluster3/netsvc.yaml

@@ -0,0 +1,7 @@
+---
+apiVersion: networkservicemesh.io/v1
+kind: NetworkService
+metadata:
+  name: floating-forwarder-death
+spec:
+  payload: ETHERNET

examples/multicluster_heal/floating-forwarder-death/cluster3/ns-floating-forwarder-death.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ns-floating-forwarder-death

examples/multicluster_heal/floating-nse-death/README.md

@@ -0,0 +1,105 @@
+# NSE death in floating interdomain scenario
+
+This example shows that NSM keeps working after NSE on the second cluster is deleted.
+
+NSC and NSE use the `kernel` mechanism to connect to their local forwarders.
+Forwarders from the first and the second cluster use the `vxlan` mechanism to connect to each other.
+
+NSE registers itself in the floating registry.
+
+
+## Requires
+
+Make sure that you have completed steps from [multicluster_heal](../)
+
+## Run
+
+**1. Deploy network service on cluster3**
+
+Deploy NS:
+```bash
+kubectl --kubeconfig=$KUBECONFIG3 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-nse-death/cluster3?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+**2. Deploy endpoint on cluster2**
+
+Deploy NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-nse-death/cluster2?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+Wait for applications ready:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ns-floating-nse-death
+```
+
+**3. Deploy client on cluster1**
+
+Deploy NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 apply -k https://github.com/networkservicemesh/deployments-k8s/examples/multicluster_heal/floating-nse-death/cluster1?ref=b7a0736c9257da4c7e0880b8338f254f94097d4c
+```
+
+Wait for applications ready:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 wait --for=condition=ready --timeout=5m pod -l app=alpine -n ns-floating-nse-death
+```
+
+**4. Check connectivity**
+
+Ping from NSC to NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 exec pods/alpine -n ns-floating-nse-death -- ping -c 4 172.16.1.2
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 exec deployments/nse-kernel -n ns-floating-nse-death -- ping -c 4 172.16.1.3
+```
+
+**5. Find NSE on the second cluster**
+
+```bash
+NSE=$(kubectl --kubeconfig=$KUBECONFIG2 get pods -l app=nse-kernel -n ns-floating-nse-death --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+**6. Delete NSE**
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 delete pod ${NSE} -n ns-floating-nse-death
+```
+
+**7.Wait until a new NSE is ready**
+
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ns-floating-nse-death
+```
+
+**8. Check connectivity with the new NSE**
+
+Ping from NSC to NSE:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 exec pods/alpine -n ns-floating-nse-death -- ping -c 4 172.16.1.2
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 exec deployments/nse-kernel -n ns-floating-nse-death -- ping -c 4 172.16.1.3
+```
+
+## Cleanup
+
+Cleanup resources for *cluster1*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG1 delete ns ns-floating-nse-death
+```
+
+Cleanup resources for *cluster2*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG2 delete ns ns-floating-nse-death
+```
+
+3. Cleanup resources for *cluster3*:
+```bash
+kubectl --kubeconfig=$KUBECONFIG3 delete ns ns-floating-nse-death
+```

examples/multicluster_heal/floating-nse-death/cluster1/client.yaml

@@ -0,0 +1,17 @@
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: alpine
+  labels:
+    app: alpine
+  annotations:
+    networkservicemesh.io: kernel://[email protected]/nsm-1
+spec:
+  containers:
+  - name: alpine
+    image: alpine:3.15.0
+    imagePullPolicy: IfNotPresent
+    # simple `sleep` command would work
+    # but we need `trap` to be able to delete pods quckly
+    command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"]

examples/multicluster_heal/floating-nse-death/cluster1/kustomization.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-floating-nse-death
+
+resources:
+- ns-floating-nse-death.yaml
+- client.yaml

examples/multicluster_heal/floating-nse-death/cluster1/ns-floating-nse-death.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ns-floating-nse-death

examples/multicluster_heal/floating-nse-death/cluster2/kustomization.yaml

@@ -0,0 +1,12 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-floating-nse-death
+
+resources:
+- ns-floating-nse-death.yaml
+- ../../../../apps/nse-kernel
+
+patchesStrategicMerge:
+- patch-nse.yaml