Highload

examples/highload/README.md

@@ -0,0 +1,11 @@
+# Highload NSM scenarios
+
+This folder contains links to highload NSM scenarios
+
+## Requires
+
+To run any highload scenario follow steps for [Basic NSM setup](../basic)
+
+## Includes
+
+ - [Continuous NSC and NSE scaling](./nsc-nse-scaling)

examples/highload/nsc-nse-scaling/README.md

@@ -0,0 +1,95 @@
+# Continuous NSC and NSE scaling
+
+This scenario checks memory, goroutine and `vpp` interface leaks. We continuously scale up and down NSCs and NSEs.
+Between scaling up and down we also check that all NSCs connections are alive. 
+
+## Requires
+
+Make sure that you have completed steps from [basic](../../basic) setup.
+
+## Run
+
+Deploy metrics server:
+```bash
+kubectl apply -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/4b42afefaf090a724f79767ed6b3f2d61643a726/examples/highload/nsc-nse-scaling/metrics-server.yaml
+```
+
+Wait for metrics server's readiness:
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l k8s-app=metrics-server -n kube-system
+```
+
+Collect pod metrics before scaling:
+```bash
+mkdir -p $ARTIFACTS_DIR/highload/nsc-nse-scaling
+kubectl top pod -A > $ARTIFACTS_DIR/highload/nsc-nse-scaling/metrics-before
+```
+
+Deploy NSCs and NSEs, 0 replicas each:
+```bash
+kubectl apply -k https://github.com/networkservicemesh/deployments-k8s/examples/highload/nsc-nse-scaling?ref=4b42afefaf090a724f79767ed6b3f2d61643a726
+```
+
+A function to check connectivity between NSCs and NSEs:
+```bash
+function ping() {
+    nscs=$(kubectl get pods -l app=nsc-kernel -o go-template --template="{{range .items}}{{.metadata.name}} {{end}}" -n ns-nsc-nse-scaling)
+    for nsc in $nscs; do
+        ipv4=$(kubectl exec $nsc -n ns-nsc-nse-scaling -- ip route | grep -Eo '172\.16\.0\.[0-9]{1,3}')
+        kubectl exec $nsc -n ns-nsc-nse-scaling -- ping -c2 -i0.5 $ipv4 || return 1
+    done
+    return 0
+}
+```
+
+Define the number of scaling iterations:
+```bash
+SCALING_COUNT=100
+NSC_COUNT=5
+NSE_COUNT=5
+```
+
+Main loop function:
+```bash
+function scaling() {
+    for i in $(seq 1 $SCALING_COUNT); do
+        kubectl scale deployment -n ns-nsc-nse-scaling nsc-kernel --replicas=$NSC_COUNT
+        kubectl scale deployment -n ns-nsc-nse-scaling nse-kernel --replicas=$NSE_COUNT
+        sleep 60
+
+        ping || return 1
+
+        kubectl scale deployment -n ns-nsc-nse-scaling nsc-kernel --replicas=0
+        kubectl scale deployment -n ns-nsc-nse-scaling nse-kernel --replicas=0
+        sleep 60
+    done
+    return 0
+}
+```
+
+
+Run the loop:
+```bash
+scaling
+```
+
+Collect metrics after the test:
+```bash
+kubectl top pod -A > $ARTIFACTS_DIR/highload/nsc-nse-scaling/metrics-after
+```
+
+Collect `vpp` interfaces from the forwarders:
+```bash
+fwds=$(kubectl get pods -l app=forwarder-vpp -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+for fwd in $fwds; do
+    kubectl exec -n nsm-system $fwd -- vppctl show int > $ARTIFACTS_DIR/highload/nsc-nse-scaling/$fwd-ifaces
+done
+```
+
+## Cleanup
+
+Delete ns:
+```bash
+kubectl delete ns ns-nsc-nse-scaling
+kubectl delete -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/4b42afefaf090a724f79767ed6b3f2d61643a726/examples/highload/nsc-nse-scaling/metrics-server.yaml
+```

examples/highload/nsc-nse-scaling/kustomization.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ns-nsc-nse-scaling
+
+resources:
+- ns-nsc-nse-scaling.yaml
+- netsvc.yaml
+- nsc.yaml
+- nse.yaml

examples/highload/nsc-nse-scaling/metrics-server.yaml

@@ -0,0 +1,203 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: metrics-server
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    k8s-app: metrics-server
+    rbac.authorization.k8s.io/aggregate-to-admin: "true"
+    rbac.authorization.k8s.io/aggregate-to-edit: "true"
+    rbac.authorization.k8s.io/aggregate-to-view: "true"
+  name: system:aggregated-metrics-reader
+rules:
+- apiGroups:
+  - metrics.k8s.io
+  resources:
+  - pods
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: system:metrics-server
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - nodes/metrics
+  verbs:
+  - get
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: metrics-server-auth-reader
+  namespace: kube-system
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: extension-apiserver-authentication-reader
+subjects:
+- kind: ServiceAccount
+  name: metrics-server
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: metrics-server:system:auth-delegator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:auth-delegator
+subjects:
+- kind: ServiceAccount
+  name: metrics-server
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: system:metrics-server
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: system:metrics-server
+subjects:
+- kind: ServiceAccount
+  name: metrics-server
+  namespace: kube-system
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: metrics-server
+  namespace: kube-system
+spec:
+  ports:
+  - name: https
+    port: 443
+    protocol: TCP
+    targetPort: https
+  selector:
+    k8s-app: metrics-server
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: metrics-server
+  namespace: kube-system
+spec:
+  selector:
+    matchLabels:
+      k8s-app: metrics-server
+  strategy:
+    rollingUpdate:
+      maxUnavailable: 0
+  template:
+    metadata:
+      labels:
+        k8s-app: metrics-server
+    spec:
+      containers:
+      - args:
+        - --cert-dir=/tmp
+        - --secure-port=10250
+        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
+        - --kubelet-use-node-status-port
+        - --metric-resolution=15s
+        - --kubelet-insecure-tls
+        image: registry.k8s.io/metrics-server/metrics-server:v0.7.1
+        imagePullPolicy: IfNotPresent
+        livenessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /livez
+            port: https
+            scheme: HTTPS
+          periodSeconds: 10
+        name: metrics-server
+        ports:
+        - containerPort: 10250
+          name: https
+          protocol: TCP
+        readinessProbe:
+          failureThreshold: 3
+          httpGet:
+            path: /readyz
+            port: https
+            scheme: HTTPS
+          initialDelaySeconds: 20
+          periodSeconds: 10
+        resources:
+          requests:
+            cpu: 100m
+            memory: 200Mi
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop:
+            - ALL
+          readOnlyRootFilesystem: true
+          runAsNonRoot: true
+          runAsUser: 1000
+          seccompProfile:
+            type: RuntimeDefault
+        volumeMounts:
+        - mountPath: /tmp
+          name: tmp-dir
+      nodeSelector:
+        kubernetes.io/os: linux
+      priorityClassName: system-cluster-critical
+      serviceAccountName: metrics-server
+      volumes:
+      - emptyDir: {}
+        name: tmp-dir
+---
+apiVersion: apiregistration.k8s.io/v1
+kind: APIService
+metadata:
+  labels:
+    k8s-app: metrics-server
+  name: v1beta1.metrics.k8s.io
+spec:
+  group: metrics.k8s.io
+  groupPriorityMinimum: 100
+  insecureSkipTLSVerify: true
+  service:
+    name: metrics-server
+    namespace: kube-system
+  version: v1beta1
+  versionPriority: 100

examples/highload/nsc-nse-scaling/netsvc.yaml

@@ -0,0 +1,7 @@
+---
+apiVersion: networkservicemesh.io/v1
+kind: NetworkService
+metadata:
+  name: nsc-nse-scaling
+spec:
+  payload: ETHERNET

examples/highload/nsc-nse-scaling/ns-nsc-nse-scaling.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: ns-nsc-nse-scaling

examples/highload/nsc-nse-scaling/nsc.yaml

@@ -0,0 +1,56 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nsc-kernel
+  labels:
+    app: nsc-kernel
+spec:
+  selector:
+    matchLabels:
+      app: nsc-kernel
+  replicas: 0
+  template:
+    metadata:
+      labels:
+        app: nsc-kernel
+        "spiffe.io/spiffe-id": "true"
+    spec:
+      containers:
+        - name: nsc
+          image: ghcr.io/networkservicemesh/cmd-nsc:v1.13.0
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: SPIFFE_ENDPOINT_SOCKET
+              value: unix:///run/spire/sockets/agent.sock
+            - name: NSM_LOG_LEVEL
+              value: INFO
+            - name: NSM_NETWORK_SERVICES
+              value: kernel://nsc-nse-scaling/nsm-1
+            - name: NSM_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+          volumeMounts:
+            - name: spire-agent-socket
+              mountPath: /run/spire/sockets
+              readOnly: true
+            - name: nsm-socket
+              mountPath: /var/lib/networkservicemesh
+              readOnly: true
+          resources:
+            requests:
+              cpu: 100m
+              memory: 40Mi
+            limits:
+              memory: 80Mi
+              cpu: 200m
+      volumes:
+        - name: spire-agent-socket
+          hostPath:
+            path: /run/spire/sockets
+            type: Directory
+        - name: nsm-socket
+          hostPath:
+            path: /var/lib/networkservicemesh
+            type: DirectoryOrCreate