diff --git a/README.md b/README.md index dfa9168..b395c37 100644 --- a/README.md +++ b/README.md @@ -1,416 +1,77 @@ # SDRAN-in-a-Box (RiaB) -SDRAN-in-a-Box (RiaB) is a SD-RAN cluster which is able to operate within a single host machine. +SDRAN-in-a-Box (RiaB) is a SD-RAN cluster which is able to operate within a single host machine . It provides a development/test environment for developers/users in ONF SD-RAN community. -RiaB deploys SD-RAN infrastructure - the EPC (OMEC), emulated RAN (CU/DU/UE), and RIC (ONOS-RIC) - over Kubernetes. -On top of the SD-RAN infrastructure, we can conduct E2E tests in terms of the user plane and the SD-RAN control plane. - -## Supported machines -* CloudLab Wisconsin and Utah cluster - * CPU: Intel CPU and Haswell microarchitecture or beyond; at least 4 cores - * OS: Ubuntu 18.04 (e.g., OnePC-Ubuntu18.04 profile in CloudLab) - * RAM: At least 16GB - * Storage: At least 50GB (recommendation: 100GB) -* Any baremetal server or VM - * CPU: Intel CPU and Haswell microarchitecture or beyond; at least 4 cores - * OS: Ubuntu 18.04 or 20.04 (e.g., OnePC-Ubuntu18.04 profile in CloudLab) - * RAM: At least 16GB - * Storage: At least 50GB (recommendation: 100GB) - -## Quick Start -### Clone this repository -To begin with, clone this repository: -```bash -$ git clone https://github.com/onosproject/sdran-in-a-box -``` - -NOTE: If we want to use a specific release, we can change the branch with `git checkout [args]` command. -```bash -$ cd /path/to/sdran-in-a-box -$ git checkout v1.0.0 # for release 1 -$ git checkout master # for the latest -``` - -### Deploy RiaB - CU-CP/OAI version (option 1) -Go to the `sdran-in-a-box` directory and build/deploy CU-CP, OAI DU, OAI UE, OMEC, and RIC. We can choose one of three commands: -```bash -$ cd /path/to/sdran-in-a-box -# Use latest charts and images -$ make #or make riab-oai, make riab-oai-latest -# Use charts and images for SD-RAN release 1.0 -$ make riab-oai-v1.0.0 -# Use charts in the local directory (~/helm-charts/) and images defined in sdran-in-a-box-values.yaml file -# Useful for the SD-RAN development -$ make riab-oai-dev -``` - -### Deploy RiaB - RANSim version (option 2) -Go to the `sdran-in-a-box` directory and build/deploy RIC and RANSim. We can choose one of three commands: -```bash -$ cd /path/to/sdran-in-a-box -# Use latest charts and images -$ make riab-ransim # or make riab-ransim-latest -# Use charts and images for SD-RAN release 1.0 -$ make riab-ransim-v1.0.0 -# Use charts in the local directory (~/helm-charts/) and images defined in sdran-in-a-box-values.yaml file -# Useful for the SD-RAN development -$ make riab-ransim-dev -``` - -### Write credentials when deploying RiaB -Running the Makefile script, we have to write some credentials for (i) opencord gerrit, (ii) onosproject github, and (iii) sdran private Helm chart repository. -```bash -aether-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires HTTPS key -Cloning into '/users/wkim/helm-charts/aether-helm-charts'... -Username for 'https://gerrit.opencord.org': -Password for 'https://@gerrit.opencord.org': -remote: Total 1103 (delta 0), reused 1103 (delta 0) -Receiving objects: 100% (1103/1103), 526.14 KiB | 5.31 MiB/s, done. -Resolving deltas: 100% (604/604), done. -sdran-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires Github credential -Cloning into '/users/wkim/helm-charts/sdran-helm-charts'... -Username for 'https://github.com': -Password for 'https://@github.com': -remote: Enumerating objects: 19, done. -remote: Counting objects: 100% (19/19), done. -remote: Compressing objects: 100% (17/17), done. -remote: Total 2259 (delta 7), reused 3 (delta 2), pack-reused 2240 -Receiving objects: 100% (2259/2259), 559.35 KiB | 2.60 MiB/s, done. -Resolving deltas: 100% (1558/1558), done. - -..... - -helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ -"incubator" has been added to your repositories -helm repo add cord https://charts.opencord.org -"cord" has been added to your repositories -Username for ONF SDRAN private chart: -Password for ONF SDRAN private chart: -"sdran" has been added to your repositories -touch /tmp/build/milestones/helm-ready -``` - -If we did not see any error, everything is deployed. - -### Deployed K8s pods for CU-CP/OAI version (option 1) -```bash -# It shows the result when we deploy CU-CP/OAI version (option 1) -$ kubectl get po -n riab -NAME READY STATUS RESTARTS AGE -cassandra-0 1/1 Running 0 3h26m -hss-0 1/1 Running 0 3h26m -mme-0 4/4 Running 0 3h26m -oai-enb-cu-0 1/1 Running 0 3h25m -oai-enb-du-0 1/1 Running 0 3h25m -oai-ue-0 1/1 Running 0 3h25m -onos-config-66744d748-xr95k 1/1 Running 0 3h25m -onos-consensus-db-1-0 1/1 Running 0 3h25m -onos-e2sub-5f5fd6fdfb-5k92l 1/1 Running 0 3h25m -onos-e2t-74c765bc79-9h4sm 1/1 Running 0 3h25m -onos-kpimon-5649d85d57-7jvcw 1/1 Running 0 3h25m -onos-sdran-cli-678d547cbd-hbrtn 1/1 Running 0 3h25m -onos-topo-5bfbffb577-vvdcw 1/1 Running 0 3h25m -pcrf-0 1/1 Running 0 3h26m -spgwc-0 2/2 Running 0 3h26m -upf-0 4/4 Running 0 3h26m -``` - -### Deployed K8s pods for RANSim version (option 2) -```bash -# It shows the result when we deploy RANSim version (option 2) -$ kubectl get po -n riab -NAME READY STATUS RESTARTS AGE -onos-config-76f5b95d8f-zjfpj 1/1 Running 0 92s -onos-consensus-db-1-0 1/1 Running 0 92s -onos-e2sub-5dff4f9b9-cljn7 1/1 Running 0 92s -onos-e2t-67cbfb8c55-gpph4 1/1 Running 0 92s -onos-kpimon-5649d85d57-28kt4 1/1 Running 0 92s -onos-sdran-cli-7f4fc59b47-5sppt 1/1 Running 0 92s -onos-topo-85647b8cc6-98nd5 1/1 Running 0 92s -ran-simulator-57956df985-hc5n4 1/1 Running 0 92s -``` - -### Verification - -#### The user plane (for option 1 - CU-CP/OAI version) -We should check if the user plane is working by using `make test-user-plane` command: -```bash -$ make test-user-plane -*** T1: Internal network test: ping 192.168.250.1 (Internal router IP) *** -PING 192.168.250.1 (192.168.250.1) from 172.250.255.253 oaitun_ue1: 56(84) bytes of data. -64 bytes from 192.168.250.1: icmp_seq=1 ttl=64 time=38.5 ms -64 bytes from 192.168.250.1: icmp_seq=2 ttl=64 time=47.0 ms -64 bytes from 192.168.250.1: icmp_seq=3 ttl=64 time=33.4 ms - ---- 192.168.250.1 ping statistics --- -3 packets transmitted, 3 received, 0% packet loss, time 2002ms -rtt min/avg/max/mdev = 33.456/39.694/47.038/5.599 ms -*** T2: Internet connectivity test: ping to 8.8.8.8 *** -PING 8.8.8.8 (8.8.8.8) from 172.250.255.253 oaitun_ue1: 56(84) bytes of data. -64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=53.6 ms -64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=52.0 ms -64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=33.9 ms - ---- 8.8.8.8 ping statistics --- -3 packets transmitted, 3 received, 0% packet loss, time 2002ms -rtt min/avg/max/mdev = 33.988/46.556/53.665/8.912 ms -*** T3: DNS test: ping to google.com *** -PING google.com (172.217.12.78) from 172.250.255.253 oaitun_ue1: 56(84) bytes of data. -64 bytes from dfw28s05-in-f14.1e100.net (172.217.12.78): icmp_seq=1 ttl=113 time=51.8 ms -64 bytes from dfw28s05-in-f14.1e100.net (172.217.12.78): icmp_seq=2 ttl=113 time=51.3 ms -64 bytes from dfw28s05-in-f14.1e100.net (172.217.12.78): icmp_seq=3 ttl=113 time=50.4 ms - ---- google.com ping statistics --- -3 packets transmitted, 3 received, 0% packet loss, time 2001ms -rtt min/avg/max/mdev = 50.472/51.219/51.872/0.632 ms -``` - -If we can see all above Kubernetes pods running and ping is running, the user plane is working well. - -#### RIC by using ONOS-KPIMON xAPP (for both options) -Also, we should check whether the ONOS-RIC micro-services are working by using ONOS-KPIMON xAPP. -For this test, we can use `make test-kpimon` command: -```bash -$ make test-kpimon -*** Get KPIMON result through CLI *** -Key[PLMNID, nodeID] num(Active UEs) -{eNB-CU-Eurecom-LTEBox [0 2 16] 57344} 1 -``` - -If we can see that `num(Active UEs)` is `1`, RIC is working well. - -### Completely delete/reset RiaB -This deletes not only deployed Helm chart but also Kubernetes and Helm. -```bash -make clean # if we want to keep the ~/helm-charts directory - option to develop/test changed/new Helm charts -make clean-all # if we also want to delete ~/helm-charts directory -``` - -## Useful commands -### Deploy specific charts -For the development, we should deploy/reset some specific charts; this Makefile script also supports deploying/deleting some specific charts. - -#### Deploy OMEC -```bash -$ make omec -``` - -#### Deploy OAI and CU-CP -```bash -$ make oai -``` -NOTE: When deploying OAI and CU-CP chart, Makefile script automatically deploy OMEC chart (if it is not deployed). - -#### Deploy ONOS-RIC micro-services -```bash -$ make ric -``` -NOTE: When deploying ONOS-RIC micro-services chart, Makefile script automatically deploy Atomix chart (if it is not deployed). - -#### Deploy Atomix controllers -```bash -$ make atomix -``` - -#### Deploy all undeployed Helm charts -We can reset/delete some specific charts by using some commands described in the below subsection (`Reset/delete specific charts`). To deploy all the reset/deleted charts, we can use the below command. -```bash -$ make riab-oai-[version] # for Option 1 -$ make riab-ransim-[version] # for Option 2 -``` - -### Reset/delete specific charts -In order to remove some specific charts already deployed, this Makefile script provides some commands for the development. - -#### Reset/delete OMEC -```bash -$ make reset-omec -``` -NOTE: When deleting OMEC chart, Makefile script automatically deletes OAI and CU-CP chart. - -#### Reset/delete OAI and CU-CP -```bash -$ make reset-oai -``` - -#### Reset/delete ONOS-RIC micro-services -```bash -$ make reset-ric -``` - -#### Reset/delete Atomix controller -```bash -$ make reset-atomix -``` - -#### Delete/Reset charts for RiaB -This deletes all deployed Helm charts for SD-RAN development/test (i.e., Atomix, RIC, OAI, and OMEC). It does not delete K8s, Helm, or other software. -```bash -$ make reset-test -``` - -### Manage UEs (for Option 1) -#### Deploy multiple UEs. -Currently not working yet; under development. - -#### Manually detach a UE -Currently, RiaB can emulates a single UE running on `oai-ue` pod. In order to manually detach this UE, we can use the below command: -```bash -$ echo -en "AT+CPIN=0000\r" | nc -u -w 1 localhost 10000 -$ echo -en "AT+CGATT=0\r" | nc -u -w 1 localhost 10000 -``` - -NOTE: Since reattachment is not working, we have to redeploy all charts again by using `make reset-oai reset-ric && make riab-oai`. This will be fixed in the next release. - -#### Manually reattach a UE -Currently not working yet; Under development. This will support in the near future (next release). - -### Miscellaneous -#### Fetch Aether and SD-RAN Helm charts -For the development perspective, we need to fetch the latest Helm chart commits, although the RiaB uses a specific chart version. This command fetches all latest commits: -```bash -$ make fetch-all-charts -``` -It just fetches the all latest commits, i.e., it does not change/checkout the specific branch/commit. - -NOTE: It may request credentials for the OpenCORD gerrit and SD-RAN Github. - -#### Use SD-RAN CLI -In order to use the SD-RAN CLI, we should access to the onos-sdran-cli pod first. Then, we can use SD-RAN CLI commands. -```bash -$ kubectl exec -it deployment/onos-sdran-cli -n riab -- bash -$ sdran [arg1] [arg2] ... -``` - -## Troubleshooting -This section covers how to solve the reported issues. This section will be updated, continuously. - -### SPGW-C or UPF is not working -Please check the log with below commands: -```bash -$ kubectl logs spgwc-0 -n riab -c spgwc # for SPGW-C log -$ kubectl logs upf-0 -n riab -c bess # for UPF log -``` - -In the log, if we can see `unsupported CPU type` or `a specific flag (e.g., AES) is missing`, we should check the CPU microarchitecture. RiaB requires Intel Haswell or more recent CPU microarchitecture. -If we have the appropriate CPU type, we should build SPGW-C or UPF image on the machine where RiaB will run. - -To build SPGW-C, first clone the SPGW-C repository on the machine with `git clone https://github.com/omec-project/spgw`. Then, edit below line in Makefile: -```makefile -DOCKER_BUILD_ARGS ?= --build-arg RTE_MACHINE='native' -``` -Then, run `make` on the `spgw` directory. - -Likewise, for building UPF image, we should clone UPF repository with `git clone https://github.com/omec-project/upf-epc`. Then, edit below line in Makefile: -```makefile -CPU ?= native -``` -Then, run `make` on the `upf-epc` directory. - -After building those images, we should modify overriding value yaml file (i.e., `sdran-in-a-box-values.yaml`). Go to the file and write down below: -```yaml -images: - tags: - spgwc: - bess: - pfcpiface: - pullPolicy: IfNotPresent -``` -Then, run below commands: -```bash -$ cd /path/to/sdran-in-a-box -$ make reset-test -# after all OMEC pods are deleted, run make again -$ make -``` - -### ETCD is not working -Sometimes, we see the below outputs when building RiaB. -```text -TASK [etcd : Configure | Ensure etcd is running] *********************************************************************** -FAILED - RETRYING: Configure | Check if etcd cluster is healthy (4 retries left). -FAILED - RETRYING: Configure | Check if etcd cluster is healthy (3 retries left). -FAILED - RETRYING: Configure | Check if etcd cluster is healthy (2 retries left). -FAILED - RETRYING: Configure | Check if etcd cluster is healthy (1 retries left). -``` - -If we see this, we can command below: -```bash -$ sudo systemctl restart docker -$ cd /path/to/sdran-in-a-box -$ make -``` - -### Atomix controllers cannot be deleted/reset -Sometimes, Atomix controllers cannot be deleted (maybe we will get stuck when deleting Atomix controller pods) when we command `make reset-test`. -```bash -rm -f /tmp/build/milestones/oai-enb-cu -rm -f /tmp/build/milestones/oai-enb-du -rm -f /tmp/build/milestones/oai-ue -helm delete -n riab sd-ran || true -release "sd-ran" uninstalled -cd /tmp/build/milestones; rm -f ric -kubectl delete -f https://raw.githubusercontent.com/atomix/kubernetes-controller/master/deploy/atomix-controller.yaml || true -customresourcedefinition.apiextensions.k8s.io "databases.cloud.atomix.io" deleted -customresourcedefinition.apiextensions.k8s.io "partitions.cloud.atomix.io" deleted -customresourcedefinition.apiextensions.k8s.io "members.cloud.atomix.io" deleted -customresourcedefinition.apiextensions.k8s.io "primitives.cloud.atomix.io" deleted -serviceaccount "atomix-controller" deleted -clusterrole.rbac.authorization.k8s.io "atomix-controller" deleted -clusterrolebinding.rbac.authorization.k8s.io "atomix-controller" deleted -service "atomix-controller" deleted -deployment.apps "atomix-controller" deleted -``` - -If the script is stopped here, we can command: -```bash -# Commmand Ctrl+c first to stop the Makefile script if the make reset-test is got stuck. Then command below. -$ make reset-atomix # Manually delete Atomix controller pods -$ make atomix # Manually install Atomix controller pods -$ make reset-test # Then, make reset-test again -``` - -Or, sometimes we see this when deploying RiaB: -```text -Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/atomix/kubernetes-controller/master/deploy/atomix-controller.yaml": object is being deleted: customresourcedefinitions.apiextensions.k8s.io "members.cloud.atomix.io" already exists -Makefile:231: recipe for target '/tmp/build/milestones/atomix' failed -``` - -In this case, we can manually delete atomix with the command `make atomix || make reset-atomix`, and then resume to deploy RiaB. - -### Pod onos-consensus-db-1-0 initialization failed - -In Ubuntu 20.04 (kernel 5.4.0-65-generic), the k8s pod named `onos-consensus-db-1-0` might fail due to a bug of using go and alpine together (e.g., https://github.com/docker-library/golang/issues/320). - -It can be seen in `kubectl logs -n riab onos-consensus-db-1-0` as: -```bash -runtime: mlock of signal stack failed: 12 -runtime: increase the mlock limit (ulimit -l) or -runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ -fatal error: mlock failed -``` - -Such pod utilizes the docker image atomix/raft-storage-node:v0.5.3, tagged from the build of the image atomix/dragonboat-raft-storage-node:latest available at https://github.com/atomix/dragonboat-raft-storage-node. - -A quick fix (allowing an unlimited amount memory to be locked by the pod) to this issue is cloning the repository https://github.com/atomix/dragonboat-raft-storage-node, and changing the Makefile: - -```bash -# Before change -image: build - docker build . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} - -# After change: unlimited maximum locked-in-memory address space -image: build - docker build --ulimit memlock=-1 . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} -``` - -Then running in the source dir of this repository the command `make image`, and tagging the built image as: - -```bash -docker tag atomix/dragonboat-raft-storage-node:latest atomix/raft-storage-node:v0.5.3 -``` - -After that proceed with the execution of the Riab setup again. - -### Other issues? -Please contact ONF SD-RAN team, if you see any issue. Any issue report from users is very welcome. -Mostly, the redeployment by using `make reset-test and make [option]` resolves issues. \ No newline at end of file +RiaB deploys SD-RAN infrastructure - the EPC ([OMEC](https://github.com/omec-project)), emulated RAN (CU/DU/UE), and ONOS RAN Intelligent Controller (ONOS RIC) services - over Kubernetes. +On top of the SD-RAN infrastructure, we can conduct end-to-end tests in terms of the user plane and the SD-RAN control plane. + +## Features +* Installs Kubernetes and Helm that are the required infrastructure for SD-RAN services +* Provides one of three choices to emulate/simulate Radio Access Networks (RANs) + * RAN-Simulator + * Simulates multiple E2 nodes (CU-CP/DU/RU) and UEs + * Generates SD-RAN control plane messages + * Does not support the LTE traffic + * OMEC / CU-CP / OAI nFAPI emulator for DU/UE + * Completely runs OMEC, a 4G/LTE core network - not emulation + * Completely runs CU-CP, which generates SD-RAN control plane messages - not emulation + * Emulates DU and UEs (up to three UEs) with OAI nAFPI emulator, which generate LTE control and user plane traffic + * OMEC / CU-CP / OAI DU and UE with USRP hardware and/or LTE smartphones + * Completely runs OMEC, a 4G/LTE core network - not emulation + * Completely runs CU-CP, which generates SD-RAN control plane messages - not emulation + * Completely runs OAI DU together with the CU-CP and a USRP device to open a cell - Software Defined Radio (SDR) device-based emulation which commercial LTE phones can attach + * Completely runs OAI UE with a USRP device to attach the cell the CU-CP and OAI DU opens +* Support End-to-End (E2E) connectivity test + * The user plane E2E test + * Works with CU-CP / OAI nFAPI emulator and CU-CP / OAI DU and UE with USRP hardware cases, since RAN-Sim does not support the data traffic + * The SD-RAN control plane E2E test + * Works with xAPPs such as onos-kpimon and onos-pci + +## RiaB versions and options + +### Version +RiaB has three versions: **latest**, **master-stable**, **dev**, and each release/tag such as **v1.0.0** and **v1.1.0**. + +#### Latest version +The *latest* version of RiaB deploys latest Helm charts and latest Docker container images. + +#### Master-stable version +The *master-stable* version of RiaB deploys latest Helm charts but not latest Docker container images. +It deploys the Docker containers according to the image tag described in each Helm chart. + +#### Release/tag versions +The release/tag version such as *v1.0.0* and *v1.1.0* deployes a specific SD-RAN release version of Helm charts and Docker container images. + +#### Dev version +The *dev* version deploys Helm charts in the `~/helm-charts/sdran-helm-charts` directory and Docker container images `sdran-in-a-box-values.yaml` file. +All other versions initially change the `~/helm-charts/sdran-helm-charts` branch to the specific version. +For example, the *latest* version and *master-stable* version change the the `~/helm-charts/sdran-helm-charts` branch to `master`, +while the *v1.0.0* and *v1.1.0* versions change that branch to *v1.0.0* and *v1.1.0*, respectively. +If we change some Helm chart code in `~/helm-charts/sdran-helm-charts` directory for a test, the above versions will reset to the *master* or a specific version branch. +However, since the *dev* option just uses the Helm chart in `~/helm-charts/sdran-helm-charts` as is, we can test the Helm chart chages. +Also, once we specify image tags in the `sdran-in-a-box-values.yaml` files, this version deploys the Docker containers with the Helm chart change. + +### Options +RiaB also has four options: **ONOS RIC with OAI nFAPI emulator**, **ONOS RIC with RAN-Simulator**, **ONOS RIC with OAI and USRP devices**, and **Facebook-AirHop xAPP use case**. + +#### ONOS RIC with OAI nFAPI emulator option +The *ONOS RIC with OAAI nFAPI emulator* option deploys ONOS RIC services (i.e., Atomix, onos-operator, onos-topo, onos-config, onos-cli, onos-e2t, onos-e2sub, etc) with OMEC, CU-CP, and OAI nFAPI emulator. +It supports E2E connectivities on the data plane and SD-RAN control plane. +The data plane is running without the real radio signal, but nFAPI. + +#### ONOS RIC with RAN-Simulator option +The *ONOS RIC with RAN-Simulator* option deploys ONOS RIC services with RAN-Simulator. +It supports an E2E connectivity on the SD-RAN control plane. + +#### ONOS RIC with OAI and USRP devices option +The *ONOS RIC with OAI and USRP devices* option deploys ONOS RIC services with OMEC, CU-CP, OAI DU, and OAI UE with USRP devices. +It supports E2E connectivities on the data plane and SD-RAN control plane. +The data plane is running with the real radio signal, so that we can also test with commercial LTE smartphones. + +#### Facebook-AirHop use case option +The *Facebook-AirHop use case* option is similar to the *ONOS RIC with RAN-Simulator* option. +The only difference is the PCI xAPP. +This option deploys the PCI xAPP from Facebook-AirHop, while *ONOS RIC with RAN-Simulator* option uses ONF ONOS-PCI xAPP. + + +## Detailed information +See [docs](docs) directory for details of RiaB. diff --git a/docs/Configuration_files.md b/docs/Configuration_files.md new file mode 100644 index 0000000..566f25a --- /dev/null +++ b/docs/Configuration_files.md @@ -0,0 +1,244 @@ +# RiaB configuration + +## Configuration files for each version +If we want to tune RiaB, we should modify below YAML file for each version before the RiaB deployment. +* For `latest` and `dev` version: `sdran-in-a-box-values.yaml` file +* For `master-stable` version: `sdran-in-a-box-values-master-stable.yaml` file +* For a specific release version: `sdran-in-a-box-values-.yaml` file + +## First block - Enables Cassandra DB in OMEC +This is the Cassandra DB configuration for OMEC. We don't really need to change this. +```yaml +cassandra: + config: + cluster_size: 1 + seed_size: 1 +``` + +## Second block - Enables CPU/MEM size configuration +This is for the CPU/MEM size configuration. We don't need to change this. +```yaml +resources: + enabled: false +``` + +## Third block - OMEC, CU-CP, and OAI nFAPI parameter configuration block +This block has parameters for OMEC, CU-CP, OAI DU (nFAPI), and OAI UE (nFAPI) parameter configuration. +```yaml +config: + spgwc: + pfcp: true + ueIpPool: + ip: 172.250.0.0 # if we use RiaB, Makefile script will override this value with the value defined in Makefile script. + upf: + name: "oaisim" + sriov: + enabled: false + hugepage: + enabled: false + cniPlugin: simpleovs + ipam: static + cfgFiles: + upf.json: + mode: af_packet + mme: + cfgFiles: + config.json: + mme: + mcc: + dig1: 2 + dig2: 0 + dig3: 8 + mnc: + dig1: 0 + dig2: 1 + dig3: -1 + apnlist: + internet: "spgwc" + hss: + bootstrap: + users: + - apn: "internet" + key: "465b5ce8b199b49faa5f0a2ee238a6bc" + opc: "d4416644f6154936193433dd20a0ace0" + sqn: 96 + imsiStart: "208014567891201" + msisdnStart: "1122334455" + count: 10 + mmes: + - id: 1 + mme_identity: mme.riab.svc.cluster.local + mme_realm: riab.svc.cluster.local + isdn: "19136246000" + unreachability: 1 + oai-enb-cu: + networks: + f1: + interface: eno1 # if we use RiaB, Makefile script will automatically apply appropriate interface name + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + s1mme: + interface: eno1 # if we use RiaB, Makefile script will automatically apply appropriate interface name + s1u: + interface: enb + oai-enb-du: + mode: nfapi #or local_L1 for USRP and BasicSim + networks: + f1: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + nfapi: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + oai-ue: + numDevices: 1 # support up to 3 + networks: + nfapi: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + onos-e2t: + enabled: "yes" + networks: + e2: + address: 127.0.0.1 # if we use RiaB, Makefile script will automatically apply appropriate interface name + port: 36421 +``` + +Normally, we don't need to touch one of those values except for `numDevices` in `oai-ue` block. +If we want to deploy more UEs, we should change `numDevices` and then deploy RiaB with OAI nFAPI emulator option. + +## Fourth block - image tag block +This block has all containers' repository, tag/version, and pulling policy. +To test a specific Docker container image within RiaB, we should change imamge tag and repository in this block. +```yaml +# for the development, we can use the custom images +# For ONOS-RIC +onos-topo: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-topo + tag: latest +onos-config: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-config + tag: latest +onos-e2t: + service: + external: + enabled: true + e2: + nodePort: 36421 + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-e2t + tag: latest +onos-e2sub: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-e2sub + tag: latest +onos-cli: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-cli + tag: latest +ran-simulator: + image: + pullPolicy: IfNotPresent + repository: onosproject/ran-simulator + tag: latest +onos-kpimon-v1: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-kpimon + tag: latest +onos-kpimon-v2: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-kpimon + tag: latest +onos-pci: + image: + pullPolicy: IfNotPresent + repository: onosproject/onos-pci + tag: latest +fb-ah-xapp: + image: + repository: onosproject/fb-ah-xapp + tag: 0.0.1 + pullPolicy: IfNotPresent +fb-ah-gui: + image: + repository: onosproject/fb-ah-gui + tag: 0.0.1 + pullPolicy: IfNotPresent +ah-eson-test-server: + image: + repository: onosproject/ah-eson-test-server + tag: 0.0.1 + pullPolicy: IfNotPresent + +# For OMEC & OAI +images: + pullPolicy: IfNotPresent + tags: +# For OMEC - Those images are stable image for RiaB +# latest Aether helm chart commit ID: 3d1e936e87b4ddae784a33f036f87899e9d00b95 +# init: docker.io/omecproject/pod-init:1.0.0 +# depCheck: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 + hssdb: docker.io/onosproject/riab-hssdb:v1.0.0 + hss: docker.io/onosproject/riab-hss:v1.0.0 + mme: docker.io/onosproject/riab-nucleus-mme:v1.0.0 + spgwc: docker.io/onosproject/riab-spgw:v1.0.0 + pcrf: docker.io/onosproject/riab-pcrf:v1.0.0 + pcrfdb: docker.io/onosproject/riab-pcrfdb:v1.0.0 + bess: docker.io/onosproject/riab-bess-upf:v1.0.0 + pfcpiface: docker.io/onosproject/riab-pfcpiface:v1.0.0 +# For OAI + oaicucp: docker.io/onosproject/oai-enb-cu:latest + oaidu: docker.io/onosproject/oai-enb-du:latest + oaiue: docker.io/onosproject/oai-ue:latest +``` + +## Fifth block - import block +This block is the import block to speicify which services are onboarded in ONOS RIC services. +In fact, ONOS RIC services are packaged as a single umbrella Helm chart. +By adjusting this block, we can configure which service is onboarded and which service is not deployed. +```yaml +# For SD-RAN Umbrella chart: +# ONOS-KPIMON xAPP is imported in the RiaB by default +import: + onos-kpimon-v1: + enabled: false + onos-kpimon-v2: + enabled: true + onos-pci: + enabled: true +# Other ONOS-RIC micro-services +# onos-topo: +# enabled: true +# onos-e2t: +# enabled: true +# onos-e2sub: +# enabled: true +# onos-o1t: +# enabled: false +# onos-config: +# enabled: true +# onos-sdran-cli: +# enabled: true +# ran-simulator chart is automatically imported when pushing ransim option +# ran-simulator: +# enabled: false +# onos-gui: +# enabled: false +# nem-monitoring: +# enabled: false +# fb-ah-xapp, fb-ah-gui, and ah-eson-test-server are automatically imported when pushing fbc-pci option +# fb-ah-xapp: +# enabled: false +# fb-ah-gui: +# enabled: false +# ah-eson-test-server: +# enabled: false +``` \ No newline at end of file diff --git a/docs/HW_Installation_intro.md b/docs/HW_Installation_intro.md new file mode 100644 index 0000000..c05eb13 --- /dev/null +++ b/docs/HW_Installation_intro.md @@ -0,0 +1,332 @@ +# Hardware Installation + +This installation shows how to run the ONF SDRAN setup using ONOS-RIC, OMEC, and OAI CU/DU and UE components. +The OAI components perform connection via USRP B210 radio equipment attached to NUC machines. +This setup can be utilized as a reference implementation of the ONOS-RIC in hardware. + +Prepare two NUC machines, each installed with a Ubuntu 18.04 server Operating System. +One NUC machine will be used to run a UE setup connected to a USRP B210. The other NUC machine will be used to run the eNodeB OAI components (CU/DU) connected to another B210 device. +Prepare other two machines (or Virtual Machines - VMs) to install decomposed parts the sdRan-in-a-Box (RiaB), in one of them the RIC (ONOS-RIC) will be executed, while in the other the EPC (OMEC) will be executed - both over Kubernetes. +**Those machines (or VMs) should be connected into the same subnet (via a switch or direct connection). In all machines install Ubuntu 18.04 server first.** + +*NOTE: In the below sections, we have the following IP addresses assigned: NUC-OAI-CU/DU (192.168.13.21), NUC-UE (192.168.13.22), ONOS-RIC (192.168.10.22), and EPC-OMEC (192.168.10.21). +These IP addresses are assigned to the eno1 interface in each server, i.e., the interface associated with the default gateway route. In case of a custom setup with different IP addresses assigned to the VMs, make sure the IP addresses (and their subnets/masks) are properly referenced in the configurations utilized in this tutorial. The NUC-UE machine only needs an IP address for the setup of the OAI software requirements and components.* + +When complete, the hardware installation detains the settings shown in the following diagram: + +![Hardware Installation Setup - Logical View](./figures/hw_install.png) + +In the diagram above are presented the 4 machines described in this setup (NUC OAI-CU/DU, NUC OAI-UE, ONOS-RIC, EPC-OMEC) interconnected via a logical layer 2 network. In each represented machine, the red squared items represent the components instantiated on them using RiaB. In particular, the EPC-OMEC machine highlights the internal structure of the omec-user-plane components, presenting how the UPF and Quagga router are interconnected via different (Open vSwitch) bridges to perform the user plane communication of the EPC. Notice, the UE component is represented by a NUC OAI-UE, likewise it could be replaced by a LTE handset. + +## Credentials +The tutorials here presented utilize the sdRan-in-a-Box (RiaB) repository, while installing and running the RiaB components, we might have to write some credentials for (i) opencord gerrit, (ii) onosproject github, and (iii) sdran private Helm chart repository. Make sure you have this member-only credentials before starting to install RiaB. + +```bash +aether-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires HTTPS key +Cloning into '/users/wkim/helm-charts/aether-helm-charts'... +Username for 'https://gerrit.opencord.org': +Password for 'https://@gerrit.opencord.org': +remote: Total 1103 (delta 0), reused 1103 (delta 0) +Receiving objects: 100% (1103/1103), 526.14 KiB | 5.31 MiB/s, done. +Resolving deltas: 100% (604/604), done. +sdran-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires Github credential +Cloning into '/users/wkim/helm-charts/sdran-helm-charts'... +Username for 'https://github.com': +Password for 'https://@github.com': +remote: Enumerating objects: 19, done. +remote: Counting objects: 100% (19/19), done. +remote: Compressing objects: 100% (17/17), done. +remote: Total 2259 (delta 7), reused 3 (delta 2), pack-reused 2240 +Receiving objects: 100% (2259/2259), 559.35 KiB | 2.60 MiB/s, done. +Resolving deltas: 100% (1558/1558), done. + +..... + +helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ +"incubator" has been added to your repositories +helm repo add cord https://charts.opencord.org +"cord" has been added to your repositories +Username for ONF SDRAN private chart: +Password for ONF SDRAN private chart: +"sdran" has been added to your repositories +touch /tmp/build/milestones/helm-ready +``` + +## Get the RiaB source code +To get the source code, please see: `https://github.com/onosproject/sdran-in-a-box`. + +Since SDRAN-in-a-Box repository is a member-only repository, a user should log in github and then check the git clone command on that web site. + +**Clone the RiaB repository to all the machines EPC-OMEC, ONOS-RIC, OAI-CU/DU, OAI-UE.**. + +In all the machines, after downloading the source code, in the cloned source code folder, edit the sdran-in-a-box-values.yaml file and change the file as below (we can copy and paste). + +## Change sdran-in-a-box-values yaml file + +In the cloned source code of the folder sdran-in-a-box, overwrite the content of the file sdran-in-a-box-values.yaml as provided below (copy and paste). + +```yaml +# Copyright 2020-present Open Networking Foundation +# +# SPDX-License-Identifier: LicenseRef-ONF-Member-Only-1.0 + +# cassandra values +cassandra: + config: + cluster_size: 1 + seed_size: 1 + +resources: + enabled: false + +config: + spgwc: + pfcp: true + multiUpfs: true + jsonCfgFiles: + subscriber_mapping.json: + subscriber-selection-rules: + - selected-user-plane-profile: "menlo" + keys: + serving-plmn: + mcc: 315 + mnc: 10 + tac: 1 + priority: 5 + selected-access-profile: + - access-all + selected-apn-profile: "apn-internet-menlo" + selected-qos-profile: "qos-profile1" + user-plane-profiles: + menlo: + user-plane: "upf.riab.svc.cluster.local" + apn-profiles: + apn-internet-default: + apn-name: "internet" + usage: 1 + network: "lbo" + gx_enabled: true + dns_primary: "1.1.1.1" + dns_secondary: "8.8.8.8" + mtu: 1400 + apn-internet-menlo: + apn-name: "internet" + usage: 1 + network: "lbo" + gx_enabled: true + dns_primary: "8.8.8.8" + dns_secondary: "1.1.1.1" + mtu: 1400 + ueIpPool: + ip: 172.250.0.0 # if we use RiaB, Makefile script will override this value with the value defined in Makefile script. + upf: + name: "oaisim" + sriov: + enabled: false + hugepage: + enabled: false + cniPlugin: simpleovs + ipam: static + cfgFiles: + upf.json: + mode: af_packet + mme: + address: 192.168.10.21 # Set here the IP address of the interface eno1 (default gw) in the VM where OMEC is executed + cfgFiles: + config.json: + mme: + mcc: + dig1: 3 + dig2: 1 + dig3: 5 + mnc: + dig1: 0 + dig2: 1 + dig3: 0 + apnlist: + internet: "spgwc" + hss: + bootstrap: + users: + - apn: "internet" + key: "000102030405060708090a0b0c0d0e0f" + opc: "69d5c2eb2e2e624750541d3bbc692ba5" + sqn: 96 + imsiStart: "315010206000001" + msisdnStart: "1122334455" + count: 30 + mmes: + - id: 1 + mme_identity: mme.riab.svc.cluster.local + mme_realm: riab.svc.cluster.local + isdn: "19136246000" + unreachability: 1 + oai-enb-cu: + networks: + f1: + interface: eno1 # if we use RiaB, Makefile script will automatically apply appropriate interface name + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + s1mme: + interface: eno1 # if we use RiaB, Makefile script will automatically apply appropriate interface name + s1u: + interface: eno1 + plmnID: + mcc: "315" + mnc: "010" + length: 3 + fullName: "ONF SDRAN" + shortName: "SDRAN" + oai-enb-du: + enableUSRP: true + mode: nfapi #or local_L1 for USRP and BasicSim + networks: + f1: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + nfapi: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + oai-ue: + enableUSRP: true + networks: + nfapi: + interface: eno1 #if we use RiaB, Makefile script will automatically apply appropriate IP address + address: 10.128.100.100 #if we use RiaB, Makefile script will automatically apply appropriate IP address + sim: + msinStart: "206000001" + apiKey: "000102030405060708090a0b0c0d0e0f" + opc: "69d5c2eb2e2e624750541d3bbc692ba5" + msisdnStart: "1122334455" + onos-e2t: + enabled: "yes" + networks: + e2: + address: 192.168.10.22 # Set here the IP address of the interface eno1 (default gw) in the VM where RIC is executed + port: 36421 + +# for the development, we can use the custom images +# For ONOS-RIC +# onos-topo: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-topo +# tag: latest +# onos-config: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-config +# tag: latest +onos-e2t: + service: + external: + enabled: true + e2: + nodePort: 36421 +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-e2t +# tag: latest +# onos-e2sub: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-e2sub +# tag: latest +# onos-cli: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-cli +# tag: latest +# ran-simulator: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/ran-simulator +# tag: latest +# onos-kpimon-v1: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-kpimon +# tag: latest +# onos-kpimon-v2: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-kpimon +# tag: latest +# onos-pci: +# image: +# pullPolicy: IfNotPresent +# repository: onosproject/onos-pci +# tag: latest +# fb-ah-xapp: +# image: +# repository: onosproject/fb-ah-xapp +# tag: 0.0.1 +# pullPolicy: IfNotPresent +# fb-ah-gui: +# image: +# repository: onosproject/fb-ah-gui +# tag: 0.0.1 +# pullPolicy: IfNotPresent +# ah-eson-test-server: +# image: +# repository: onosproject/ah-eson-test-server +# tag: 0.0.1 +# pullPolicy: IfNotPresent + +# For OMEC & OAI +images: + pullPolicy: IfNotPresent + tags: +# For OMEC - Those images are stable image for RiaB +# latest Aether helm chart commit ID: 3d1e936e87b4ddae784a33f036f87899e9d00b95 +# init: docker.io/omecproject/pod-init:1.0.0 +# depCheck: quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 + hssdb: docker.io/onosproject/riab-hssdb:v1.0.0 + hss: docker.io/onosproject/riab-hss:v1.0.0 + mme: docker.io/onosproject/riab-nucleus-mme:v1.0.0 + spgwc: docker.io/onosproject/riab-spgw:v1.0.0 + pcrf: docker.io/onosproject/riab-pcrf:v1.0.0 + pcrfdb: docker.io/onosproject/riab-pcrfdb:v1.0.0 + bess: docker.io/onosproject/riab-bess-upf:v1.0.0 + pfcpiface: docker.io/onosproject/riab-pfcpiface:v1.0.0 +# For OAI + oaicucp: docker.io/onosproject/oai-enb-cu:v0.1.4 + oaidu: docker.io/onosproject/oai-enb-du:v0.1.4 + oaiue: docker.io/onosproject/oai-ue:v0.1.4 + +# For SD-RAN Umbrella chart: +# ONOS-KPIMON xAPP is imported in the RiaB by default +import: + onos-kpimon-v1: + enabled: false + onos-kpimon-v2: + enabled: true + onos-pci: + enabled: true +# Other ONOS-RIC micro-services +# onos-topo: +# enabled: true +# onos-e2t: +# enabled: true +# onos-e2sub: +# enabled: true +# onos-o1t: +# enabled: false +# onos-config: +# enabled: true +# onos-cli: +# enabled: true +# ran-simulator chart is automatically imported when pushing ransim option +# ran-simulator: +# enabled: false +# onos-gui: +# enabled: false +# nem-monitoring: +# enabled: false +# fb-ah-xapp, fb-ah-gui, and ah-eson-test-server are automatically imported when pushing fbc-pci option +# fb-ah-xapp: +# enabled: false +# fb-ah-gui: +# enabled: false +# ah-eson-test-server: +# enabled: false +``` \ No newline at end of file diff --git a/docs/HW_Installation_oai_enb.md b/docs/HW_Installation_oai_enb.md new file mode 100644 index 0000000..54f94e5 --- /dev/null +++ b/docs/HW_Installation_oai_enb.md @@ -0,0 +1,74 @@ +# Install OAI CU/DU + +This section explains how to execute only the OAI components using the RiaB Makefile. + +Bofere proceeding, make sure you follow all the instructions on how to install the OAI/USRP requirements prerequisites in the NUC machines. + +In the RiaB makefile targets are included options to execute OAI CU/DU/UE components using helm charts. Those steps do not require any source code compilation of OAI, the OAI components run in docker images and have their parameters configured by the sdran-in-a-box-values.yaml file. + +**Notice: The sdran-in-a-box-values.yaml contain the latest versions/tags of the OAI docker images. In order to use the versions of the OAI docker images specified in RiaB v1.0.0 or v1.1.0 releases make sure to respectively copy and paste to the sdran-in-a-box-values.yaml file the contents of the files sdran-in-a-box-values-v1.0.0.yaml and sdran-in-a-box-values-v1.1.0.yaml as needed.** + +## Network parameter configuration + +We should then configure the network parameters (e.g., routing rules, MTU size, and packet fregmentation) on the OAI-CU/DU machine. + + +### Configure the secondary IP address on the OAI NUC +Before run CU-CP, the NUC machine for OAI should have a secondary IP address on the Ethernet port. +The secondary IP address should have one of the IP address in `192.168.11.8/29` subnet. +The purpose of this IP address is to communicate with the other NUC machine which RiaB is running inside. +```bash +$ sudo ip a add 192.168.11.10/29 dev eno1 +``` +*NOTE: The reference setup has 192.168.11.10/29 for the secondary IP address, as defined in the same network prefix 192.168. as OMEC-EPC.* + + +### Install some network tools +```bash +$ sudo apt install net-tools ethtool +``` + +*NOTE: Normally, those tools are already installed. If not, we can command it.* + +### Configuration in OAI-CU/DU machine +Last, we should configure network configuration in the OAI-CU/DU NUC machine. +We should go to the the OAI-CU/DU NUC machine and change the network configuration such as TX/RX checksum, GRO, and routing rules. + +```bash +$ sudo ethtool -K eno1 tx off rx off gro off gso off +$ sudo route del -net 192.168.11.8/29 dev eno1 # ignore error if happened +$ sudo route add -net 192.168.11.0/29 gw 192.168.10.21 dev eno1 # This route forwards traffic to the EPC machine +$ sudo route add -net 192.168.11.8/29 gw 192.168.10.21 dev eno1 # This route forwards traffic to the EPC machine +$ sudo route add -net 192.168.11.16/29 gw 192.168.10.21 dev eno1 # This route forwards traffic to the EPC machine +``` + +## Run OAI eNB CU/DU + +```bash +$ cd /path/to/sdran-in-a-box +$ sudo apt install build-essential +$ make oai-enb-usrp +``` + +This command starts the execution of oai-enb-cu and oai-enb-du components. + +This step might take some time due to the download of the oai-enb-cu and oai-enb-du docker images. +After the conditions (pod/oai-enb-cu-0 condition met and pod/oai-enb-du-0 condition met) were achieved the deployment was successful. + +The pod pod/oai-enb-du-0 takes some time to start as it needs to configure the USRP first. + + +## Stop/Clean OAI components + +After finishing the hardware installation setup procedures, run the command below to delete all deployed Helm charts for OAI CU/DU components: + +```bash +$ make reset-oai +``` + +And this deletes not only deployed Helm chart but also Kubernetes and Helm. + +```bash +make clean # if we want to keep the ~/helm-charts directory - option to develop/test changed/new Helm charts +make clean-all # if we also want to delete ~/helm-charts directory +``` diff --git a/docs/HW_Installation_oai_source.md b/docs/HW_Installation_oai_source.md new file mode 100644 index 0000000..60d91d3 --- /dev/null +++ b/docs/HW_Installation_oai_source.md @@ -0,0 +1,362 @@ +# Install OAI CU/DU/UE on Baremetal + +This section explains how to compile and execute only the OAI components from the source code. + +Bofere proceeding, make sure you follow all the instructions on how to install the OAI/USRP requirements prerequisites in the NUC machines. + +## Install UHD driver and push UHD image to USRP B210 device +Once we finished to check that the power management and CPU frequency configuration are good, we should reboot NUC machine again. +After the NUC is completely rebooted, then we should connect the USRP B210 device to the NUC machine. +To make the USRP B210 device run along with the NUC board, we should install UHD driver on the NUC machine. +And then, we should push the UHD image to USRP B210 device. + +```bash +$ # Install UHD driver +$ sudo apt-get install libuhd-dev libuhd003 uhd-host +$ # Push UHD image to the USRP B210 device +$ sudo uhd_images_downloader +``` + +*NOTE 1: When we cannot install `libuhd003`, we can replace it with `libuhd003.010.003`.* + +*NOTE 2: USRP B210 device has a power cable. We should keep that plugged in.* +*If we plugged off the USRP B210 power due to whatever reasons, we should push UHD image to USRP B210 device again by using `uhd_image_downloader` command.* + +## Verification of UHD driver installation and UHD image push +In order to verify the UHD driver installation and UHD image push to the USRP B210 driver, we can use below command. +```bash +$ uhd_usrp_probe +[INFO] [UHD] linux; GNU C++ version 7.5.0; Boost_106501; UHD_3.15.0.0-release +[INFO] [B200] Detected Device: B210 +[INFO] [B200] Operating over USB 2. +[INFO] [B200] Initialize CODEC control... +[INFO] [B200] Initialize Radio control... +[INFO] [B200] Performing register loopback test... +[INFO] [B200] Register loopback test passed +[INFO] [B200] Performing register loopback test... +[INFO] [B200] Register loopback test passed +[INFO] [B200] Setting master clock rate selection to 'automatic'. +[INFO] [B200] Asking for clock rate 16.000000 MHz... +[INFO] [B200] Actually got clock rate 16.000000 MHz. + _____________________________________________________ + / +| Device: B-Series Device +| _____________________________________________________ +| / +| | Mboard: B210 +| | serial: 31EABDC +| | name: MyB210 +| | product: 2 +| | revision: 4 +| | FW Version: 8.0 +| | FPGA Version: 16.0 +| | +| | Time sources: none, internal, external, gpsdo +| | Clock sources: internal, external, gpsdo +| | Sensors: ref_locked +| | _____________________________________________________ +| | / +| | | RX DSP: 0 +| | | +| | | Freq range: -8.000 to 8.000 MHz +| | _____________________________________________________ +| | / +| | | RX DSP: 1 +| | | +| | | Freq range: -8.000 to 8.000 MHz +| | _____________________________________________________ +| | / +| | | RX Dboard: A +| | | _____________________________________________________ +| | | / +| | | | RX Frontend: A +| | | | Name: FE-RX2 +| | | | Antennas: TX/RX, RX2 +| | | | Sensors: temp, rssi, lo_locked +| | | | Freq range: 50.000 to 6000.000 MHz +| | | | Gain range PGA: 0.0 to 76.0 step 1.0 dB +| | | | Bandwidth range: 200000.0 to 56000000.0 step 0.0 Hz +| | | | Connection Type: IQ +| | | | Uses LO offset: No +| | | _____________________________________________________ +| | | / +| | | | RX Frontend: B +| | | | Name: FE-RX1 +| | | | Antennas: TX/RX, RX2 +| | | | Sensors: temp, rssi, lo_locked +| | | | Freq range: 50.000 to 6000.000 MHz +| | | | Gain range PGA: 0.0 to 76.0 step 1.0 dB +| | | | Bandwidth range: 200000.0 to 56000000.0 step 0.0 Hz +| | | | Connection Type: IQ +| | | | Uses LO offset: No +| | | _____________________________________________________ +| | | / +| | | | RX Codec: A +| | | | Name: B210 RX dual ADC +| | | | Gain Elements: None +| | _____________________________________________________ +| | / +| | | TX DSP: 0 +| | | +| | | Freq range: -8.000 to 8.000 MHz +| | _____________________________________________________ +| | / +| | | TX DSP: 1 +| | | +| | | Freq range: -8.000 to 8.000 MHz +| | _____________________________________________________ +| | / +| | | TX Dboard: A +| | | _____________________________________________________ +| | | / +| | | | TX Frontend: A +| | | | Name: FE-TX2 +| | | | Antennas: TX/RX +| | | | Sensors: temp, lo_locked +| | | | Freq range: 50.000 to 6000.000 MHz +| | | | Gain range PGA: 0.0 to 89.8 step 0.2 dB +| | | | Bandwidth range: 200000.0 to 56000000.0 step 0.0 Hz +| | | | Connection Type: IQ +| | | | Uses LO offset: No +| | | _____________________________________________________ +| | | / +| | | | TX Frontend: B +| | | | Name: FE-TX1 +| | | | Antennas: TX/RX +| | | | Sensors: temp, lo_locked +| | | | Freq range: 50.000 to 6000.000 MHz +| | | | Gain range PGA: 0.0 to 89.8 step 0.2 dB +| | | | Bandwidth range: 200000.0 to 56000000.0 step 0.0 Hz +| | | | Connection Type: IQ +| | | | Uses LO offset: No +| | | _____________________________________________________ +| | | / +| | | | TX Codec: A +| | | | Name: B210 TX dual DAC +| | | | Gain Elements: None +``` + +If we see the above results which shows the device name `B210`, we are good to go. + +## Build OAI + +The OAI repository (https://github.com/onosproject/openairinterface5g) used in this tutorial requires member-only access, a user should log in github and then check the git clone command on that web site. + +After the USRP configuration, we should build the OAI code. +```bash +$ git clone https://github.com/onosproject/openairinterface5g +$ cd /path/to/openairinterface5g +$ source oaienv +$ cd cmake_targets/ +$ ./build_oai -c -I --eNB --UE -w USRP -g --build-ric-agent +``` + +*NOTE: It takes really a long time to build OAI.* + + +## Configure CU-CP on the OAI NUC +After that, we should copy the sample CU-CP configuration file in the HOME directory. +```bash +$ cp /path/to/openairinterface5g/ci-scripts/conf_files/cu.band7.tm1.25PRB.conf ~/cu.onf.conf +``` + +Then, modify below parameters in the copied file `~/cu.onf.conf`: +```text +… +////////// RIC parameters: +RIC : { + remote_ipv4_addr = "192.168.10.22"; + remote_port = 36421; + enabled = "yes"; +}; +… + +tracking_area_code = 1; +plmn_list = ( { mcc = 315; mnc = 010; mnc_length = 3; } ) +… + ////////// MME parameters: + mme_ip_address = ( + { + ipv4 = "192.168.10.21"; // *Write down EPC-CORE SDRAN-in-a-Box IP* + ipv6 = "192:168:30::17"; // *Don’t care* + active = "yes"; + preference = "ipv4"; + } + ); + + NETWORK_INTERFACES : { + ENB_INTERFACE_NAME_FOR_S1_MME = "eno1"; // Ethernet interface name of OAI NUC + ENB_IPV4_ADDRESS_FOR_S1_MME = "192.168.13.21/16"; // OAI NUC IP address + ENB_INTERFACE_NAME_FOR_S1U = "eno1"; // Ethernet interface name of OAI NUC + ENB_IPV4_ADDRESS_FOR_S1U = "192.168.11.10/29"; // Write the secondary IP address which we set above + ENB_PORT_FOR_S1U = 2152; # Spec 2152 # Don't touch + ENB_IPV4_ADDRESS_FOR_X2C = "192.168.13.21/16"; // OAI NUC IP address + ENB_PORT_FOR_X2C = 36422; # Spec 36422 # Don't touch + }; + } + +``` + +## Configure DU on the OAI NUC +Likewise, we should copy the sample DU configuration file in the HOME directory. +```bash +$ cp /path/to/openairinterface5g/ci-scripts/conf_files/du.band7.tm1.25PRB.conf ~/du.onf.conf +``` + +And then, we should open the copied file `~/du.onf.conf` and change the blow variables: +```text +tracking_area_code = 1001; +plmn_list = ( { mcc = 315; mnc = 010; mnc_length = 3; } ) +``` + +## Run CU and DU in the OAI-CU/DU machine + +Before running CU and DU, make sure to follow the instructions provided at [Network Parameter Configuration](#network-parameter-configuration). + +### Run CU-CP +On the OAI NUC machine, we should go to `/path/to/openairinterface5g/cmake_targets/ran_build/build` and run the command below: +```bash +ENODEB=1 sudo -E ./lte-softmodem -O ~/cu.onf.conf +``` + +### Run DU +After CU-CP is running, in another terminal we should go to `/path/to/openairinterface5g/cmake_targets/ran_build/build` and run the command below: +```bash +while true; do ENODEB=1 sudo -E ./lte-softmodem -O ~/du.onf.conf; done +``` + +## Run the User Equipment (UE) on the OAI-UE machine + +Write down a file named ~/sim.conf with the following content: + +```text +# List of known PLMNS +PLMN: { + PLMN0: { + FULLNAME="Test network"; + SHORTNAME="OAI4G"; + MNC="01"; + MCC="001"; + + }; + PLMN1: { + FULLNAME="SFR France"; + SHORTNAME="SFR"; + MNC="10"; + MCC="208"; + + }; + PLMN2: { + FULLNAME="SFR France"; + SHORTNAME="SFR"; + MNC="11"; + MCC="208"; + }; + PLMN3: { + FULLNAME="SFR France"; + SHORTNAME="SFR"; + MNC="13"; + MCC="208"; + }; + PLMN4: { + FULLNAME="Aether"; + SHORTNAME="Aether"; + MNC="010"; + MCC="315"; + }; + PLMN5: { + FULLNAME="T-Mobile USA"; + SHORTNAME="T-Mobile"; + MNC="280"; + MCC="310"; + }; + PLMN6: { + FULLNAME="FICTITIOUS USA"; + SHORTNAME="FICTITIO"; + MNC="028"; + MCC="310"; + }; + PLMN7: { + FULLNAME="Vodafone Italia"; + SHORTNAME="VODAFONE"; + MNC="10"; + MCC="222"; + }; + PLMN8: { + FULLNAME="Vodafone Spain"; + SHORTNAME="VODAFONE"; + MNC="01"; + MCC="214"; + }; + PLMN9: { + FULLNAME="Vodafone Spain"; + SHORTNAME="VODAFONE"; + MNC="06"; + MCC="214"; + }; + PLMN10: { + FULLNAME="Vodafone Germ"; + SHORTNAME="VODAFONE"; + MNC="02"; + MCC="262"; + }; + PLMN11: { + FULLNAME="Vodafone Germ"; + SHORTNAME="VODAFONE"; + MNC="04"; + MCC="262"; + }; +}; + +UE0: +{ + USER: { + IMEI="356113022094149"; + MANUFACTURER="EURECOM"; + MODEL="LTE Android PC"; + PIN="0000"; + }; + + SIM: { + MSIN="206000001"; + USIM_API_K="000102030405060708090a0b0c0d0e0f"; + OPC= "69d5c2eb2e2e624750541d3bbc692ba5"; + MSISDN="1122334455"; + }; + + # Home PLMN Selector with Access Technology + HPLMN= "315010"; + + # User controlled PLMN Selector with Access Technology + UCPLMN_LIST = (); + + # Operator PLMN List + OPLMN_LIST = ("00101", "20810", "20811", "20813", "315010", "310280", "310028"); + + # Operator controlled PLMN Selector with Access Technology + OCPLMN_LIST = ("22210", "21401", "21406", "26202", "26204"); + + # Forbidden plmns + FPLMN_LIST = (); + + # List of Equivalent HPLMNs +#TODO: UE does not connect if set, to be fixed in the UE +# EHPLMN_LIST= ("20811", "20813"); + EHPLMN_LIST= (); +}; + +``` + +On the OAI-UE NUC machine, we should go to `/path/to/openairinterface5g/cmake_targets/ran_build/build`. + +Then execute the command below to generate the UE settings. + +```bash +../../nas_sim_tools/build/conf2uedata -c ~/sim.conf -o . +``` + +After that, initialize the UE process: + +```bash +sudo ./lte-uesoftmodem -C 2630000000 -r 25 --ue-rxgain 120 --ue-txgain 0 --ue-max-power 0 --ue-scan-carrier --nokrnmod 1 2>&1 | tee UE.log +``` \ No newline at end of file diff --git a/docs/HW_Installation_oai_ue.md b/docs/HW_Installation_oai_ue.md new file mode 100644 index 0000000..3e18fb5 --- /dev/null +++ b/docs/HW_Installation_oai_ue.md @@ -0,0 +1,37 @@ +# Install OAI UE + +This section explains how to execute only the OAI components using the RiaB Makefile. + +Bofere proceeding, make sure you follow all the instructions on how to install the OAI/USRP requirements prerequisites in the NUC machines. + +In the RiaB makefile targets are included options to execute OAI CU/DU/UE components using helm charts. Those steps do not require any source code compilation of OAI, the OAI components run in docker images and have their parameters configured by the sdran-in-a-box-values.yaml file. + +**Notice: The sdran-in-a-box-values.yaml contain the latest versions/tags of the OAI docker images. In order to use the versions of the OAI docker images specified in RiaB v1.0.0 or v1.1.0 releases make sure to respectively copy and paste to the sdran-in-a-box-values.yaml file the contents of the files sdran-in-a-box-values-v1.0.0.yaml and sdran-in-a-box-values-v1.1.0.yaml as needed.** + +## Run OAI UE +```bash +$ cd /path/to/sdran-in-a-box +$ sudo apt install build-essential +$ make oai-ue-usrp +``` + +This step might take some time due to the download of the oai-ue docker image. +After the condition (pod/oai-ue-0 condition met) were achieved proceed to the next topic. + +The pod pod/oai-ue-0 takes some time to start as it needs to configure the USRP first. + + +## Stop/Clean OAI components + +After finishing the hardware installation setup procedures, run the command below to delete all deployed Helm charts for OAI UE component. + +```bash +$ make reset-oai +``` + +And this deletes not only deployed Helm chart but also Kubernetes and Helm. + +```bash +make clean # if we want to keep the ~/helm-charts directory - option to develop/test changed/new Helm charts +make clean-all # if we also want to delete ~/helm-charts directory +``` diff --git a/docs/HW_Installation_omec.md b/docs/HW_Installation_omec.md new file mode 100644 index 0000000..ef0b875 --- /dev/null +++ b/docs/HW_Installation_omec.md @@ -0,0 +1,156 @@ +# Install OMEC + +This section explains how to install the EPC OMEC components using RiaB in the EPC-OMEC machine. + +## Change the target /fabric in the Makefile + +In the cloned RiaB repository at the EPC-OMEC machine, edit the Makefile target to look like the following lines below. + +*Note: The IP addresses prefix (i.e., 192.168.x.z) correspond to the prefix assigned to the same subnet where the whole setup is defined. In a custom setup, make sure these IP addresses subnet match too.* + +```bash +$(M)/fabric: | $(M)/setup /opt/cni/bin/simpleovs /opt/cni/bin/static + sudo apt install -y openvswitch-switch + sudo ovs-vsctl --may-exist add-br br-enb-net + sudo ovs-vsctl --may-exist add-port br-enb-net enb -- set Interface enb type=internal + sudo ip addr add 192.168.11.12/29 dev enb || true + sudo ip link set enb up + sudo ethtool --offload enb tx off + sudo ip route replace 192.168.11.16/29 via 192.168.11.9 dev enb + kubectl apply -f $(RESOURCEDIR)/router.yaml + kubectl wait pod -n default --for=condition=Ready -l app=router --timeout=300s + kubectl -n default exec router ip route add $(UE_IP_POOL)/$(UE_IP_MASK) via 192.168.11.10 + kubectl delete net-attach-def core-net + touch $@ +``` + +**These IP addresses are assigned to OMEC because they must be reachable by the NUC-OAI-CU/DU machine, so the oai-enb-cu component can communicate with the omec-mme component. More details about custom settings are explained in the [Custom Settings](#network-routes-and-ip-addresses).** + + +## Change the router networks + +In the cloned RiaB repository at the EPC-OMEC machine, edit the file located at path-to/sdran-in-a-box/resources/router.yaml, so the router Pod have its networks annotations to look like the lines below: + +*Note: The IP addresses prefix (i.e., 192.168.x.z) correspond to the prefix assigned to the same subnet where the whole setup is defined. In a custom setup, make sure these IP addresses subnet match too.* + +```text +… +apiVersion: v1 +kind: Pod +metadata: + name: router + labels: + app: router + annotations: + k8s.v1.cni.cncf.io/networks: '[ + { "name": "core-net", "interface": "core-rtr", "ips": ["192.168.11.1/29"] }, + { "name": "enb-net", "interface": "enb-rtr", "ips": ["192.168.11.9/29"] }, + { "name": "access-net", "interface": "access-rtr", "ips": ["192.168.11.17/29"] } + ]' +… +``` + +**These IP addresses are assigned to a router pod in the OMEC VM, making possible the UPF component of OMEC can communicate with the enb and core networks. More details about custom settings are explained in the [Custom Settings](#network-routes-and-ip-addresses).** + + +## Start the RiaB EPC-OMEC components + +After changing the file `sdran-in-a-box-values.yaml`, run the following commands: + +```bash +$ cd /path/to/sdran-in-a-box +$ sudo apt install build-essential +$ make omec +``` + +## Verify whether everything is up and running +After a while, RiaB Makefile completes to install K8s and deploy OMEC CP, OMEC UP, and an internal router. +Once it is done, you can check with the below command in the EPC-OMEC machine. +```bash +$ kubectl get pods --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +default router 1/1 Running 0 19h +kube-system calico-kube-controllers-865c7978b5-k6f62 1/1 Running 0 19h +kube-system calico-node-bldr4 1/1 Running 0 19h +kube-system coredns-dff8fc7d-hqfcn 1/1 Running 0 19h +kube-system dns-autoscaler-5d74bb9b8f-5w2j4 1/1 Running 0 19h +kube-system kube-apiserver-node1 1/1 Running 0 19h +kube-system kube-controller-manager-node1 1/1 Running 0 19h +kube-system kube-multus-ds-amd64-jzvzr 1/1 Running 0 19h +kube-system kube-proxy-wclnq 1/1 Running 0 19h +kube-system kube-scheduler-node1 1/1 Running 0 19h +kube-system kubernetes-dashboard-667c4c65f8-bqkgl 1/1 Running 0 19h +kube-system kubernetes-metrics-scraper-54fbb4d595-7kjss 1/1 Running 0 19h +kube-system nodelocaldns-p6j8m 1/1 Running 0 19h +omec cassandra-0 1/1 Running 0 113m +omec hss-0 1/1 Running 0 113m +omec mme-0 4/4 Running 0 113m +omec pcrf-0 1/1 Running 0 113m +omec spgwc-0 2/2 Running 0 113m +omec upf-0 4/4 Running 0 112m +``` +If you can see the router and all OMEC PODs are running, then everything is good to go. + +## Network parameter configuration + +We should then configure the network parameters (e.g., routing rules, MTU size, and packet fregmentation) on EPC-OMEC so it can reach out to OAI-CU/DU machine. + +### Install some network tools +```bash +$ sudo apt install net-tools ethtool +``` + +*NOTE: Normally, those tools are already installed. If not, we can command it.* + +### Configuration in EPC-OMEC machine +First, we should go to the EPC-OMEC machine. + +We should add a single routing rule and disable TCP TX/RX checksum and Generic Receive Offloading (GRO) configuration. +```bash +$ ROUTER_IP=$(kubectl exec -it router -- ifconfig eth0 | grep inet | awk '{print $2}' | awk -F ':' '{print $2}') +$ ROUTER_IF=$(route -n | grep $ROUTER_IP | awk '{print $NF}') +$ sudo ethtool -K $ROUTER_IF gro off rx off +$ sudo ethtool -K eno1 rx off tx on gro off gso on #Notice here, this is the primary interface of the EPC-OMEC machine +$ sudo ethtool -K enb rx off tx on gro off gso on +$ sudo route add -host 192.168.11.10 gw 192.168.13.21 dev eno1 #Defines the route to OAI-CU/DU secondary IP address + +``` + +### Configuration in EPC-OMEC internal router +Second, we should configure network parameters in the EPC-OMEC internal router. +In order to access the EPC-OMEC internal router, go to the EPC-OMEC machine and command below: + +```bash +$ kubectl exec -it router -- bash +``` + +On the router prompt, we initially add a routing rule and MTU size. +Then, we should disable TX/RX checksum and GRO for all network interfaces in the router. + +```bash +$ # Add routing rule +$ route add -host 192.168.11.10 gw 192.168.11.12 dev enb-rtr #Defines the route to OAI-CU/DU machine (secondary IP address) via the enb interface (attached to br-enb-rtr bridge) + +$ # Change MTU size +$ ifconfig core-rtr mtu 1550 +$ ifconfig access-rtr mtu 1550 + +$ # Disable checksum and GRO +$ apt update; apt install ethtool +$ ethtool -K eth0 tx off rx off gro off gso off +$ ethtool -K enb-rtr tx off rx off gro off gso off +$ ethtool -K access-rtr tx off rx off gro off gso off +$ ethtool -K core-rtr tx off rx off gro off gso off +``` + +### Configuration in UPF +Next, we should go to the UPF running in the RiaB NUC machine: +```bash +$ kubectl exec -it upf-0 -n riab -- bash +``` + +On the UPF prompt, we should change the MTU size. +```bash +$ ip l set mtu 1550 dev access +$ ip l set mtu 1550 dev core +``` \ No newline at end of file diff --git a/docs/HW_Installation_ops.md b/docs/HW_Installation_ops.md new file mode 100644 index 0000000..33251c3 --- /dev/null +++ b/docs/HW_Installation_ops.md @@ -0,0 +1,112 @@ +# Operations Guide + +## Evaluate the RIC operation + +In the ONOS-RIC machine, log in the onos-cli pod, running: + +```bash +$ kubectl -n riab exec -ti deployment/onos-cli -- bash +``` + +Once inside the onos-cli pod, check the ONOS-RIC connections and subscriptions: + +```bash +$ onos e2t list connections #Shows the associated CU/DU connection +$ onos e2sub list subscriptions #Shows the apps subscrition to the CU/DU nodes +``` + +In the output of the kpimonv2 list of metrics, there should appear 1 UE registered. It means the UE was attached to the DU/CU setup. + +```bash +$ kubectl exec -it deployment/onos-cli -n riab -- onos kpimonv2 list metrics +Cell ID RRC.ConnEstabAtt.sum RRC.ConnEstabSucc.sum RRC.ConnMax RRC.ConnMean RRC.ConnReEstabAtt.sum +1112066:57344 0 0 0 1 0 +``` + +## Custom Network Routes and IP Addresses + +It is important to explain the custom settings associated with the hardware installation setup, in specific the network routes and IP addresses defined in the EPC-OMEC router and the OAI-CU/DU machine and the cu.onf.conf file. + +In the EPC-OMEC, a router Pod (running the Quagga engine) interconnects the core, enb and access networks, each one respectively in the following subnets 192.168.11.0/29, 192.168.11.8/29, and 192.168.11.16/29. + +Via the definition of the secondary IP address (192.168.11.10/29) in the OAI-CU/DU machine, it was possible to configure the EPC-OMEC core to forward traffic to the host 192.168.11.10 via the gateway 192.168.13.21 (the primary OAI-CU/DU IP address). + +In the OAI-CU/DU machine, the set of routes had to be configured so the traffic from the CU/DU be forwarded to the EPC-OMEC machine. + +Inside the router of the EPC-OMEC, a route had to be configured to reach the secondary IP address of OAI-CU/DU via the enb interface. + +And the cu.onf.conf file in the OAI-CU/DU machine had to be correctly configured using the IP addresses of the MME (EPC-CORE) and RIC machines. + +**Notice, in summary the routing rule and IP addresses configuration are performed so OAI-CU/DU can reach EPC-OMEC and vice-versa.** + + +## User Equipment (UE) Handset +As of now, the current OAI with RiaB setup is running over LTE Band 7. +To communicate with this setup, we should prepare the Android smartphone which supports LTE Band 7. +We should then insert a SIM card to the smartphone, where the SIM card should have the below IMSI, Key, and OPc values: + +* IMSI: `315010999912340-315010999912370` +* Key: `465b5ce8b199b49faa5f0a2ee238a6bc` +* OPc: `69d5c2eb2e2e624750541d3bbc692ba5` + +If we want to use the different IMSI number, we have to change the HSS configuration. +In order to change SIM information in HSS, we first go to the ONOS-RIC machine and open the `sdran-in-a-box-values.yaml` file. +And change this section to the appropriate number: +```yaml + hss: + bootstrap: + users: + - apn: "internet" + key: "000102030405060708090a0b0c0d0e0f" # Change me + opc: "69d5c2eb2e2e624750541d3bbc692ba5" # Change me + sqn: 135 + imsiStart: "315010999912340" # Change me + msisdnStart: "9999334455" + count: 30 +``` + +If the new SIM information has the different PLMN ID, we should also change the PLMN ID into MME, HSS, CU-CP, and DU configuration files. +We should find PLMN ID or MCC/MNC values and change them to the appropriate number. + +`sdran-in-a-box-values.yaml`: +```yaml + spgwc: + pfcp: true + multiUpfs: true + jsonCfgFiles: + subscriber_mapping.json: + subscriber-selection-rules: + - selected-user-plane-profile: "menlo" + keys: + serving-plmn: + mcc: 315 # Change me + mnc: 10 # Change me +... + mme: + cfgFiles: + config.json: + mme: + logging: debug + mcc: + dig1: 3 # Change me + dig2: 1 # Change me + dig3: 5 # Change me + mnc: + dig1: 0 # Change me + dig2: 1 # Change me + dig3: 0 # Change me + apnlist: + internet: "spgwc" +``` + +`cu.onf.conf`: +```text +tracking_area_code = 1001; +plmn_list = ( { mcc = 315; mnc = 010; mnc_length = 3; } ) // Change me +``` + +`du.onf.conf`: +```text +tracking_area_code = 1001; +plmn_list = ( { mcc = 315; mnc = 010; mnc_length = 3; } ) // Change me +``` \ No newline at end of file diff --git a/docs/HW_Installation_prereq.md b/docs/HW_Installation_prereq.md new file mode 100644 index 0000000..5d7d4f5 --- /dev/null +++ b/docs/HW_Installation_prereq.md @@ -0,0 +1,93 @@ +# Prerequisites + +This section explains how to install the requirements needed for the execution of an OAI component in a host machine connected to a USRP. + +In the case of the hardware installation tutorial using 2 NUCs, proceed with the execution of the steps below on both NUCs. + +Before we start this section, we consider the host machine already have Ubuntu 18.04 server OS installed. +**Also, please DO NOT connect the USRP B210 device to the host machines yet.** +**Otherwise, the host machine may not boot up.** + + +## Install Linux Image low-latency + +```bash +$ sudo apt install linux-image-lowlatency linux-headers-lowlatency +``` + +## Power management and CPU frequency configuration +To run on OAI, we must disable p-state and c-state in Linux. +Go to `/etc/default/grub` file and add change `GRUB_CMDLINE_LINUX_DEFAULT` line as below: +```text +GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_pstate=disable processor.max_cstate=1 intel_idle.max_cstate=0 idle=poll" +``` + +After save that file, we should command this: +```bash +$ sudo update-grub2 +``` + +Next, go to `/etc/modprobe.d/blacklist.conf` file and append below at the end of the file: +```text +# for OAI +blacklist intel_powerclamp +``` + +After that, reboot the NUC machine. When rebooting, we have to change the `BIOS` configuration. +Go to the BIOS setup page and change some parameters: +* Disable secure booting option +* Disable hyperthreading +* Enable virtualization +* Disable all power management functions (c-/p-state related) +* Enable real-time tuning and Intel Turbo boost +Once it is done, we should save and exit. Then, we reboot NUC board again. + +When NUC is up and running, we should install the below tool: +```bash +$ sudo apt-get install cpufrequtils +``` + +After the installation, go to `/etc/default/cpufrequtils` and write below: +```text +GOVERNOR="performance" +``` + +*NOTE: If the `/etc/default/cpufrequtils` file does not exist, we should make that file.* + +Next, we should command below: +```bash +$ sudo systemctl disable ondemand.service +$ sudo /etc/init.d/cpufrequtils restart +``` + +After that, we should reboot this machine again. + +## Verification of the power management and CPU frequency configuration +In order to verify configurations for the power management and CPU frequency, we should use `i7z` tool. +```bash +$ sudo apt install i7z +$ sudo i7z +True Frequency (without accounting Turbo) 1607 MHz + CPU Multiplier 16x || Bus clock frequency (BCLK) 100.44 MHz + +Socket [0] - [physical cores=6, logical cores=6, max online cores ever=6] + TURBO ENABLED on 6 Cores, Hyper Threading OFF + Max Frequency without considering Turbo 1707.44 MHz (100.44 x [17]) + Max TURBO Multiplier (if Enabled) with 1/2/3/4/5/6 Cores is 47x/47x/41x/41x/39x/39x + Real Current Frequency 3058.82 MHz [100.44 x 30.45] (Max of below) + Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % Temp VCore + Core 1 [0]: 3058.81 (30.45x) 100 0 0 0 64 0.9698 + Core 2 [1]: 3058.82 (30.45x) 100 0 0 0 63 0.9698 + Core 3 [2]: 3058.82 (30.45x) 100 0 0 0 64 0.9698 + Core 4 [3]: 3058.81 (30.45x) 100 0 0 0 64 0.9698 + Core 5 [4]: 3058.81 (30.45x) 100 0 0 0 65 0.9698 + Core 6 [5]: 3058.82 (30.45x) 100 0 0 0 62 0.9686 +``` + +In the above results, we have to see that all cores should get `C0%` as `100` and `Halt(C1)%` as `0`. +If not, some of the above configuration are missing. +Or, some of BIOS configurations are incorrect. + +**The steps above conclude the installation of OAI/USRP requirements.** + +**Now, please connect the USRP B210 device to the host machines (usb 3.0).** \ No newline at end of file diff --git a/docs/HW_Installation_ric.md b/docs/HW_Installation_ric.md new file mode 100644 index 0000000..ec5125f --- /dev/null +++ b/docs/HW_Installation_ric.md @@ -0,0 +1,46 @@ +# Install the ONOS-RIC + +This section explains how to install the RIC components using RiaB in the ONOS-RIC machine. + +## Start the RiaB ONOS-RIC components + +```bash +$ cd /path/to/sdran-in-a-box +$ sudo apt install build-essential +$ make ric-oai-latest +``` + +## Verify whether everything is up and running +After a while, RiaB Makefile completes to install K8s and deploy ONOS-RIC components. +Once it is done, you can check with the below command in the ONOS-RIC machine. + +```bash +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system atomix-controller-694586d498-6jfll 1/1 Running 0 12m +kube-system cache-storage-controller-5996c8fd45-5pvl4 1/1 Running 0 12m +kube-system calico-kube-controllers-7597dc5bf7-z9czz 1/1 Running 0 14m +kube-system calico-node-sd55n 1/1 Running 0 14m +kube-system config-operator-69f7498fb5-6lcjw 1/1 Running 0 11m +kube-system coredns-dff8fc7d-mskrc 1/1 Running 0 14m +kube-system dns-autoscaler-5d74bb9b8f-ql9rq 1/1 Running 0 14m +kube-system kube-apiserver-node1 1/1 Running 0 13m +kube-system kube-controller-manager-node1 1/1 Running 0 13m +kube-system kube-multus-ds-amd64-qrbcl 1/1 Running 0 14m +kube-system kube-proxy-d8tgv 1/1 Running 0 14m +kube-system kube-scheduler-node1 1/1 Running 0 13m +kube-system kubernetes-dashboard-667c4c65f8-jdm86 1/1 Running 0 14m +kube-system kubernetes-metrics-scraper-54fbb4d595-8r98c 1/1 Running 0 14m +kube-system nodelocaldns-k5p82 1/1 Running 0 14m +kube-system raft-storage-controller-7755865dcd-z68ws 1/1 Running 0 12m +kube-system topo-operator-558f4545bd-n8pbn 1/1 Running 0 11m +riab onos-cli-6655c68cb4-dzg5m 1/1 Running 0 10m +riab onos-config-59884c6766-kgdwd 2/2 Running 0 10m +riab onos-consensus-db-1-0 1/1 Running 0 10m +riab onos-e2sub-7588dcbc7b-tvpkz 1/1 Running 0 10m +riab onos-e2t-56549f6648-bbzxz 1/1 Running 0 10m +riab onos-kpimon-v2-846f556cfb-nd9fs 1/1 Running 0 10m +riab onos-pci-85f465c9cf-gfkp2 1/1 Running 0 10m +riab onos-topo-5df4cf454c-8lrcf 1/1 Running 0 10m +``` + +**Note: RIC does not have a fixed IP address by which oai-enb-cu (or another eNB) can communicate with it. The onos-e2t component exposes a service in port 36421, which is associated with the IP address of the eno1 interface (i.e., the default gateway interface) where it is running. To check that IP address run the command "kubectl -n riab get svc". In the output of this command, one of the lines should show something similar to "onos-e2t-external NodePort x.y.w.z 36421:36421/SCTP 0m40s". The IP address "x.y.w.z" shown in the output of the previous command (listed in the onos-e2t-external service) is the one that is accessible from the outside of the RIC VM, i.e., by the oai-enb-cu in case of this tutorial. In a test case with another eNB, that should be the IP address to be configured in the eNB so it can communicate with onos RIC.** \ No newline at end of file diff --git a/docs/HW_Installation_ric_only.md b/docs/HW_Installation_ric_only.md new file mode 100644 index 0000000..533c298 --- /dev/null +++ b/docs/HW_Installation_ric_only.md @@ -0,0 +1,65 @@ +# Install RIC only + +This section explains how to execute only the RIC components (without RanSim/OAI) using the RiaB Makefile. + +## Get the RiaB source code + +To get the source code, please see: `https://github.com/onosproject/sdran-in-a-box`. + +Since SDRAN-in-a-Box repository is a member-only repository, a user should log in github and then check the git clone command on that web site. +Clone the RiaB repository to the target machine. + +This option is usefull to test RIC with CU/DU components running in other machines. + + +## Run RIC + +In the sdran-in-a-box folder, edit the Makefile to disable the ran-simulator execution, it should look like the line below: + +```bash +RANSIM_ARGS ?= --set import.ran-simulator.enabled=false # Change this value from true to false +``` + +Then run the RIC components with the commands below. + +```bash +$ cd /path/to/sdran-in-a-box +$ sudo apt install build-essential +$ make ric +``` + +**Notice: The sdran-in-a-box-values.yaml contain the latest versions of the RIC components. In order to use the v1.0.0 or v1.1.0 versions make sure to respectively copy and paste to the sdran-in-a-box-values.yaml file the contents of the files sdran-in-a-box-values-v1.0.0.yaml and sdran-in-a-box-values-v1.1.0.yaml as needed.** + +Check the deployed RIC components using the commands: +```bash +$ kubectl -n riab get pods +$ kubectl -n riab get svc +``` + +Notice, in the output of the command `kubectl -n riab get svc` the service onos-e2t-external must be present in order to E2 nodes reach the RIC running node using a remote SCTP connection in port 36421. The IP address to be configured in E2 nodes connecting to RIC must be the IP address of the primary network interface of the RIC host machine. + +If such service (onos-e2t-external) does not exist, make sure in the file sdran-in-a-box-values.yaml the lines below are not commented. + +```yaml +onos-e2t: + service: + external: + enabled: true + e2: + nodePort: 36421 +``` + +## Stop/Clean RIC + +This deletes all deployed Helm charts for RIC components (keeps Kubernetes and Helm installed/running). + +```bash +$ make reset-ric +``` + +And this deletes not only deployed Helm charts but also Kubernetes and Helm. + +```bash +make clean # if we want to keep the ~/helm-charts directory - option to develop/test changed/new Helm charts +make clean-all # if we also want to delete ~/helm-charts directory +``` \ No newline at end of file diff --git a/docs/HW_Installation_troubleshooting.md b/docs/HW_Installation_troubleshooting.md new file mode 100644 index 0000000..0bfb9c8 --- /dev/null +++ b/docs/HW_Installation_troubleshooting.md @@ -0,0 +1,135 @@ +# Troubleshooting Guide + +This section covers how to solve the reported issues. This section will be updated, continuously. + +## SPGW-C or UPF is not working +Please check the log with below commands: +```bash +$ kubectl logs spgwc-0 -n riab -c spgwc # for SPGW-C log +$ kubectl logs upf-0 -n riab -c bess # for UPF log +``` + +In the log, if we can see `unsupported CPU type` or `a specific flag (e.g., AES) is missing`, we should check the CPU microarchitecture. RiaB requires Intel Haswell or more recent CPU microarchitecture. +If we have the appropriate CPU type, we should build SPGW-C or UPF image on the machine where RiaB will run. + +To build SPGW-C, first clone the SPGW-C repository on the machine with `git clone https://github.com/omec-project/spgw`. Then, edit below line in Makefile: +```makefile +DOCKER_BUILD_ARGS ?= --build-arg RTE_MACHINE='native' +``` +Then, run `make` on the `spgw` directory. + +Likewise, for building UPF image, we should clone UPF repository with `git clone https://github.com/omec-project/upf-epc`. Then, edit below line in Makefile: +```makefile +CPU ?= native +``` +Then, run `make` on the `upf-epc` directory. + +After building those images, we should modify overriding value yaml file (i.e., `sdran-in-a-box-values.yaml`). Go to the file and write down below: +```yaml +images: + tags: + spgwc: + bess: + pfcpiface: + pullPolicy: IfNotPresent +``` +Then, run below commands: +```bash +$ cd /path/to/sdran-in-a-box +$ make reset-test +# after all OMEC pods are deleted, run make again +$ make +``` + +## ETCD is not working +Sometimes, we see the below outputs when building RiaB. +```text +TASK [etcd : Configure | Ensure etcd is running] *********************************************************************** +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (4 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (3 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (2 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (1 retries left). +``` + +If we see this, we can command below: +```bash +$ sudo systemctl restart docker +$ cd /path/to/sdran-in-a-box +$ make +``` + +## Atomix controllers cannot be deleted/reset +Sometimes, Atomix controllers cannot be deleted (maybe we will get stuck when deleting Atomix controller pods) when we command `make reset-test`. +```bash +rm -f /tmp/build/milestones/oai-enb-cu +rm -f /tmp/build/milestones/oai-enb-du +rm -f /tmp/build/milestones/oai-ue +helm delete -n riab sd-ran || true +release "sd-ran" uninstalled +cd /tmp/build/milestones; rm -f ric +kubectl delete -f https://raw.githubusercontent.com/atomix/kubernetes-controller/master/deploy/atomix-controller.yaml || true +customresourcedefinition.apiextensions.k8s.io "databases.cloud.atomix.io" deleted +customresourcedefinition.apiextensions.k8s.io "partitions.cloud.atomix.io" deleted +customresourcedefinition.apiextensions.k8s.io "members.cloud.atomix.io" deleted +customresourcedefinition.apiextensions.k8s.io "primitives.cloud.atomix.io" deleted +serviceaccount "atomix-controller" deleted +clusterrole.rbac.authorization.k8s.io "atomix-controller" deleted +clusterrolebinding.rbac.authorization.k8s.io "atomix-controller" deleted +service "atomix-controller" deleted +deployment.apps "atomix-controller" deleted +``` + +If the script is stopped here, we can command: +```bash +# Commmand Ctrl+c first to stop the Makefile script if the make reset-test is got stuck. Then command below. +$ make reset-atomix # Manually delete Atomix controller pods +$ make atomix # Manually install Atomix controller pods +$ make reset-test # Then, make reset-test again +``` + +Or, sometimes we see this when deploying RiaB: +```text +Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/atomix/kubernetes-controller/master/deploy/atomix-controller.yaml": object is being deleted: customresourcedefinitions.apiextensions.k8s.io "members.cloud.atomix.io" already exists +Makefile:231: recipe for target '/tmp/build/milestones/atomix' failed +``` + +In this case, we can manually delete atomix with the command `make atomix || make reset-atomix`, and then resume to deploy RiaB. + +## Pod onos-consensus-db-1-0 initialization failed + +In Ubuntu 20.04 (kernel 5.4.0-65-generic), the k8s pod named `onos-consensus-db-1-0` might fail due to a bug of using go and alpine together (e.g., https://github.com/docker-library/golang/issues/320). + +It can be seen in `kubectl logs -n riab onos-consensus-db-1-0` as: +```bash +runtime: mlock of signal stack failed: 12 +runtime: increase the mlock limit (ulimit -l) or +runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ +fatal error: mlock failed +``` + +Such pod utilizes the docker image atomix/raft-storage-node:v0.5.3, tagged from the build of the image atomix/dragonboat-raft-storage-node:latest available at https://github.com/atomix/dragonboat-raft-storage-node. + +A quick fix (allowing an unlimited amount memory to be locked by the pod) to this issue is cloning the repository https://github.com/atomix/dragonboat-raft-storage-node, and changing the Makefile: + +```bash +# Before change +image: build + docker build . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} + +# After change: unlimited maximum locked-in-memory address space +image: build + docker build --ulimit memlock=-1 . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} +``` + +Then running in the source dir of this repository the command `make image`, and tagging the built image as: + +```bash +docker tag atomix/dragonboat-raft-storage-node:latest atomix/raft-storage-node:v0.5.3 +``` + +After that proceed with the execution of the Riab setup again. + + +## Other issues? +Please contact ONF SD-RAN team, if you see any issue. Any issue report from users is very welcome. +Mostly, the redeployment by using `make reset-test and make [option]` resolves issues. \ No newline at end of file diff --git a/docs/Installation_OAI_nFAPI.md b/docs/Installation_OAI_nFAPI.md new file mode 100644 index 0000000..4779df3 --- /dev/null +++ b/docs/Installation_OAI_nFAPI.md @@ -0,0 +1,237 @@ +# Installation with CU-CP and OAI nFAPI emulator +This document covers how to install ONOS RIC services with CU-CP and OAI nFAPI emulator. +With this option, RiaB will deploy ONOS RIC services including ONOS-KPIMON-V2 (KPM 2.0 supported) and ONOS-PCI xAPPs together with CU-CP, OAI DU (nFAPI), and OAI UE (nFAPI). + +## Clone this repository +To begin with, clone this repository: +```bash +$ git clone https://github.com/onosproject/sdran-in-a-box +``` +**NOTE: If we want to use a specific release, we can change the branch with `git checkout [args]` command:** +```bash +$ cd /path/to/sdran-in-a-box +$ git checkout v1.0.0 # for release 1.0 +$ git checkout v1.1.0 # for release 1.1 +$ git checkout master # for master +``` + +## Deploy RiaB with OAI nFAPI emulator + +### Command options +To deploy RiaB with OAI nFAPI emulator, we should go to `sdran-in-a-box` directory and command below: +```bash +$ cd /path/to/sdran-in-a-box +# type one of below commands +# for "latest" version +$ make # make riab-oai, or make riab-oai-latest +# for "master-stable" version +$ make riab-oai-master-stable +# for a specific version +$ make riab-oai-v1.0.0 # for release SD-RAN 1.0 +$ make riab-oai-v1.1.0 # for release SD-RAN 1.1 +# for a "dev" version +$ make riab-oai-dev +``` + +Once we push one of above commands, the deployment procedure starts. + +### Credentials +In the deployment procedure, we should type some credentials on the prompt: +* OpenCORD username and HTTPS key +* GitHub username and password +* Aether/SD-RAN Helm chart repository credentials + +```bash +aether-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires HTTPS key +Cloning into '/users/wkim/helm-charts/aether-helm-charts'... +Username for 'https://gerrit.opencord.org': +Password for 'https://@gerrit.opencord.org': +remote: Total 1103 (delta 0), reused 1103 (delta 0) +Receiving objects: 100% (1103/1103), 526.14 KiB | 5.31 MiB/s, done. +Resolving deltas: 100% (604/604), done. +sdran-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires Github credential +Cloning into '/users/wkim/helm-charts/sdran-helm-charts'... +Username for 'https://github.com': +Password for 'https://@github.com': +remote: Enumerating objects: 19, done. +remote: Counting objects: 100% (19/19), done. +remote: Compressing objects: 100% (17/17), done. +remote: Total 2259 (delta 7), reused 3 (delta 2), pack-reused 2240 +Receiving objects: 100% (2259/2259), 559.35 KiB | 2.60 MiB/s, done. +Resolving deltas: 100% (1558/1558), done. + +..... + +helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ +"incubator" has been added to your repositories +helm repo add cord https://charts.opencord.org +"cord" has been added to your repositories +Username for ONF SDRAN private chart: +Password for ONF SDRAN private chart: +"sdran" has been added to your repositories +touch /tmp/build/milestones/helm-ready +``` + +If we don't see any error or failure messages, everything is deployed. +```bash +$ kubectl get po --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +default router 1/1 Running 0 1h +kube-system atomix-controller-694586d498-mj2kr 1/1 Running 0 1h +kube-system cache-storage-controller-5996c8fd45-z6pgr 1/1 Running 0 1h +kube-system calico-kube-controllers-86d8c59b9f-vpztg 1/1 Running 0 1h +kube-system calico-node-jxqp6 1/1 Running 0 1h +kube-system config-operator-69f7498fb5-4gzzp 1/1 Running 0 1h +kube-system coredns-dff8fc7d-9fw8d 1/1 Running 0 1h +kube-system dns-autoscaler-5d74bb9b8f-6mvnv 1/1 Running 0 1h +kube-system kube-apiserver-node1 1/1 Running 0 1h +kube-system kube-controller-manager-node1 1/1 Running 0 1h +kube-system kube-multus-ds-amd64-d6v5q 1/1 Running 0 1h +kube-system kube-proxy-rsfvr 1/1 Running 0 1h +kube-system kube-scheduler-node1 1/1 Running 0 1h +kube-system kubernetes-dashboard-667c4c65f8-cdcqz 1/1 Running 0 1h +kube-system kubernetes-metrics-scraper-54fbb4d595-ffd4g 1/1 Running 0 1h +kube-system nodelocaldns-lbhf2 1/1 Running 0 1h +kube-system raft-storage-controller-7755865dcd-j44kq 1/1 Running 0 1h +kube-system topo-operator-558f4545bd-7ncr7 1/1 Running 0 1h +riab cassandra-0 1/1 Running 0 1h +riab hss-0 1/1 Running 0 1h +riab mme-0 4/4 Running 0 1h +riab oai-enb-cu-0 1/1 Running 0 1h +riab oai-enb-du-0 1/1 Running 0 1h +riab oai-ue-0 1/1 Running 0 1h +riab onos-cli-6655c68cb4-jhprs 1/1 Running 0 1h +riab onos-config-59884c6766-r5ckf 2/2 Running 0 1h +riab onos-consensus-db-1-0 1/1 Running 0 1h +riab onos-e2sub-7588dcbc7b-qc8xk 1/1 Running 0 1h +riab onos-e2t-56549f6648-kjrh4 1/1 Running 0 1h +riab onos-kpimon-v2-846f556cfb-z5x88 1/1 Running 0 1h +riab onos-pci-85f465c9cf-qfp29 1/1 Running 0 1h +riab onos-topo-5df4cf454c-dr5bf 1/1 Running 0 1h +riab pcrf-0 1/1 Running 0 1h +riab spgwc-0 2/2 Running 0 1h +riab upf-0 4/4 Running 0 1h +``` + +NOTE: If we see any issue when deploying RiaB, please check [Troubleshooting](./troubleshooting.md) + +## End-to-End (E2E) tests for verification +In order to check whether everything is running, we should conduct some E2E tests and check their results. +Since CU-CP supports the SD-RAN control plane and OAI nFAPI services the LTE user plane, we can do E2E tests on the user plane and SD-RAN control plane. + +### The E2E test on the user plane +We can type `make test-user-plane` on the prompt for the user plane verification. + +```bash +$ make test-user-plane +*** T1: Internal network test: ping 192.168.250.1 (Internal router IP) *** +PING 192.168.250.1 (192.168.250.1) from 172.250.255.254 oaitun_ue1: 56(84) bytes of data. +64 bytes from 192.168.250.1: icmp_seq=1 ttl=64 time=20.4 ms +64 bytes from 192.168.250.1: icmp_seq=2 ttl=64 time=19.9 ms +64 bytes from 192.168.250.1: icmp_seq=3 ttl=64 time=18.6 ms + +--- 192.168.250.1 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2001ms +rtt min/avg/max/mdev = 18.664/19.686/20.479/0.758 ms +*** T2: Internet connectivity test: ping to 8.8.8.8 *** +PING 8.8.8.8 (8.8.8.8) from 172.250.255.254 oaitun_ue1: 56(84) bytes of data. +64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=57.8 ms +64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=56.4 ms +64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=55.0 ms + +--- 8.8.8.8 ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2002ms +rtt min/avg/max/mdev = 55.004/56.441/57.878/1.189 ms +*** T3: DNS test: ping to google.com *** +PING google.com (172.217.9.142) from 172.250.255.254 oaitun_ue1: 56(84) bytes of data. +64 bytes from dfw25s26-in-f14.1e100.net (172.217.9.142): icmp_seq=1 ttl=112 time=54.7 ms +64 bytes from dfw25s26-in-f14.1e100.net (172.217.9.142): icmp_seq=2 ttl=112 time=53.6 ms +64 bytes from dfw25s26-in-f14.1e100.net (172.217.9.142): icmp_seq=3 ttl=112 time=51.9 ms + +--- google.com ping statistics --- +3 packets transmitted, 3 received, 0% packet loss, time 2002ms +rtt min/avg/max/mdev = 51.990/53.452/54.712/1.136 ms +``` + +If we can see that ping is working without any loss, the user plane is working well. + +### The E2E test on SD-RAN control plane +In order to verify the SD-RAN control plane, we should command below for each version. +* `make test-kpimon`: only for SD-RAN release 1.0 +```bash +$ make test-kpimon +*** Get KPIMON result through CLI *** +Key[PLMNID, nodeID] num(Active UEs) +{eNB-CU-Eurecom-LTEBox [0 2 16] 57344} 1 +``` +* `make test-kpimon-v2`: for SD-RAN release 1.1, master-stable, latest, and dev versions +```bash +$ make test-kpimon-v2 +*** Get KPIMON result through CLI *** +Cell ID RRC.ConnEstabAtt.sum RRC.ConnEstabSucc.sum RRC.ConnMax RRC.ConnMean RRC.ConnReEstabAtt.sum +1112066:57344 0 0 0 1 0 +``` + +* `make detach-ue && make test-kpimon` for SD-RAN release 1.0; `detach-ue` detaches a UE so the number of active UEs decreases +```bash +$ make test-kpimon +*** Get KPIMON result through CLI *** +Key[PLMNID, nodeID] num(Active UEs) +{eNB-CU-Eurecom-LTEBox [0 2 16] 57344} 1 +$ make detach-ue +echo -en "AT+CPIN=0000\r" | nc -u -w 1 localhost 10000 + +OK +echo -en "AT+CGATT=0\r" | nc -u -w 1 localhost 10000 + +OK +$ make test-kpimon +*** Get KPIMON result through CLI *** +Key[PLMNID, nodeID] num(Active UEs) +{eNB-CU-Eurecom-LTEBox [0 2 16] 57344} 0 +``` + +* `make detach-ue && make test-kpimon-v2`: for SD-RAN release 1.1, master-stable, latest, and dev versions; `detach-ue` detaches a UE so the number of active UEs decreases +```bash +$ make test-kpimon-v2 +*** Get KPIMON result through CLI *** +Cell ID RRC.ConnEstabAtt.sum RRC.ConnEstabSucc.sum RRC.ConnMax RRC.ConnMean RRC.ConnReEstabAtt.sum +1112066:57344 0 0 0 1 0 +$ make detach-ue +echo -en "AT+CPIN=0000\r" | nc -u -w 1 localhost 10000 + +OK +echo -en "AT+CGATT=0\r" | nc -u -w 1 localhost 10000 + +OK +$ make test-kpimon-v2 +*** Get KPIMON result through CLI *** +Cell ID RRC.ConnEstabAtt.sum RRC.ConnEstabSucc.sum RRC.ConnMax RRC.ConnMean RRC.ConnReEstabAtt.sum +1112066:57344 0 0 0 0 0 +``` + +If we can see like the above code block, the SD-RAN cotrol plane is working fine. + +NOTE 1: If we enable ONOS-KPIMON-V1 in the `sdran-in-a-box-values-.yaml`, we can also use `make test-kpimon-v1` in SD-RAN release 1.1, master-stable, latest, and dev version. + +NOTE 2: Currently, there is no way to reattach the detached UE. For reattachment, we should redeploy RiaB (at least ONOS RIC services, CU-CP, OAI nFAPI emulator). + +NOTE 3: Even though RiaB deploys ONOS-PCI xAPP, `make test-pci` does not print anything. The reason is that CU-CP does not support the RC-PRE service model. + +## Other commands +### Reset and delete RiaB environment +If we want to reset our RiaB environment or delete RiaB compoents, we can use below commands: +* `make reset-test`: It deletes ONOS RIC services, CU-CP, and OAI nFAPI emulator but Kubernetes is still running +* `make clean`: It just deletes Kubernets environment; Eventually, all ONOS RIC services, CU-CP, and OAI nFAPI emulator are terminated; The Helm chart directory is not deleted +* `make clean-all`: It deletes all including Kubernetes environment, all componentes/PODs which RiaB deployed, and even the Helm chart directory + +### Deploy or reset a chart/service +If we want to only deploy or reset a chart/service, we can use below command: +* `make omec`: It deploys OMEC chart +* `make reset-omec`: It deletes OMEC chart +* `make atomix`: It deploys Atomix controllers +* `make reset-atomix`: It deletes Atomix controllers +* `make ric`: It deploys ONOS RIC services +* `make reset-ric`: It deletes ONOS RIC services +* `make oai`: It deploys CU-CP and OAI nFAPI emulator +* `make reset-oai`: It deletes CU-CP and OAI nFAPI emulator diff --git a/docs/Installation_RANSim_FBAH.md b/docs/Installation_RANSim_FBAH.md new file mode 100644 index 0000000..2880d2e --- /dev/null +++ b/docs/Installation_RANSim_FBAH.md @@ -0,0 +1,158 @@ +# Installation with RAN-Simulator and Facebook-AirHop xAPP +This document covers how to install ONOS RIC services with RAN-Simulator and Facebook-Airhop xAPP. +With this option, RiaB will deploy ONOS RIC services including ONOS-KPIMON-V2 (KPM 2.0 supported) together with RAN-Simulator and Facebook-AirHop xAPP. + +## Clone this repository +To begin with, clone this repository: +```bash +$ git clone https://github.com/onosproject/sdran-in-a-box +``` +**NOTE: If we want to use a specific release, we can change the branch with `git checkout [args]` command:** +```bash +$ cd /path/to/sdran-in-a-box +$ git checkout v1.0.0 # for release 1.0 +$ git checkout v1.1.0 # for release 1.1 +$ git checkout master # for master +``` + +## Deploy RiaB with RAN-Simulator and Facebook-AirHop xAPP +To deploy RiaB with RAN-Simulator and Facebook-AirHop xAPP, we should go to `sdran-in-a-box` directory and command below: +```bash +$ cd /path/to/sdran-in-a-box +# type one of below commands +# for "latest" version +$ make-fbah # or make riab-fbah-latest +# for "master-stable" version +$ make riab-fbah-master-stable +# for a specific version +$ make riab-fbah-v1.1.0 # for release SD-RAN 1.1 +# for a "dev" version +$ make riab-fbah-dev +``` + +Once we push one of above commands, the deployment procedure starts. + +### Credentials +In the deployment procedure, we should type some credentials on the prompt: +* OpenCORD username and HTTPS key +* GitHub username and password +* Aether/SD-RAN Helm chart repository credentials + +```bash +aether-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires HTTPS key +Cloning into '/users/wkim/helm-charts/aether-helm-charts'... +Username for 'https://gerrit.opencord.org': +Password for 'https://@gerrit.opencord.org': +remote: Total 1103 (delta 0), reused 1103 (delta 0) +Receiving objects: 100% (1103/1103), 526.14 KiB | 5.31 MiB/s, done. +Resolving deltas: 100% (604/604), done. +sdran-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires Github credential +Cloning into '/users/wkim/helm-charts/sdran-helm-charts'... +Username for 'https://github.com': +Password for 'https://@github.com': +remote: Enumerating objects: 19, done. +remote: Counting objects: 100% (19/19), done. +remote: Compressing objects: 100% (17/17), done. +remote: Total 2259 (delta 7), reused 3 (delta 2), pack-reused 2240 +Receiving objects: 100% (2259/2259), 559.35 KiB | 2.60 MiB/s, done. +Resolving deltas: 100% (1558/1558), done. + +..... + +helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ +"incubator" has been added to your repositories +helm repo add cord https://charts.opencord.org +"cord" has been added to your repositories +Username for ONF SDRAN private chart: +Password for ONF SDRAN private chart: +"sdran" has been added to your repositories +touch /tmp/build/milestones/helm-ready +``` + +If we don't see any error or failure messages, everything is deployed. +```bash +$ kubectl get po --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +default router 1/1 Running 0 44h +kube-system atomix-controller-694586d498-pndsn 1/1 Running 0 96s +kube-system cache-storage-controller-5996c8fd45-wc8hz 1/1 Running 0 95s +kube-system calico-kube-controllers-86d8c59b9f-vpztg 1/1 Running 0 45h +kube-system calico-node-jxqp6 1/1 Running 0 45h +kube-system config-operator-69f7498fb5-nglg2 1/1 Running 0 95s +kube-system coredns-dff8fc7d-9fw8d 1/1 Running 0 45h +kube-system dns-autoscaler-5d74bb9b8f-6mvnv 1/1 Running 0 45h +kube-system kube-apiserver-node1 1/1 Running 0 45h +kube-system kube-controller-manager-node1 1/1 Running 0 45h +kube-system kube-multus-ds-amd64-d6v5q 1/1 Running 0 45h +kube-system kube-proxy-rsfvr 1/1 Running 0 45h +kube-system kube-scheduler-node1 1/1 Running 0 45h +kube-system kubernetes-dashboard-667c4c65f8-cdcqz 1/1 Running 0 45h +kube-system kubernetes-metrics-scraper-54fbb4d595-ffd4g 1/1 Running 0 45h +kube-system nodelocaldns-lbhf2 1/1 Running 0 45h +kube-system raft-storage-controller-7755865dcd-bdjkf 1/1 Running 0 96s +kube-system topo-operator-558f4545bd-h4pds 1/1 Running 0 94s +riab ah-eson-test-server-67495c98d8-8vqth 1/1 Running 0 71s +riab fb-ah-gui-9dc8c86dc-zpmw5 1/1 Running 0 71s +riab fb-ah-xapp-7c4f7466fb-n5qg4 1/1 Running 0 71s +riab onos-cli-6655c68cb4-l8f76 1/1 Running 0 71s +riab onos-config-59884c6766-d9v4z 2/2 Running 0 71s +riab onos-consensus-db-1-0 1/1 Running 0 71s +riab onos-e2sub-7588dcbc7b-rnvdk 1/1 Running 0 71s +riab onos-e2t-56549f6648-d9zsf 1/1 Running 0 70s +riab onos-kpimon-v2-846f556cfb-pgrcf 1/1 Running 0 71s +riab onos-pci-85f465c9cf-szlgw 1/1 Running 0 71s +riab onos-topo-5df4cf454c-hj7sz 1/1 Running 0 71s +riab ran-simulator-b6754dc97-f679m 1/1 Running 0 71s +``` + +NOTE: If we see any issue when deploying RiaB, please check [Troubleshooting](./troubleshooting.md) + +## End-to-End (E2E) tests for verification +In order to check whether everything is running, we should conduct some E2E tests and check their results. +Since RAN-Sim does only generate SD-RAN control messages, we can run E2E tests on the SD-RAN control plane. + +### The E2E test on SD-RAN control plane +* `make test-kpimon-v2`: for SD-RAN release 1.1, master-stable, latest, and dev versions +```bash +$ make test-kpimon-v2 +*** Get KPIMON result through CLI *** +Cell ID RRC.Conn.Avg RRC.Conn.Max RRC.ConnEstabAtt.Tot RRC.ConnEstabSucc.Tot RRC.ConnReEstabAtt.HOFail RRC.ConnReEstabAtt.Other RRC.ConnReEstabAtt.Tot RRC.ConnReEstabAtt.reconfigFail +1279014:5153 5 5 0 0 0 0 0 0 +1279014:5154 5 5 0 0 0 0 0 0 +``` + +* Use Facebook-AirHop GUI page: for SD-RAN release 1.1, master-stable, latest, and dev versions + +To access GUI, we should open web browser like [Chrome](https://www.google.com/chrome/) or [Safari](https://www.apple.com/safari/). +Next, go to `http://:30095` +Then, we can see the xAPP webpage. + +![FBAH WEB GUI](./figures/fbah-no-map.png) + +On this page, we can see the `Cells` table which shows ECGI, PCI, and each cell's neighbor cells. + +If we want to see the Google Map View, we should make a SSH tunnel from our local machine to the RiaB server with below command: +```bash +$ ssh @ -L "*:8080::30095" +``` +After that, go to `http://localhost:8080` on the web browser. + +![FBAH WEB GUI](./figures/fbah-with-map.png) + +Since the Google Map API only allows us to use the url `localhost:8080` to show Google Map view, we should make the SSH tunnel. + +NOTE 1: Of course, all other port forwarding should work as long as we can access the GUI with `localhost:8080` URL. + +## Other commands +### Reset and delete RiaB environment +If we want to reset our RiaB environment or delete RiaB compoents, we can use below commands: +* `make reset-test`: It deletes ONOS RIC services and RAN-Simulator but Kubernetes is still running +* `make clean`: It just deletes Kubernets environment; Eventually, all ONOS RIC and RAN-Simulator are terminated; The Helm chart directory is not deleted +* `make clean-all`: It deletes all including Kubernetes environment, all componentes/PODs which RiaB deployed, and even the Helm chart directory + +### Deploy or reset a chart/service +If we want to only deploy or reset a chart/service, we can use below command: +* `make atomix`: It deploys Atomix controllers +* `make reset-atomix`: It deletes Atomix controllers +* `make ric`: It deploys ONOS RIC services +* `make reset-ric`: It deletes ONOS RIC services \ No newline at end of file diff --git a/docs/Installation_RANSim_PCI.md b/docs/Installation_RANSim_PCI.md new file mode 100644 index 0000000..73b2197 --- /dev/null +++ b/docs/Installation_RANSim_PCI.md @@ -0,0 +1,164 @@ +# Installation with RAN-Simulator and ONOS PCI xAPP +This document covers how to install ONOS RIC services with RAN-Simulator. +With this option, RiaB will deploy ONOS RIC services including ONOS-KPIMON-V2 (KPM 2.0 supported) and ONOS-PCI xAPPs together with RAN-Simulator + +## Clone this repository +To begin with, clone this repository: +```bash +$ git clone https://github.com/onosproject/sdran-in-a-box +``` +**NOTE: If we want to use a specific release, we can change the branch with `git checkout [args]` command:** +```bash +$ cd /path/to/sdran-in-a-box +$ git checkout v1.0.0 # for release 1.0 +$ git checkout v1.1.0 # for release 1.1 +$ git checkout master # for master +``` + +## Deploy RiaB with RAN-Simulator +### Command options +To deploy RiaB with RAN-Simulator, we should go to `sdran-in-a-box` directory and command below: +```bash +$ cd /path/to/sdran-in-a-box +# type one of below commands +# for "latest" version +$ make-ransim # or make riab-ransim-latest +# for "master-stable" version +$ make riab-ransim-master-stable +# for a specific version +$ make riab-ransim-v1.0.0 # for release SD-RAN 1.0 +$ make riab-ransim-v1.1.0 # for release SD-RAN 1.1 +# for a "dev" version +$ make riab-ransim-dev +``` + +Once we push one of above commands, the deployment procedure starts. + +### Credentials +In the deployment procedure, we should type some credentials on the prompt: +* OpenCORD username and HTTPS key +* GitHub username and password +* Aether/SD-RAN Helm chart repository credentials + +```bash +aether-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires HTTPS key +Cloning into '/users/wkim/helm-charts/aether-helm-charts'... +Username for 'https://gerrit.opencord.org': +Password for 'https://@gerrit.opencord.org': +remote: Total 1103 (delta 0), reused 1103 (delta 0) +Receiving objects: 100% (1103/1103), 526.14 KiB | 5.31 MiB/s, done. +Resolving deltas: 100% (604/604), done. +sdran-helm-chart repo is not in /users/wkim/helm-charts directory. Start to clone - it requires Github credential +Cloning into '/users/wkim/helm-charts/sdran-helm-charts'... +Username for 'https://github.com': +Password for 'https://@github.com': +remote: Enumerating objects: 19, done. +remote: Counting objects: 100% (19/19), done. +remote: Compressing objects: 100% (17/17), done. +remote: Total 2259 (delta 7), reused 3 (delta 2), pack-reused 2240 +Receiving objects: 100% (2259/2259), 559.35 KiB | 2.60 MiB/s, done. +Resolving deltas: 100% (1558/1558), done. + +..... + +helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ +"incubator" has been added to your repositories +helm repo add cord https://charts.opencord.org +"cord" has been added to your repositories +Username for ONF SDRAN private chart: +Password for ONF SDRAN private chart: +"sdran" has been added to your repositories +touch /tmp/build/milestones/helm-ready +``` + +If we don't see any error or failure messages, everything is deployed. +```bash +$ kubectl get po --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +default router 1/1 Running 0 44h +kube-system atomix-controller-694586d498-m55lk 1/1 Running 0 90s +kube-system cache-storage-controller-5996c8fd45-rq7tv 1/1 Running 0 89s +kube-system calico-kube-controllers-86d8c59b9f-vpztg 1/1 Running 0 45h +kube-system calico-node-jxqp6 1/1 Running 0 45h +kube-system config-operator-69f7498fb5-zwxk8 1/1 Running 0 88s +kube-system coredns-dff8fc7d-9fw8d 1/1 Running 0 45h +kube-system dns-autoscaler-5d74bb9b8f-6mvnv 1/1 Running 0 45h +kube-system kube-apiserver-node1 1/1 Running 0 45h +kube-system kube-controller-manager-node1 1/1 Running 0 45h +kube-system kube-multus-ds-amd64-d6v5q 1/1 Running 0 45h +kube-system kube-proxy-rsfvr 1/1 Running 0 45h +kube-system kube-scheduler-node1 1/1 Running 0 45h +kube-system kubernetes-dashboard-667c4c65f8-cdcqz 1/1 Running 0 45h +kube-system kubernetes-metrics-scraper-54fbb4d595-ffd4g 1/1 Running 0 45h +kube-system nodelocaldns-lbhf2 1/1 Running 0 45h +kube-system raft-storage-controller-7755865dcd-mhjvh 1/1 Running 0 89s +kube-system topo-operator-558f4545bd-jqjxm 1/1 Running 0 87s +riab onos-cli-6655c68cb4-gjgx5 1/1 Running 0 57s +riab onos-config-59884c6766-dqcnh 2/2 Running 0 57s +riab onos-consensus-db-1-0 1/1 Running 0 57s +riab onos-e2sub-7588dcbc7b-wgft7 1/1 Running 0 57s +riab onos-e2t-56549f6648-24227 1/1 Running 0 57s +riab onos-kpimon-v2-846f556cfb-x7gsz 1/1 Running 0 57s +riab onos-pci-85f465c9cf-5z8b9 1/1 Running 0 57s +riab onos-topo-5df4cf454c-dqvsp 1/1 Running 0 57s +riab ran-simulator-b6754dc97-czgvx 1/1 Running 0 57s +``` + +NOTE: If we see any issue when deploying RiaB, please check [Troubleshooting](./troubleshooting.md) + +## End-to-End (E2E) tests for verification +In order to check whether everything is running, we should conduct some E2E tests and check their results. +Since RAN-Sim does only generate SD-RAN control messages, we can run E2E tests on the SD-RAN control plane. + +### The E2E test on SD-RAN control plane +In order to verify the SD-RAN control plane, we should command below for each version. +* `make test-kpimon`: only for SD-RAN release 1.0 +```bash +$ make test-kpimon +*** Get KPIMON result through CLI *** +Key[PLMNID, nodeID] num(Active UEs) +{eNB-CU-Eurecom-LTEBox [0 2 16] 5153} 1 +{eNB-CU-Eurecom-LTEBox [0 2 16] 5154} 1 +``` + +* `make test-kpimon-v2`: for SD-RAN release 1.1, master-stable, latest, and dev versions +```bash +$ make test-kpimon-v2 +*** Get KPIMON result through CLI *** +Cell ID RRC.Conn.Avg RRC.Conn.Max RRC.ConnEstabAtt.Tot RRC.ConnEstabSucc.Tot RRC.ConnReEstabAtt.HOFail RRC.ConnReEstabAtt.Other RRC.ConnReEstabAtt.Tot RRC.ConnReEstabAtt.reconfigFail +1279014:5153 5 5 0 0 0 0 0 0 +1279014:5154 5 5 0 0 0 0 0 0 +``` + +* `make test-pci`: for SD-RAN release 1.1, master-stable, latest, and dev versions +```bash +$ make test-pci +*** Get PCI result through CLI *** +ID num(conflicts) +343332707639554 1 +343332707639811 0 +343332707639810 0 +343332707639809 1 +343332707639555 1 +343332707639553 3 +``` + +If we can see like the above code block, the SD-RAN cotrol plane is working fine. + +NOTE 1: Since SD-RAN release 1.0 does not support RC-PRE service model, `make test-pci` is not working in RiaB 1.0.0 version. + +NOTE 2: The result `make test-pci` may vary but at least IDs `343332707639554`, `343332707639809`, `343332707639555`, and `343332707639553` should have one or more `num(conflicts)`. + +## Other commands +### Reset and delete RiaB environment +If we want to reset our RiaB environment or delete RiaB compoents, we can use below commands: +* `make reset-test`: It deletes ONOS RIC services and RAN-Simulator but Kubernetes is still running +* `make clean`: It just deletes Kubernets environment; Eventually, all ONOS RIC and RAN-Simulator are terminated; The Helm chart directory is not deleted +* `make clean-all`: It deletes all including Kubernetes environment, all componentes/PODs which RiaB deployed, and even the Helm chart directory + +### Deploy or reset a chart/service +If we want to only deploy or reset a chart/service, we can use below command: +* `make atomix`: It deploys Atomix controllers +* `make reset-atomix`: It deletes Atomix controllers +* `make ric`: It deploys ONOS RIC services +* `make reset-ric`: It deletes ONOS RIC services \ No newline at end of file diff --git a/docs/RiaB_Prerequisites.md b/docs/RiaB_Prerequisites.md new file mode 100644 index 0000000..0155af2 --- /dev/null +++ b/docs/RiaB_Prerequisites.md @@ -0,0 +1,15 @@ +# Prerequisites + +## Supported machines and OS +In order to run RiaB, we should prepare at least a single machine which has the below minimum requirements. + +* CloudLab Wisconsin and Utah clusters + * CPU: Intel CPU and Haswell microarchitecture or beyond; at least 4 cores + * OS: Ubuntu 18.04 (e.g., OnePC-Ubuntu18.04 profile in CloudLab) + * RAM: At least 16GB + * Storage: At least 50GB (recommendation: 100GB) +* Any baremetal server, VM, or public cloud VM (e.g., Amazon AWS, MS Azure, Google Cloud, etc.) + * CPU: Intel CPU and Haswell microarchitecture or beyond; at least 4 cores + * OS: Ubuntu 18.04 (under test with Ubuntu 20.04) + * RAM: At least 16GB + * Storage: At least 50GB (recommendation: 100GB) diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 0000000..b780c62 --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,113 @@ +# Troubleshooting +This section covers how to solve the reported issues. This section will be updated, continuously. + +## Helm charts are out-of-date +If we want to update Helm chart up-to-date, we can fetch all charts to `~/helm-charts/aether-helm-charts` and `~/helm-charts/sdran-helm-charts` directories. +For the development perspective, sometimes we need to fetch the latest Helm chart commits, although the RiaB uses a specific chart version. This command fetches all latest commits: +```bash +$ make fetch-all-charts +``` +It just fetches the all latest commits, i.e., it does not change/checkout the specific branch/commit. + +NOTE: It may request credentials for the OpenCORD gerrit and SD-RAN Github. + +## SPGW-C or UPF is not working +Please check the log with below commands: +```bash +$ kubectl logs spgwc-0 -n riab -c spgwc # for SPGW-C log +$ kubectl logs upf-0 -n riab -c bess # for UPF log +``` + +In the log, if we can see `unsupported CPU type` or `a specific flag (e.g., AES) is missing`, we should check the CPU microarchitecture. RiaB requires Intel Haswell or more recent CPU microarchitecture. +If we have the appropriate CPU type, we should build SPGW-C or UPF image on the machine where RiaB will run. + +To build SPGW-C, first clone the SPGW-C repository on the machine with `git clone https://github.com/omec-project/spgw`. Then, edit below line in Makefile: +```makefile +DOCKER_BUILD_ARGS ?= --build-arg RTE_MACHINE='native' +``` +Then, run `make` on the `spgw` directory. + +Likewise, for building UPF image, we should clone UPF repository with `git clone https://github.com/omec-project/upf-epc`. Then, edit below line in Makefile: +```makefile +CPU ?= native +``` +Then, run `make` on the `upf-epc` directory. + +After building those images, we should modify overriding value yaml file (i.e., `sdran-in-a-box-values.yaml`). Go to the file and write down below: +```yaml +images: + tags: + spgwc: + bess: + pfcpiface: + pullPolicy: IfNotPresent +``` +Then, run below commands: +```bash +$ cd /path/to/sdran-in-a-box +$ make reset-test +# after all OMEC pods are deleted, run make again +$ make +``` + +## ETCD is not working +Sometimes, we see the below outputs when building RiaB. +```text +TASK [etcd : Configure | Ensure etcd is running] *********************************************************************** +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (4 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (3 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (2 retries left). +FAILED - RETRYING: Configure | Check if etcd cluster is healthy (1 retries left). +``` + +If we see this, we can command below: +```bash +$ sudo systemctl restart docker +$ cd /path/to/sdran-in-a-box +$ make +``` + +## Pod onos-consensus-db-1-0 initialization failed + +In Ubuntu 20.04 (kernel 5.4.0-65-generic), the k8s pod named `onos-consensus-db-1-0` might fail due to a bug of using go and alpine together (e.g., https://github.com/docker-library/golang/issues/320). + +It can be seen in `kubectl logs -n riab onos-consensus-db-1-0` as: +```bash +runtime: mlock of signal stack failed: 12 +runtime: increase the mlock limit (ulimit -l) or +runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ +fatal error: mlock failed +``` + +Such pod utilizes the docker image atomix/raft-storage-node:v0.5.3, tagged from the build of the image atomix/dragonboat-raft-storage-node:latest available at https://github.com/atomix/dragonboat-raft-storage-node. + +A quick fix (allowing an unlimited amount memory to be locked by the pod) to this issue is cloning the repository https://github.com/atomix/dragonboat-raft-storage-node, and changing the Makefile: + +```bash +# Before change +image: build + docker build . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} + +# After change: unlimited maximum locked-in-memory address space +image: build + docker build --ulimit memlock=-1 . -f build/dragonboat-raft-storage-node/Dockerfile -t atomix/dragonboat-raft-storage-node:${RAFT_STORAGE_NODE_VERSION} +``` + +Then running in the source dir of this repository the command `make image`, and tagging the built image as: + +```bash +docker tag atomix/dragonboat-raft-storage-node:latest atomix/raft-storage-node:v0.5.3 +``` + +After that proceed with the execution of the Riab setup again. + +## Cannot see Google Map view on the Facebook-AirHop xAPP GUI +The Google Map API in the Facebook-AirHop xAPP GUI only allows us to use `localhost:8080` URL. +If we runs Facebook-AirHop xAPP on the remote machine, we have to make a SSH tunnel from the local machine to the remote machine: +```bash +$ ssh @ -L "*:8080::30095" +``` + +## Other issues? +Please contact ONF SD-RAN team, if you see any issue. Any issue report from users is very welcome. +Mostly, the redeployment by using `make clean-all and make [option]` resolves issues. \ No newline at end of file diff --git a/docs/figures/fbah-no-map.png b/docs/figures/fbah-no-map.png new file mode 100644 index 0000000..74cb0fb Binary files /dev/null and b/docs/figures/fbah-no-map.png differ diff --git a/docs/figures/fbah-with-map.png b/docs/figures/fbah-with-map.png new file mode 100644 index 0000000..f61d344 Binary files /dev/null and b/docs/figures/fbah-with-map.png differ diff --git a/docs/figures/hw_install.png b/docs/figures/hw_install.png new file mode 100644 index 0000000..57d69ff Binary files /dev/null and b/docs/figures/hw_install.png differ