Initial commit of the new IOTG TSN Reference Software

This is the first commit, comprising of over 2 years
of hard work, thanks to the many individuals listed below.

This commit introduces 3 separate C-applications and a lot of
new scripts to enable users to try it out quickly. As with
the older version, this reference software is specific to certain
Intel IOTG platforms only - which currently does not include
Apollo Lake I and the i210 Ethernet controller.

As this is an initial commit, some README and instructions may be
incomplete, and near-future commits will address them.

Co-authored-by: Noor Azura Ahmad Tarmizi <[email protected]>
Co-authored-by: Khor, Isaac Shi Yan <[email protected]>
Co-authored-by: Singh, Sartaj <[email protected]>

Signed-off-by: Ong, Boon Leong <[email protected]>
Signed-off-by: Wong, Vee Khee <[email protected]>
Signed-off-by: Kweh Hock Leong <[email protected]>
Signed-off-by: Weifeng Voon <[email protected]>
Signed-off-by: Song, Yoong Siang <[email protected]>

.gitignore

@@ -0,0 +1,33 @@
+# Editing and running artifacts
+*.plot
+*.eps
+*.log
+*.o
+*.a
+tags
+*.vim
+*.swp
+*.un
+*~
+*.pcap
+
+# Build artifacts
+tsq
+txrx-tsn
+
+# Skip result files
+results-*/
+*.txt
+*.png
+
+# Ignore convenience symlink
+a5
+a6
+
+# We only save json.i
+*.json
+
+opcua-server
+src/build-opcua/
+iperf3-gen-cmd.sh
+json/__pycache__/

LICENSE.txt

@@ -0,0 +1,30 @@
+BSD 3-Clause License
+
+Copyright (c) 2020, Intel Corporation
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+ 1. Redistributions of source code must retain the above copyright notice,
+    this list of conditions and the following disclaimer.
+
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in the
+    documentation and/or other materials provided with the distribution.
+
+ 3. Neither the name of the copyright holder nor the names of its
+    contributors may be used to endorse or promote products derived from
+    this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

README.md

@@ -0,0 +1,105 @@
+# IOTG TSN Reference Software for Linux
+
+## Overview
+
+This project is a set of scripts and C-applications used to showcase specific
+__Ethernet-TSN__ features and workloads. Its many components can be used individually
+in a variety of ways but this project will use them specifically for the
+below 3 applications and their dedicated/respective examples (aka configs):
+
+* **TSQ** [Time Synchronization Quality Measurement]
+  * __tsq1__ showcases the nanosecond-precision time synchronization between platforms
+    using pulse-per-second(PPS) output & auxiliary timestamping(AUX_TS) input pins.
+
+* **TXRX-TSN** [AF_PACKET & AF_XDP socket-based application]
+  * __vs1__ showcases the bounded low-latency transmission/reception achievable via AF_PACKET
+    & AF_XDP standard linux socket interfaces, while leveraging various
+    device-specific Ethernet-TSN features.
+
+* **OPCUA-SERVER** [AF_PACKET & AF_XDP OPCUA-based application]
+  * __opcua-pkt1__ showcases the bounded low-latency
+    transmission/reception achievable via AF_PACKET over libopen62541
+    (an OPC-UA-based library) APIs, while leveraging various device-specific
+    Ethernet-TSN features over a single-ethernet connection. This application
+    uses JSON files as an input.
+  * __opcua-pkt2__, __opcua-pkt3__, __opcua-xdp2__ & __opcua-xdp3__ showcases
+    the bounded low-latency transmission/reception achievable via AF_PACKET &
+    AF_XDP over libopen62541 (an OPC-UA-based library) APIs, while leveraging
+    various device-specific Ethernet-TSN features over **two** ethernet
+    connection.This application uses JSON files as an input and require platforms
+    with at least 2 Ethernet-TSN ports (e.g. EHL). Users can choose either __opcua-*2__
+    or __opcua-*3__ to enable/disable background (iperf3) traffic.
+
+Refer to the full documentation for details as this README serves as a
+high-level overview only. Many specifics will not be present here due to the
+project's complexity.
+
+## Disclaimers
+
+This project serves as an example to demonstrate TSN functionality only on the
+supported platforms. This project is not for intended for production use.
+
+Users are responsible for their own products functionality and performance.
+Refer to LICENSE.txt for the license used by this project.
+
+Each example defaults to a set of parameter values that has been tested on the
+supported platforms to achieve ideal bounded-latency for each packet while the
+platform is running on a preempt-rt kernel.
+
+## Dependencies
+
+This project is designed to work on specific Intel hardware and their
+respective bundled software (e.g. IFWI, BSP, preempt-rt kernel). These examples
+may not perform as expected when run on other platforms or configurations.
+Currently supported platforms are:
+
+* Elkhart Lake
+* Tiger Lake UP3
+
+These packages are required to be installed for the scripts to run. (Note: If
+you are running from an Intel-provided Yocto BSP, these dependencies would have
+already been installed):
+
+* linuxptp - clock synchronization utilties (e.g. ptp4l & phc2sys)
+* gnuplot
+* python (3.7.7)
+* libopen62541
+* libbpf
+* libelf
+* librt
+* libpthread
+* libjson
+* iperf3
+
+### IceWM shortcuts:
+
+A GUI/window manager should be used while executing these applications. The Yocto BSP
+uses IceWM as its default window manager. Here are some keyboard shortcuts,
+in case a mouse is not available.
+
+Notation: C is control, M is meta/alt, S is shift, s is super
+
+1. `<C-M-t>` to open xterm
+2. `<M-f8>` to resize window (using arrow keys)
+3. `<M-S-f10>` to maximise vertical space
+4. `<M-f7>` to move window
+5. `<C-M-leftarrow>` to move to the left, rightarrow for right
+6. `<C-M-esc>` to show window list
+7. `<M-space>` to show window actions menu
+
+## Build the project
+
+In a Yocto BSP with IOTG TSN Ref Sw included:
+
+1. To build all 3 applications:
+
+```bash
+cd /usr/share/iotg-tsn-ref-sw/
+./build.sh
+```
+
+## Run the examples
+
+For instructions to run TSQ or TXRX-TSN,refer to [README_sh.md](README_sh.md)
+
+For instructions to run OPCUA-SERVER, refer to [README_json.md](README_json.md)

README_json.md

@@ -0,0 +1,184 @@
+# Running examples using JSON
+
+All examples are run with 2 units of the same platform. Mind the notation
+"[Board A or B]". The following steps assumes both platforms are connected
+to each other via an Ethernet connection and user has a terminal open in the
+/usr/share/iotg-tsn-ref-sw directory - with the C-applications already built.
+
+## Role of run.sh & JSON
+
+Unlike TSQ and TXRX-TSN which are regular C-application that takes in input via
+parameter lists. OPCUA-SERVER uses JSON files for input. This is to allow OPCUA-SERVER
+to have more customizable parameters and enable pre-written example configurations
+in the form of JSON. To make it even easier, all Linux configurations are also
+customizable via JSON file.
+
+Run.sh serves only as a shortcut to call the desired scripts and pass them
+the JSON files. Each pre-defined example has either a *.json.i or *-tsn.json.i
+file for either Board A or Board B.
+
+Important: when using run.sh, users should modify the corresponding
+**.json.i** file NOT the generated .json file. **.json.i** denotes incomplete or
+interim json file as it has fields intentionally left blank for run.sh to fill in
+automatically.
+
+When a user executes run.sh with a command (init/setup/run), the following scripts
+are called respectively:
+
+```bash
+
+#Init only
+run.sh <plat> <iface> [iface2] <json.i> init
+    |__ json/opcua-run.sh       #create a json using correct iface(s)
+        |__ json/helpers.sh     #set ip/mac address, single/dual port
+
+#Setup only
+run.sh <plat> <iface> [iface2] <json.i> setup
+    |__ json/opcua-run.sh       #create a json using correct iface(s)
+        |__ json/gen_setup.py       #parse json and generate setup scripts
+        |__ generated_setup.sh      #set up clocks, different TCs
+            |__ ptp4l
+            |__ phc2sys
+            |__ tc
+
+#Run only
+run.sh <plat> <iface> [iface2] <json.i> run
+    |__ json/opcua-run.sh       #create a json using correct iface(s)
+        |__ ./iperf-gen-cmd.sh      #run iperf if required
+        |__ ./opcua-server <json>   #use new json opcua parameters to execute
+        |__ gnuplot & save results
+```
+
+## OPCUA-SERVER - AF_PACKET & AF_XDP OPCUA-based application
+
+Refer to the full documentation for details as this README will serve as an
+overview only.
+
+OPCUA-SERVER is a C-application that can transmit and receive ETH_UADP packets using
+AF_PACKET or AF_XDP sockets using libopen62541 (OPCUA-based library) APIs. It
+accepts only 1 .json.i file as input and supported .json.i entries are as follows
+(kindly refer to the examples in json/<plat>/ for examples of its structure tree):
+
+|     Category      |            Field             |  Type   |  Min   |   Max    |
+| ----------------- | ---------------------------- | ------- | ------ | -------- |
+| Mandatory/General | publisher_interface          | String  |        |          |
+| Mandatory/General | subscriber_interface         | String  |        |          |
+| Mandatory/General | use_xdp                      | Boolean |        |          |
+| Mandatory/General | packet_count                 | Int     | 1      | 10000000 |
+| Mandatory/General | cycle_time_ns                | Int     | 100000 | 5000000  |
+| Mandatory/General | polling_duration_ns          | Int     | 0      | 10000000 |
+| Publisher         | url                          | String  |        |          |
+| Publisher         | pub_id                       | String  |        |          |
+| Publisher         | dataset_writer_id            | Int     | 0      | 99999    |
+| Publisher         | writer_group_id              | Int     | 0      | 99999    |
+| Publisher         | early_offset_ns              | Int     | 0      | 1000000  |
+| Publisher         | publish_offset_ns            | Int     | 0      | 10000000 |
+| Publisher         | socket_prio                  | Int     | 0      | 7        |
+| Publisher         | two_way_data                 | Boolean |        |          |
+| Publisher         | cpu_affinity                 | Int     | 0      | 3        |
+| Publisher         | xdp_queue                    | Int     | 1      | 3        |
+| Subscriber        | url                          | String  |        |          |
+| Subscriber        | sub_id                       | String  | 0      | 99999    |
+| Subscriber        | subscribed_pub_id            | Int     | 0      | 99999    |
+| Subscriber        | subscribed_dataset_writer_id | Int     | 0      | 99999    |
+| Subscriber        | subscribed_writer_group_id   | Int     | 0      | 99999    |
+| Subscriber        | offset_ns                    | Int     | 0      | 10000000 |
+| Subscriber        | subscriber_output_file       | String  |        |          |
+| Subscriber        | two_way_data                 | Boolean |        |          |
+| Subscriber        | cpu_affinity                 | Int     | 0      | 3        |
+| Subscriber        | xdp_queue                    | Int     | 1      | 3        |
+
+### Single-port example
+
+Similar to TSQ & TXRX-TSN examples, 2 platforms are required to be connected to
+each other via a single-ethernet connection.
+
+1.  [Board A] Run the setup script to initialize IP and MAC address, then start
+    clock synchronization and setup TAPRIO + ETF qdiscs.
+
+    ```bash
+    cd /usr/share/iotg-tsn-ref-sw/
+    ./run.sh <PLAT> $IFACE opcua-pkt1a init
+    ./run.sh <PLAT> $IFACE opcua-pkt1a setup
+    ```
+
+2.  [Board A] Run the setup script to initialize IP and MAC address, then start
+    clock synchronization and setup ingress qdisc.
+
+    ```bash
+    cd /usr/share/iotg-tsn-ref-sw/
+    ./run.sh <PLAT> $IFACE opcua-pkt1b init
+    ./run.sh <PLAT> $IFACE opcua-pkt1b setup
+    ```
+
+3.  [Board B] Start listening for packets.
+
+    ```bash
+    ./run.sh <PLAT> $IFACE opcua-pkt1b run
+    ```
+
+4.  [Board B] Immediately after step 3, start transmitting packets. The
+    application will terminate itself after completion. Note that AF_PACKET or
+    opcua-pkt* configurations will take longer as they use much higher packet
+    intervals compared to AF_XDP or opuca-xdp* configurations.
+
+    ```bash
+    ./run.sh <PLAT> $IFACE opcua-pkt1a run
+    ```
+
+5.  [Board B] A gnuplot window should appear showing the time taken for each
+    packet to traverse from the application in Board A to Board B.
+
+    Each run will enumerate and store its raw timestamps/data & plot images in the
+    results-<DATE> folder for easy reference, repeat steps 3-4 as needed.
+
+### Dual-port example
+
+In this example, 2 platforms are required to be connected to each other via
+**two** ethernet connections. Meaning there is a total of four
+ethernet interfaces (as opposed to the usual two).
+
+A simplified explanation of the data flow:
+1. Board A's first interface ($IFACE1) is used to transmit
+2. Board B's first interface ($IFACE1) is used to receive
+3. Board B's second interface ($IFACE2) is used to transmit back to Board A
+4. Board A's second interface ($IFACE2) is used to receive
+
+When using dual-ports, the <PLAT> should be replaced with "ehl2". (TGL not supported)
+
+1.  [Board A] Run the setup script to initialize IP and MAC address, then start
+    clock synchronization and setup TAPRIO + ETF qdiscs.
+
+    ```bash
+    cd /usr/share/iotg-tsn-ref-sw/
+    ./run.sh <PLAT> $IFACE $IFACE2 opcua-pkt3a init
+    ./run.sh <PLAT> $IFACE $IFACE2 opcua-pkt3a setup
+    ```
+
+2.  [Board B] Run the setup script to initialize IP and MAC address, then start
+    clock synchronization and setup ingress qdisc.
+
+    ```bash
+    cd /usr/share/iotg-tsn-ref-sw/
+    ./run.sh <PLAT> $IFACE $IFACE2 opcua-pkt3b init
+    ./run.sh <PLAT> $IFACE $IFACE2 opcua-pkt3b setup
+    ```
+
+3.  [Board B] Start listening for packets.
+
+    ```bash
+    ./run.sh <PLAT> $IFACE $IFACE2 opcua-pkt3b run
+    ```
+
+4.  [Board A] Immediately after step 3, start transmitting packets.
+
+    ```bash
+    ./run.sh <PLAT> $IFACE opcua-pkt3a run
+    ```
+
+5.  [Board A] A gnuplot window should appear showing the time taken for each
+    packet to traverse from the application in Board A to Board B and back to
+    Board A.
+
+    Each run will enumerate and store its raw timestamps/data & plot images in the
+    results-<DATE> folder for easy reference, repeat steps 3-4 as needed.