-
Notifications
You must be signed in to change notification settings - Fork 46
NetFPGA SUME Reference Router Vivado 2020.4 and Ubuntu 2020.1
reference_router
projects/reference_router
- NetFPGA-SUME Specific cores
- Microblaze Subsystem
Note: The CAM core must be generated prior to running this project. Please follow the instructions in the CAM wiki page.
[root@nf-test109 NetFPGA-SUME-live]# cd projects/reference_router [root@nf-test109 NetFPGA-SUME-live]# make
A standalone CLI (Command Line Interpreter) that allows you to change routing table entries, ARP cache entries, and other settings is also provided in case the user doesn't want to run SCONE. The CLI is under NetFPGA-SUME-live/projects/reference_router/sw/host/cli. To build it, type make in that directory. You should see output similar to the following:
gcc -g -c -o cli.o cli.c gcc -g -c -o ../common/util.o ../common/util.c gcc -lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o cli gcc -g -c -o regdump.o regdump.c gcc -lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h -o regdump gcc -g -c -o show_stats.o show_stats.c gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h -o show_stats
For help on using the CLI, start it by typing ./cli and then type help in the CLI.
We invite you to extend this CLI and any of our software tools and contribute them back so we can expand our library and make it easier for anybody to use NetFPGA.
The router SCONE ( Software Component Of NetFPGA) is a user level router that performs IPv4 forwarding, handles ARPs and various ICMP messages, has telnet (port 23) and web (port 8080) interfaces to handle router control, and also implements a subset of OSPF named PW-OSPF. SCONE mirrors a copy of its MAC addresses, IP addresses, routing table, and ARP table to the NetFPGA card which hardware accelerates the forwarding path.
- Ensure your NetFPGA is programmed with the Reference Router bitfile.
- Check that your nf0, nf1, nf2, nf3 interfaces are up and do not have an assigned IPv4 address
- To remove an IPv4 address from an interface run
ifconfig nfX 0.0.0.0replacing X with the interface name
- Setup the cpuhw file
- A text file named cpuhw must exist in the sw directory that you execute the scone binary. The format is as follows:
- <interface name> <ip address> <ip mask> <mac address>
- An example:
eth0 192.168.0.2 255.255.255.0 00:00:00:00:00:01
- Inside SCONE the interfaces are typically named and referred to as eth0-3, which correspond to the nf0-3 ports.
- Specify a text file containing static routes (Optional)
- A text file containing static routes to be added on startup can be specified from the command line using the '-r' parameter, ie
-r rtable.netfpga. The format is as follows:- <net> <next hop> <mask> <outgoing interface name>
- An example:
192.168.130.14 192.168.130.9 255.255.255.254 eth1
- Specify a file to log packets in pcap format (Optional)
- Specify a file using the '-l' command line parameter, ie
-l scone.log, all packets received by SCONE will be logged using pcap format to this file. This file can be opened and examined using Wireshark. Note packets that take the hardware forwarding path will not appear in this log file.
To modify the way SCONE operates after launch, use the telnet command line interface. Connect to one of the IP addresses specified in the cpuhw file on port 23. To get a list of commands using telnet issue the ? command. For help with a command enter the command and put ? for its arguments. ie show ip ?.
For an incoming packet to take the hardware forwarding path the packet must get a hit on the destination MAC address table, the routing table, and the ARP table for the next hop. SCONE builds and maintains the routing table containing static and dynamic routes, and the ARP table in software. When changes are detected in the software tables, SCONE copies them down into the NetFPGA's hardware tables. If you want packets to be passed up to software that match the IP addresses assigned to the interfaces you must also push these IP addresses down into a hardware table. Following are some code snippets for writing to these various tables.
writeReg(&netfpga, SUME_OUTPUT_PORT_LOOKUP_0_MAC_0_HI, mac_hi);
writeReg(&netfpga, SUME_OUTPUT_PORT_LOOKUP_0_MAC_0_LOW, mac_lo);
// 1,2,3 can be substituted in for 0 to set the MAC for the other ports
NOTE : One confusing aspect of using a user level process to control the router is that the MAC addresses of the router will not match the kernel MAC addresses. That is, the MAC addresses of the nfX interfaces do not necessarily reflect those that are actually in the hardware. The nfX interfaces are software independent entities that do not necessarily reflect the state of the hardware.
The HCORR (Hardware Component of Reference Router) has a table that stores IP addresses called the destination IP filter table. Packets whose destination IP address matches an entry in this table will be sent to the hardware. We use this table to filter out the IP addresses of the router and the IP addresses of PW-OSPF multicast packets. We write into the table as follows:
struct in_addr ip;
writeReg(rs->netfpga_regs,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_B_LOW, ntohl(entry->ip.s_addr));
// i is the row to write the IP address to
writeReg(rs->netfpga_regs,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTADDRESS, SUME_OUTPUT_PORT_LOOKUP_0_MEM_DEST_IP_CAM_ADDRESS | i);
// write command
writeReg(rs->netfpga_regs,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTCOMMAND, 0x01);
The indirect access register infrastructure provide access to the tables (dest_ip, ip_arp, ip_lpm) of the router. In order to write data into the tables, the following registers must be configured.
SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_*_*
SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTADDRESS
SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTCOMMAND
More information about the indirect register access can be found here.
Writing to the Routing table is similar to writing to the Destination IP filter table. Note that the output port(s) is specified in one-hot-encoded format corresponding to the output port of the User Data Path to the Tx Queues. The function getOneHotPortNumber returns the values (in decimal): 1 , 4, 16, 64 which correspond to physical output ports 0-3 on the NetFPGA card. You can also tell a route to specifically send to software by specifying 2, 8, 32, 128 corresponding to the nf0,1,2,3 interfaces. To send out of MAC ports 0 and 1, write 1+4=5 as the output port entry.
/* write the ip */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_A_HI, ntohl(entry->ip.s_addr));
/* write the mask */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_B_HI, ntohl(entry->mask.s_addr));
/* write the next hop */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_A_LOW, ntohl(entry->gw.s_addr));
/* write the port */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_B_LOW, getOneHotPortNumber(entry->iface));
/* write the row number */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTADDRESS, SUME_OUTPUT_PORT_LOOKUP_0_MEM_IP_LPM_TCAM_ADDRESS | i);
/* write command */
writeReg(sume,SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTCOMMAND, 0x01);
/* write the mac address */
writeReg(sume, SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_A_HI, mac_hi);
writeReg(sume, SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_A_LO, mac_lo);
/* write the next hop ip data */
writeReg(sume, SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTWRDATA_B_LOW, ntohl(ip.s_addr));
/* set the row */
writeReg(sume, SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTADDRESS, SUME_OUTPUT_PORT_LOOKUP_0_MEM_IP_ARP_CAM_ADDRESS | i);
/* write command */
writeReg(sume, SUME_OUTPUT_PORT_LOOKUP_0_INDIRECTCOMMAND, 0x01);
Router Kit is a simple approach to providing hardware acceleration to an unmodified Linux system. It is comprised of a single program, rkd (router kit daemon), which monitors the Linux routing table and ARP cache and mirrors it down to the NetFPGA IPv4 reference router implementation.
rkd [-h||--help} [-d||--daemon} [-i||--interval]
rkd should work from the command line without any external configuration options. Simply run (./rkd). To run the process in the background use -d. You may specify the polling time in milliseconds using the -i option.
Router Kit is only useful on a Linux host with NetFPGA installed, and the ipv4 reference router bitfile loaded. Given this setup each port on the NetFPGA card is available to Linux via a nf* interface (i.e. nf0, nf1, nf2, and nf3 assuming a single card is installed).
rkd will attempt to mirror all ARP cache and routing table entries associate with a NetFPGA interface into hardware. This provides a very simple (and familiar) method of adding entries to the hardware. For example, to add a static ARP entry, simply use the arp(8) command. The following command will add a static ARP entry.
arp -s 1.2.3.4 ca:fe:de:ad:d0:d0 -i nf0
To add an entry into the routing table, use route(8) (or ip(8)). For example, adding a default entry with a next hop of 10.0.0.1 out of the first port would look something like:
route add default gw 10.0.0.1 dev nf0
Router kit is not limited to manual manipulation from the command line. All state (including dynamic state) is mirrored. To wit, running rkd alongside a standard routing daemon, such as XoRP, or Zebra/Quagga, should provide hardware acceleration of the forwarding table without any further configuration (provided the routing software is using the NetFPGA interfaces for forwarding).
rkd continuously polls the routing table and ARP cache state from /proc/net/route and /proc/net/arp respectively. When a change in state is detected, ./rkd writes the updated state to the NetFPGA through the register interface. All traffic not handled by the hardware is DMA'd to software where it is processed by the Linux kernel.
The division of the hardware into modules was hinted in the previous section. Understanding these modules is essential in making the most of the available designs. The distributed projects in the NFP, including the Router, all follow the same modular structure. This design is a pipeline where each stage is a separate module. Please, refer to the Reference NIC project for more information.
The first stage in the pipeline consists of several queues which we call the Rx queues. These queues receive packets from IO physical ports (i.e., 10GMAC) and provide a unified interface (AXI) to the rest of the system.
In the main datapath, the first module a packet passes through is the Input Arbiter. The input arbiter decides which Rx queue to service next, and pulls the packet from that Rx queue and hands it to the next module in the pipeline: The output port lookup module. The output port lookup module is responsible for deciding which port a packet goes out of. Here it is actually implemented the router forwarding logic. The 32-entries TCAM where the LPM table is stored is consulted to find the next-hop-ip value. Such a value is then used in the 32-entried CAM (ARP table) to find the right destination MAC address. After that decision is made, the packet is then handed to the output queues module which stores the packet in the output queues corresponding to the output port until the Tx queue is ready to accept the packet for transmission.
The Tx queues are analogous to the Rx queues and they send packets out of the IO ports instead of receiving.
Note: The reference router is not able to support the fully 40G line rate in the worst case scenario.
All the information could be found here.
Each projects has some features that are verified by doing Simulation tests and HW tests. The test infrastructure is based on the python. You can find the tests inside the projects/{project_name}/test folder.