-
Notifications
You must be signed in to change notification settings - Fork 141
ZFilter
A Bloom-filters-based multicast forwarding node is implemented in this project. The project also contains software components that can be used to test the implementation.
- Status :
- Version :
- Authors :
- Demonstration :
- NetFPGA source :
Among the many future networking projects, an EU FP7 project PSIRP and the Finnish national project ICT SHOK FI are evaluating new approaches to inter-networking. One of the most promising new ideas is to apply the publish/subscribe paradigm in the internetworking layer, instead of the currently dominant send/receive paradigm, which is used in most networking protocols today.
Traditionally, publish/subscribe networks have been overlays on top of IP. However, now that we are studying what would be the gains of applying this paradigm directly on top of link layer or even bare hardware, we need to evaluate the performance of this architecture against the dominant IP architecture. NetFPGA is a good tool to do that, as there is the reference IP router implementation, to which we can compare our own pub/sub-based information router.
The goal of zFilter sprouter project is to create a forwarding node prototype for a publish/subscribe networks. Implementation is based on reference switch design but it is modified so that forwarding decisions will be made based on Bloom filters. In more detail, Bloom filters are included in the packet header, we call this new header zFilter, and all links are named with LinkID. Packets are then forwarded to links if the specific LinkID is contained in the Bloom filter.
- In-packet Bloom-filters-based ("zFilter") multicast forwarding (see the References)
- User-space configurable
- Simple TTL for loop mitigation
- Packets are not sent back to port where those came from
- Information related to last forwarding decision can be read from userspace
- Testing tools
Sources for the project can be downloaded from
http://www.psirp.org/files/Deliverables/netfpga.zfilter_sprouter.1_1.tar.gz
There is test script under regression/regress.sh under project directory. It requires two network interfaces on NetFPGA host machine. Names of those interfaces are defined in the beginning of that file and should be modified according to used setup.
Script sends packets from one interface and monitors status registers of the NetFPGA and check the other interface for incoming packets. This is used to test basic forwarding functionality.
Two network ordinary interfaces are required. Those should be connected to the NetFPGA ports zero and one. Interface names are defined in the beginning of regress.sh file and should be modified to match current setup. Regress test needs to be run as a root user.
At first setting and reading linkID:s is tested. If that is passed packets are sent with different filters. In the end amount of failures is displayed. If all went well there should be 0 failures.
Tests include sending of packets with different normal and virtual linkID:s. Also sending with full filter is tested, as well as setting the maximum amount of ones allowed. Sending with different TTL values is tested as well.
The source packet contains only sources for the project, reference lib files must be already installed. Those can be downloaded from Software Package]. The netfpga_base_beta files are required from there. Guide to use the reference designs can be found from NetFPGA guide. K ernel drivers and CPCI configuration are required and should be installed as described in that guide.
LinkIDs are set by using tool called "psm". It is located in "/sw/PubSubManager" and it can be compiled with "make" in that directory (installation of packages byacc and flex might be needed). Easiest way to set all LinkIDs is to read the present filters to a file withpsm get filter file conf_fileand to modify the file "conf_file" accordingly. After that, with
psm set filter file conf_filethe changes can be written back to the NetFPGA. The same psm software that is used to set LinkID:s can be used also to send packets. The command
psm -usageshows all option. A packet can be sent, for example, with
psm send packet ttl 1 tid 1 if 1 cnt 1 delay 100 len 1000 output F000000000000000000000000000000000000000000000000000000000000000Last long parameter is the zFilter that is to be used in the packet, and unfortunately it can't be read from a file yet. Demonstration of multicast video streaming by using zFilter based forwarding. Demonstration includes one publisher (server) and multiple subscribers (clients). Server sends stream to the network when ever there is subscribers. Multiple clients can subscribe to the same stream. When new subscription request is received, server modifies outgoing zFilter. In this demostration, subscription requests are sent by virtual tree that is directed to the server. Backwards path is collected by subscription packets and it is then used to transmit the actual stream. Compilation of the software is needed before anything. Command "make" should be run in both /sw/PubSubManager and /sw/counterdump directories. Demonstration files are in the project directory under /demo. There is two directories netfpga_config for the NetFPGA card configuration and pubsubdemo. NetFPGA card can be configured by running ./setup script with number between 1-3 as a parameter. The script needs root priviledges. Port 0 should be either connected to the server or to a node that is closet to the server. This should be done to every forwarding node in the network. Actual demonstration is located in /demo/pubsubdemo/. The demonstration requires reasonably new Python and VLC player. There is multiple scripts to help with parameters. Easiest way is to use publish_video or publish_audio on the server. Audio and video file names can be easily found inside those scripts so it is straight forward to use any file. Subscribers can then use respective scripts to subscribe the stream. Separate audio and video files can be streamed simultaneously. Even more streams are allowed simultaneously if scripts with different port numbers are used. This implementation should be used with zFilter packets. Format shown in following table. Packet header format
| MAC header | zFilter type | TTL | LIT | Unused | zFilter | |
| Byte no. | 0-14 | 15-16 | 17 | 18 | 19-24 | 25 - 57 |
| Desc | UDP base | TAG addr | Index | Table idx | Dep idx | Hi/Lo | Unused |
| Bit no. | 26-23 | 22-10 | 9-7 | 6-5 | 4-3 | 2 | 1-0 |
| Value | 4'b0001 | 12'h102 | XXX | XX | XX | X | 00 |
| Name | Address | Description |
| ZF_CONFIGURATION | 0x02103000 | Configuration of current implementation (each value is two bytes) |
| [FILTER_DEP,MAX_LITS,MAX_IDS,LINKS] | ||
| ZF_MAX_ONES | 0x02103004 | Maximum number of ones that is accepted (can be read and written) |
| ZF_LAST_PKT | 0x02103008 | Info from last packet (each value is two bytes) |
| [incoming] | ||
| ZF_SELECTIONS | 0x0210300c | zFiltering results from last packet |
| ZF_DEBUG_LOW | 0x02103010 | Debug data can be written in NetFPGA to this register if needed (low bits) |
| ZF_DEBUG_HIGH | 0x02103014 | Debug data can be written in NetFPGA to this register if needed (high bits) |
Guide to use the reference designs can be found from NetFPGA guide. K ernel drivers and CPCI configuration are required and should be installed as described in that guide.
LinkIDs are set by using tool called "psm". It is located in "/sw/PubSubManager" and it can be compiled with "make" in that directory (installation of packages byacc and flex might be needed). Easiest way to set all LinkIDs is to read the present filters to a file with
psm get filter file conf_file
and to modify the file "conf_file" accordingly. After that, with
psm set filter file conf_file
the changes can be written back to the NetFPGA.
The same psm software that is used to set LinkID:s can be used also to send packets. The command
psm -usage
shows all option. A packet can be sent, for example, with
psm send packet ttl 1 tid 1 if 1 cnt 1 delay 100 len 1000 output F000000000000000000000000000000000000000000000000000000000000000
Last long parameter is the zFilter that is to be used in the packet, and unfortunately it can't be read from a file yet.
Demonstration of multicast video streaming by using zFilter based forwarding. Demonstration includes one publisher (server) and multiple subscribers (clients). Server sends stream to the network when ever there is subscribers. Multiple clients can subscribe to the same stream. When new subscription request is received, server modifies outgoing zFilter. In this demostration, subscription requests are sent by virtual tree that is directed to the server. Backwards path is collected by subscription packets and it is then used to transmit the actual stream.
Compilation of the software is needed before anything. Command "make" should be run in both /sw/PubSubManager and /sw/counterdump directories.
Demonstration files are in the project directory under /demo. There is two directories netfpga_config for the NetFPGA card configuration and pubsubdemo. NetFPGA card can be configured by running ./setup script with number between 1-3 as a parameter. The script needs root priviledges. Port 0 should be either connected to the server or to a node that is closet to the server. This should be done to every forwarding node in the network.
Actual demonstration is located in /demo/pubsubdemo/. The demonstration requires reasonably new Python and VLC player. There is multiple scripts to help with parameters. Easiest way is to use publish_video or publish_audio on the server. Audio and video file names can be easily found inside those scripts so it is straight forward to use any file. Subscribers can then use respective scripts to subscribe the stream. Separate audio and video files can be streamed simultaneously. Even more streams are allowed simultaneously if scripts with different port numbers are used.
This implementation should be used with zFilter packets. Format shown in following table.
Packet header format
| MAC header | zFilter type | TTL | LIT | Unused | zFilter | |
| Byte no. | 0-14 | 15-16 | 17 | 18 | 19-24 | 25 - 57 |
When packet arrives to output_port_selector ethertype is checked and index of the linkID table (LIT) and TTL is read from the beginning of the packet. Based on LIT correct 64bit long pieces of LinkID:s that will be used on next clock cycle are fetched from ID storage. On next clock cycle matching is done parallel between each ports each link ID and incoming zFilter and new 64bit long pieces are fetched. After matching is done for the whole zFilter, results are combined to make final forwarding decisions.
During the comparison the amount of ones in zFilter is also counted and if it exceeds the limit the packet is then dropped. The packet is also dropped if ethertype was wrong or TTL was zero, otherwise TTL is decreased by one. Also the port where packet came in is removed from the list of ports where packet should go.
LinkID:s of the node can be modified from user space. This is done by accessing following adresses.
Register addresses for LinkID:s
| Desc | UDP base | TAG addr | Index | Table idx | Dep idx | Hi/Lo | Unused |
| Bit no. | 26-23 | 22-10 | 9-7 | 6-5 | 4-3 | 2 | 1-0 |
| Value | 4'b0001 | 12'h102 | XXX | XX | XX | X | 00 |
Another memory area is used for status and settings. Address definitions are located in /sw/common/zf_defines.h
Register addresses settings and status
| Name | Address | Description |
| ZF_CONFIGURATION | 0x02103000 | Configuration of current implementation (each value is two bytes) |
| [FILTER_DEP,MAX_LITS,MAX_IDS,LINKS] | ||
| ZF_MAX_ONES | 0x02103004 | Maximum number of ones that is accepted (can be read and written) |
| ZF_LAST_PKT | 0x02103008 | Info from last packet (each value is two bytes) |
| [incoming] | ||
| ZF_SELECTIONS | 0x0210300c | zFiltering results from last packet |
| ZF_DEBUG_LOW | 0x02103010 | Debug data can be written in NetFPGA to this register if needed (low bits) |
| ZF_DEBUG_HIGH | 0x02103014 | Debug data can be written in NetFPGA to this register if needed (high bits) |