Support for User Defined Metrics

[Vibaswan]

Redocly View
ReDoc Interactive Demo (redocly.github.io)

New Objects at:

• set_config/receive_bins
• get_metrics/port/received_bins

Requirement
Offer a means for users to configure and retrieve metrics based on custom conditions established by the user on packets generated by the test tool.

Feature Support and Limiations

Platform	Max Bins Per Rx Port	Supported Mode	Max filters per bin	Default bin
ixia-c/ ixia-hw	6	1 bin: both modes, >1 bin: parallel	3 (src_mac and/or dst_mac and/or 1 any other filter)	Only for sequential mode
UHD	6	1 bin: both modes, >1 bin: sequential	N/A	Only for sequential mode

Note: supported mode refers to the way the configured bins are populated with rx packets, for more information please refer to the description of the mode attribute in the model.

Usage For Mode Parallel

// supported by otg-hw and ixia-c 
// configure filter metrics for ports
config := gosnappi.NewConfig()
api := gosnappi.NewApi()

d1 := config.devices.Add()
eth1 := d1.Ethernets.Add().SetMac("00:01:02:03:04:05")

rb1 := config.ReceiveBins().Add()
// configuring bin parameters like association and mode
rb1.Association().SetPortName("p2")
rb1.Mode().Parallel()
b1 := rb1.Bins().Add().SetName("filter_mac_and_ip")
// packets with initial src mac as 00:01:02:03:04:XX, eg: any mac within 00:01:02:03:04:01
b1.BinFilters().PacketHeaders().Add().Ethernet().Src().SetValue(eth1.Mac()).SetMask("0000000000ff")
// packets with initial dst mac as 00:02:02:03:04:XX, eg: any mac within 00:01:02:03:04:ff
b1.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("00:02:01:03:04:05").SetMask("0000000000ff")
// packets with dst ip as 1.X.X.X eg; 1.2.3.4, 1.55.65.128
b1.BinFilters().PacketHeaders().Add().Ipv4().Dst().SetValue("1.1.1.1").SetMask("00ffffff")


rb2 := config.ReceiveBins().Add()
rb2.Association().SetPortName("p3")
rb2.Mode().Parallel()
b2 := rb2.Bins().Add().SetName("filter_custom")
b2.BinFilters().PacketHeaders().Add().Custom().SetValue("01010101").SetMask("ffff0000").SetOffset(18)

// rest of the script

mr := gosnappi.NewMetricsRequest()
bins := mr.Port().SetPortNames([]string{"p2", "p3"}).ReceivedBins()
bins.SetBinNames([]string{"filter_mac_and_ip", "filter_custom"})
api.GetMetrics(mr)

// to showcase a complex scenario
// if a user wants to tract Ip over Ip , then the packet structure needs to be formed in the same way 
rb3 := config.ReceiveBins().Add()
rb3.Association().SetPortName("p4")
rb3.Mode().Parallel()
b3 := rb3.Bins().Add().SetName("filter_ip_in_ip")
b3.BinFilters().PacketHeaders().Add().Ethernet()
b3.BinFilters().PacketHeaders().Add().Ipv4()
b3.BinFilters().PacketHeaders().Add().Ipv4().Dst().SetValue("1.1.1.1").SetMask("00ffffff")
// otherwise the user always has a way for directly filtering value with offset using custom

/*

Result structure
------------------------------------
/*
	 port_metrics:
	  - name: p2
		location: 10.39.33.43;1;2
		link: "up"
	   	frames_tx: 0
	   	frames_rx: 1000
		bytes_tx: 0,
		bytes_rx: 1000011,
		frames_tx_rate: 0,
		frames_rx_rate: 0,
		bytes_tx_rate: 0,
		bytes_rx_rate: 0,
		transmit: "stopped",
	   	received_bins:
	           - name: "filter_mac_and_ip"
	           - value: 500
	  - name: p3
		location: 10.39.33.43;1;3
		link: "up"
	   	frames_tx: 0
	   	frames_rx: 1000
		bytes_tx: 0,
		bytes_rx: 1000011,
		frames_tx_rate: 0,
		frames_rx_rate: 0,
		bytes_tx_rate: 0,
		bytes_rx_rate: 0,
		transmit: "stopped",
	   	received_bins:
	           - name: "filter_custom"
	           - value: 1000
*/

Usage For Mode Sequential

// supported by UHD as of now
// configure filter metrics for ports
config := gosnappi.NewConfig()
api := gosnappi.NewApi()

d1 := config.devices.Add()
eth1 := d1.Ethernets.Add().SetMac("00:01:02:03:04:05")

rb1 := config.ReceiveBins().Add()
// configuring bin parameters like association and mode
rb1.Association().SetPortName("p2")
rb1.Mode().Sequential()
b1 := rb1.Bins().Add().SetName("filter_mac_and_ip")
// packets with initial src mac as 00:01:02:03:04:XX, eg: any mac within 00:01:02:03:04:01
b1.BinFilters().PacketHeaders().Add().Ethernet().Src().SetValue(eth1.Mac()).SetMask("0000000000ff")
// packets with initial dst mac as 00:02:02:03:04:XX, eg: any mac within 00:01:02:03:04:ff
b1.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("00:02:01:03:04:05").SetMask("0000000000ff")
// packets with dst ip as 1.X.X.X eg; 1.2.3.4, 1.55.65.128
b1.BinFilters().PacketHeaders().Add().Ipv4().Dst().SetValue("1.1.1.1").SetMask("00ffffff")


rb2 := config.ReceiveBins().Add()
rb2.Association().SetPortName("p3")
rb2.Mode().Sequential()
b2 := rb2.Bins().Add().SetName("filter_custom")
b2.BinFilters().PacketHeaders().Add().Custom().SetValue("01010101").SetMask("ffff0000").SetOffset(18)

// rest of the script

mr := gosnappi.NewMetricsRequest()
bins := mr.Port().SetPortNames([]string{"p2", "p3"}).ReceivedBins()
bins.SetBinNames([]string{"filter_mac_and_ip", "filter_custom"})
api.GetMetrics(mr)


/*
Note: In Sequential mode a default bin is added, which contains packets which does not match any other configured bins
Result structure
------------------------------------
/*
	 port_metrics:
	  - name: p2
		location: 10.39.33.43;1;2
		link: "up"
	   	frames_tx: 0
	   	frames_rx: 1000
		bytes_tx: 0,
		bytes_rx: 1000011,
		frames_tx_rate: 0,
		frames_rx_rate: 0,
		bytes_tx_rate: 0,
		bytes_rx_rate: 0,
		transmit: "stopped",
	   	received_bins:
	           - name: "filter_mac_and_ip"
	              value: 500
	           - name: "default"
	              value: 500
	  - name: p3
		location: 10.39.33.43;1;3
		link: "up"
	   	frames_tx: 0
	   	frames_rx: 1000
		bytes_tx: 0,
		bytes_rx: 1000011,
		frames_tx_rate: 0,
		frames_rx_rate: 0,
		bytes_tx_rate: 0,
		bytes_rx_rate: 0,
		transmit: "stopped",
	   	received_bins:
	           - name: "filter_custom"
	             value: 1000
                   - name: "default"
                     value: 0
*/

gNMI Usage

Example query path for ports:

"/ports/port[name=p1]/received_bins/received_bin[name=*]"
"/ports/port[name=p1]/received_bins/received_bin[name=filter_mac_and_ip]"

To get gonsappi code from dev branch go get github.com/open-traffic-generator/snappi/gosnappi@dev-uds

FP tests that can be modified to use this feature

Test RT-261

Current code

ethTag.SetName("MACTrackingv4").SetOffset(36).SetLength(12)
ethTag.SetName("MACTrackingv6").SetOffset(36).SetLength(12)

With this feature

rb1 := config.ReceiveBins().Add()
rb1.Association().SetPortName("p1")
rb1.Mode().Parallel()

rb1b1 := rb1.Bins().Add().SetName("filter_mac_v4")
// all received packets on the port p1 with initial part of the MAC as 02:00:01:01:0X:XX will be counted in this bin. (ipv4)
// Note: other packets which satisfies this condition will also be counted such as BGP if enabled (not present in this test)
rb1b1.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("02:00:01:01:01:01").SetMask("000000000fff")
rb1b1.BinFilters().PacketHeaders().Add().Ethernet().EtherType().SetValue(2048).SetMask("0000")

rb1b2 := rb1.Bins().Add().SetName("filter_mac_v6")
// all received packets on the port p1 with initial part of the MAC as 02:00:01:01:0X:XX will be counted in this bin. (ipv6)
rb1b2.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("02:00:01:01:01:03").SetMask("000000000fff")
rb1b2.BinFilters().PacketHeaders().Add().Ethernet().EtherType().SetValue(2269).SetMask("0000")

// same filters for port p2 as well
rb2 := config.ReceiveBins().Add()
rb2.Association().SetPortName("p2")
rb2.Mode().Parallel()

rb2b1 := rb2.Bins().Add().SetName("filter_mac_v4")
rb2b1.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("02:00:01:01:01:01").SetMask("000000000fff")
rb2b1.BinFilters().PacketHeaders().Add().Ethernet().EtherType().SetValue(2048).SetMask("0000")

rb2b2 := rb2.Bins().Add().SetName("filter_mac_v6")
rb2b2.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("02:00:01:01:01:03").SetMask("000000000fff")
rb2b2.BinFilters().PacketHeaders().Add().Ethernet().EtherType().SetValue(2269).SetMask("0000")

// fetching metrics
mr := gosnappi.NewMetricsRequest()
bins := mr.Port().SetPortNames([]string{"p1", "p2"}).ReceivedBins()
bins.SetBinNames([]string{"filter_mac_v4", "filter_mac_v6})
res:= api.GetMetrics(mr)

// validation snippet 
// if we want to handle extra control packets being recieved with same match criteria , final checks can be bin_rx > flow_rx,
for _, metrics := range res.PortMetrics().Items() {
  for _, bin := range metrics.ReceivedBins().Items() {
	if bin.Name() == "filter_mac_v4" {
		if got := bin.Value(); got < flow1Rx {
			t.Errorf("filter_mac_v4 got %q, want >= %q", got, flow1Rx)
		}
	} else if bin.Name() == "filter_mac_v6" {
		if got := bin.Value(); got < flow2Rx {
			t.Errorf("filter_mac_v6 got %q, want >= %q", got, flow2Rx)
		}
	}
  }
}

Test 11.3

Current code

ipSrcTracking.SetName(srcTrackingName).SetOffset(27).SetLength(5)
ipDstTracking.SetName(dstTrackingName).SetOffset(0).SetLength(5)

With this feature

// rb1 and rb12 for tracking IpinIp packet without stripping the outer packet
rb1 := config.ReceiveBins().Add()
rb1.Association().SetPortName("p2")
rb1.Mode().Parallel()
b1 := rb1.Bins().Add().SetName("filter_ip_src")
// Note: for Non UHD we can only one non MAC field
b1.BinFilters().PacketHeaders().Add().Ipv4().Src().SetValue("198.51.100.2").SetMask("000000e0")

b2 := rb1.Bins().Add().SetName("filter_ip_dst")
b2.BinFilters().PacketHeaders().Add().Ipv4().Dst().SetValue("198.51.100.1").SetMask("e0000000")

// same filters can be created for port p3 and p4

// similar can be done for checking the packet where DUT strips outer Ip packet 
// only change will be in the value to check the src and dst for the inner Ip packet

// validation can be same as the first test example

Test 1.1`

Current code

ethTag.SetName("EgressTrackingFlow").SetOffset(36).SetLength(12)

With this feature

rb1 := config.ReceiveBins().Add()
rb1.Association().SetPortName("p2")
rb1.Mode().Parallel()

// for non poisoned sub-test
b1 := rb1.Bins().Add().SetName("filter_static_mac")
b1.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("02:00:01:01:01:01").SetMask("000000000fff")

// for poisoned sub test
b2 := rb1.Bins().Add().SetName("filter_poisoned_mac")
b2.BinFilters().PacketHeaders().Add().Ethernet().Dst().SetValue("12:34:56:78:7f:ff").SetMask("000000000fff")

// validation can be same as the first test example

Portability
If the number of bins per Rx port configured is 1 then the behavior is same across all platforms, or else for bins more than 1, otg-hw/ixia-c follow parallel mode and UHD follows sequential mode

[PrasenjitAdhikary]

In the Usage description, we need to add a flow example.
In the example for the result, aggregated tx / rx are not retrieved through filter stats and should not be part of the UDS response.

[Vibaswan]

In the Usage description, we need to add a flow example. In the example for the result, aggregated tx / rx are not retrieved through filter stats and should not be part of the UDS response.

Corrected the usage description according to suggestion

[ghost]

In the Usage example, the masks should not be 0.

[Vibaswan]

Example given in usage is wrong; for every receive bin we can have only one header field (other than src and dst mac). So configuring both IPv4->dst and custom fields for rb1 in usage is wrong. You can give two separate examples.

Corrected the example in the description

[Vibaswan]

How do we specify the correct header field for receive bins if we have say multiple headers of same type (say inner and outer IPv4 and I want to set receive bins only for inner IPv4 dst field).

Added a example for Ip in Ip in the description

[Vibaswan]

In the get metrics, we specify the list of ports and also the list of receive bins. What if some bins does not apply to some ports. Will we silently ignore those bins for that port, or return error. Not all bins will apply to all specified ports.

Secondly, the include flag is redundant I think. If we specify the bin name, means we want to include it.

removed redundant fields from metrics request ,
I feel if it does not find a match, implementation should ignore, I am not aware of any metrics response which returns error if it does not find a perfect match @PrasenjitAdhikary would please let us know your views on this

[laurchoy]

In the get metrics, we specify the list of ports and also the list of receive bins. What if some bins does not apply to some ports. Will we silently ignore those bins for that port, or return error. Not all bins will apply to all specified ports.
Secondly, the include flag is redundant I think. If we specify the bin name, means we want to include it.

removed redundant fields from metrics request , I feel if it does not find a match, implementation should ignore, I am not aware of any metrics response which returns error if it does not find a perfect match @PrasenjitAdhikary would please let us know your views on this

If the port doesn't find a match, there will be a default bin that it will drop into for all packets not matching any criteria. This way you can match back where the packet went. Also, counts should not be duplicated across bins. This is the user responsibility to make sure of that. The total number of packets across all bins should always equal to number of packets received.

[PrasenjitAdhikary]

In the get metrics, we specify the list of ports and also the list of receive bins. What if some bins does not apply to some ports. Will we silently ignore those bins for that port, or return error. Not all bins will apply to all specified ports.
Secondly, the include flag is redundant I think. If we specify the bin name, means we want to include it.

removed redundant fields from metrics request , I feel if it does not find a match, implementation should ignore, I am not aware of any metrics response which returns error if it does not find a perfect match @PrasenjitAdhikary would please let us know your views on this

If the port doesn't find a match, there will be a default bin that it will drop into for all packets not matching any criteria. This way you can match back where the packet went. Also, counts should not be duplicated across bins. This is the user responsibility to make sure of that. The total number of packets across all bins should always equal to number of packets received.

This behavior is not available in otg-hw or ixia-c (usstream). All UDS are independent, and there is no concept of a default bin, for un-matched packets. We cannot change this behavior in hardware/ixia-c.
So, to support minimum features across all platforms, UHD400 needs to support this behavior (I believe the same is already happening for UHD-100). Why this cannot be supported in UHD400?

We can add additional behavior specific to UHD only, if there is UHD UHD-specific requirement from the customer, in future sprints. But to support the UDS feature across all platforms uniformly, within the committed timeline, we should need this basic support, with independent bins.

[laurchoy]

UHD400 needs to support this behavior (I believe the same is already happening for UHD-100). Why this cannot be >supported in UHD400?
This has been discussed before. The decision was to have multiple pattern matches rather than independent matches. Jasdeep agreed to that as well.

[PrasenjitAdhikary]

UHD400 needs to support this behavior (I believe the same is already happening for UHD-100). Why this cannot be >supported in UHD400?
This has been discussed before. The decision was to have multiple pattern matches rather than independent matches. Jasdeep agreed to that as well.

At that point, it was planned to have a different workflow for UHD (flow-level) & OTG-HW (port-level). Now we agreed to have a common minimum that all platforms will support.

[laurchoy]

At that point, it was planned to have a different workflow for UHD (flow-level) & OTG-HW (port-level). Now we agreed to have a common minimum that all platforms will support.

So let me explain the reasoning we took for having default bins and multiple matches rather than independent matches. Let's assume we have 100 packets receive. independent match A claim it received 50 and independent match B claimed it received 60. This is a single test case where we receive packets from the DUT which can drop or replicate packets. From the received statistics, you cannot conclude that 50 packets that matches A matched B and the total number of packets that didn't match A or B just by looking at the receive packet count. This is not the case for the decision that we had put in place to enforce this and also to help debug issues.