Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bidirectional packet capture #6882

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add bidirectional packet capture #6882

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
875dde4
add bidirection flag
AryanBakliwal Dec 22, 2024
1307c17
add bidirectional capture
AryanBakliwal Dec 23, 2024
25d581b
Bump golang.org/x/net from 0.32.0 to 0.33.0 in the golang-org-x group…
dependabot[bot] Dec 24, 2024
60d3e7e
Bump github.com/onsi/ginkgo/v2 from 2.22.0 to 2.22.1 in the ginkgo gr…
dependabot[bot] Dec 24, 2024
79a26c0
update code and tests
AryanBakliwal Jan 1, 2025
6e5898c
Merge branch 'main' into bidirection
AryanBakliwal Jan 5, 2025
e5ef842
add enum, tests and docs
AryanBakliwal Jan 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion build/yamls/antrea-aks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3035,7 +3035,10 @@ spec:
type: integer
minimum: 1
maximum: 65535

direction:
type: string
enum: ["SourceToDestination", "DestinationToSource", "Both"]
default: "SourceToDestination"
timeout:
type: integer
minimum: 1
Expand Down
5 changes: 4 additions & 1 deletion build/yamls/antrea-eks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3035,7 +3035,10 @@ spec:
type: integer
minimum: 1
maximum: 65535

direction:
type: string
enum: ["SourceToDestination", "DestinationToSource", "Both"]
default: "SourceToDestination"
timeout:
type: integer
minimum: 1
Expand Down
5 changes: 4 additions & 1 deletion build/yamls/antrea-gke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3035,7 +3035,10 @@ spec:
type: integer
minimum: 1
maximum: 65535

direction:
type: string
enum: ["SourceToDestination", "DestinationToSource", "Both"]
default: "SourceToDestination"
timeout:
type: integer
minimum: 1
Expand Down
5 changes: 4 additions & 1 deletion build/yamls/antrea-ipsec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3035,7 +3035,10 @@ spec:
type: integer
minimum: 1
maximum: 65535

direction:
type: string
enum: ["SourceToDestination", "DestinationToSource", "Both"]
default: "SourceToDestination"
timeout:
type: integer
minimum: 1
Expand Down
5 changes: 4 additions & 1 deletion build/yamls/antrea.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3035,7 +3035,10 @@ spec:
type: integer
minimum: 1
maximum: 65535

direction:
type: string
enum: ["SourceToDestination", "DestinationToSource", "Both"]
default: "SourceToDestination"
Comment on lines +3038 to +3041
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you seem to have made that change manually, which is not correct
the source of truth is build/charts/antrea/crds/packetcapture.yaml. You need to edit that file and run make manifest.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification, I'll make the changes in the correct files and regenerate the manifests.

timeout:
type: integer
minimum: 1
Expand Down
3 changes: 3 additions & 0 deletions docs/packetcapture-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ the target traffic flow:
* Destination Pod, or IP address
* Transport protocol (TCP/UDP/ICMP)
* Transport ports
* Direction (SourceToDestination/DestinationToSource/Both)

You can start a new packet capture by creating a `PacketCapture` CR. An optional `fileServer`
field can be specified to store the generated packets file. Before that,
Expand Down Expand Up @@ -74,6 +75,8 @@ spec:
pod:
namespace: default
name: backend
# Available options for direction: `SourceToDestination` (default), `DestinationToSource` or `Both`.
direction: SourceToDestination # optional to specify
packet:
ipFamily: IPv4
protocol: TCP # support arbitrary number values and string values in [TCP,UDP,ICMP] (case insensitive)
Expand Down
164 changes: 143 additions & 21 deletions pkg/agent/packetcapture/capture/bpf.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ func compareProtocol(protocol uint32, skipTrue, skipFalse uint8) bpf.Instruction
// compilePacketFilter compiles the CRD spec to bpf instructions. For now, we only focus on
// ipv4 traffic. Compared to the raw BPF filter supported by libpcap, we only need to support
// limited use cases, so an expression parser is not needed.
func compilePacketFilter(packetSpec *crdv1alpha1.Packet, srcIP, dstIP net.IP) []bpf.Instruction {
size := uint8(calculateInstructionsSize(packetSpec))
func compilePacketFilter(packetSpec *crdv1alpha1.Packet, srcIP, dstIP net.IP, direction crdv1alpha1.CaptureDirection) []bpf.Instruction {
size := uint8(calculateInstructionsSize(packetSpec, direction))

// ipv4 check
inst := []bpf.Instruction{loadEtherKind}
Expand All @@ -101,20 +101,8 @@ func compilePacketFilter(packetSpec *crdv1alpha1.Packet, srcIP, dstIP net.IP) []
}
}

// source ip
if srcIP != nil {
inst = append(inst, loadIPv4SourceAddress)
addrVal := binary.BigEndian.Uint32(srcIP[len(srcIP)-4:])
// from here we need to check the inst length to calculate skipFalse. If no protocol is set, there will be no related bpf instructions.
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: addrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})

}
// dst ip
if dstIP != nil {
inst = append(inst, loadIPv4DestinationAddress)
addrVal := binary.BigEndian.Uint32(dstIP[len(dstIP)-4:])
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: addrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
srcAddrVal := binary.BigEndian.Uint32(srcIP[len(srcIP)-4:])
dstAddrVal := binary.BigEndian.Uint32(dstIP[len(dstIP)-4:])
Comment on lines +104 to +105
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand this change. Isn't it possible for srcIP and dstIP to be nil?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the current spec enforced srcIP and dstIP both cannot be nil, because either Pod or IP will present. It cloud be pretty easy to remove the restrictions in the crd spec and do a little tweak to make this work.(no the target of the PR)


// ports
var srcPort, dstPort uint16
Expand All @@ -134,18 +122,96 @@ func compilePacketFilter(packetSpec *crdv1alpha1.Packet, srcIP, dstIP net.IP) []
}
}

// source ip
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future we may also support tcp flags and other layer4 configs, if that happens, we should consider make the current code structure more modularized, or it would be extremely hard to extend this function . This won't be easy but i suggest to review this part and see if we can do better.

not sure if we can separate the ip section and ports section apart, call sub functions to calculate their instruments size and sums up.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try my best to improve the code structure and explore separating the IP and port sections as suggested

if srcIP != nil {
inst = append(inst, loadIPv4SourceAddress)
// from here we need to check the inst length to calculate skipFalse. If no protocol is set, there will be no related bpf instructions.
if direction == crdv1alpha1.Both {
if srcPort > 0 && dstPort > 0 { //TCP or UDP (both ports)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 13})
} else if (srcPort == 0 && dstPort > 0) || (srcPort > 0 && dstPort == 0) { // TCP or UDP (only one port)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 11})
} else { //ICMP
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 6})
}
} else if direction == crdv1alpha1.DestinationToSource {
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: dstAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
} else {
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
}
// dst ip
if dstIP != nil {
inst = append(inst, loadIPv4DestinationAddress)
if direction == crdv1alpha1.Both {
if srcPort > 0 || dstPort > 0 { //TCP or UDP
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: dstAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
} else { // ICMP
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: dstAddrVal, SkipTrue: size - uint8(len(inst)) - 3, SkipFalse: size - uint8(len(inst)) - 2})
}
} else if direction == crdv1alpha1.DestinationToSource {
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
} else {
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: dstAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
}

if srcPort > 0 || dstPort > 0 {
skipTrue := size - uint8(len(inst)) - 3
inst = append(inst, loadIPv4HeaderOffset(skipTrue)...)
if srcPort > 0 {
inst = append(inst, loadIPv4SourcePort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(srcPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
if direction == crdv1alpha1.DestinationToSource {
if dstPort > 0 {
inst = append(inst, loadIPv4SourcePort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(dstPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
} else {
if srcPort > 0 {
inst = append(inst, loadIPv4SourcePort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(srcPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
}
if dstPort > 0 {

if direction == crdv1alpha1.Both {
if dstPort > 0 {
inst = append(inst, loadIPv4DestinationPort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(dstPort), SkipTrue: size - uint8(len(inst)) - 3, SkipFalse: size - uint8(len(inst)) - 2})
}
} else if direction == crdv1alpha1.DestinationToSource {
if srcPort > 0 {
inst = append(inst, loadIPv4DestinationPort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(srcPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
} else {
inst = append(inst, loadIPv4DestinationPort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(dstPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
}

if direction == crdv1alpha1.Both {
// src ip (return traffic)
if dstIP != nil {
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: dstAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}

// dst ip (return traffic)
if srcIP != nil {
inst = append(inst, loadIPv4DestinationAddress)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: srcAddrVal, SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}

// return traffic ports
if srcPort > 0 || dstPort > 0 {
skipTrue := size - uint8(len(inst)) - 3
inst = append(inst, loadIPv4HeaderOffset(skipTrue)...)
if dstPort > 0 {
inst = append(inst, loadIPv4SourcePort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(dstPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
if srcPort > 0 {
inst = append(inst, loadIPv4DestinationPort)
inst = append(inst, bpf.JumpIf{Cond: bpf.JumpEqual, Val: uint32(srcPort), SkipTrue: 0, SkipFalse: size - uint8(len(inst)) - 2})
}
}
}

// return
Expand Down Expand Up @@ -178,7 +244,38 @@ func compilePacketFilter(packetSpec *crdv1alpha1.Packet, srcIP, dstIP net.IP) []
// (015) ret #262144 # MATCH
// (016) ret #0 # NOMATCH

func calculateInstructionsSize(packet *crdv1alpha1.Packet) int {
// When capturing return traffic also (i.e., both src -> dst and dst -> src), the filter might look like this:
// 'ip proto 6 and ((src host 10.244.1.2 and dst host 10.244.1.3 and src port 123 and dst port 124) or (src host 10.244.1.3 and dst host 10.244.1.2 and src port 124 and dst port 123))'
// And using `tcpdump -i <device> '<filter>' -d` will generate the following BPF instructions:
// (000) ldh [12] # Load 2B at 12 (Ethertype)
// (001) jeq #0x800 jt 2 jf 26 # Ethertype: If IPv4, goto #2, else #26
// (002) ldb [23] # Load 1B at 23 (IPv4 Protocol)
// (003) jeq #0x6 jt 4 jf 26 # IPv4 Protocol: If TCP, goto #4, #26
// (004) ld [26] # Load 4B at 26 (source address)
// (005) jeq #0xaf40102 jt 6 jf 15 # If bytes match(10.244.0.2), goto #6, else #15
// (006) ld [30] # Load 4B at 30 (dest address)
// (007) jeq #0xaf40103 jt 8 jf 26 # If bytes match(10.244.0.3), goto #8, else #26
// (008) ldh [20] # Load 2B at 20 (13b Fragment Offset)
// (009) jset #0x1fff jt 26 jf 10 # Use 0x1fff as a mask for fragment offset; If fragment offset != 0, #10, else #26
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe align the comments after #.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

// (010) ldxb 4*([14]&0xf) # x = IP header length
// (011) ldh [x + 14] # Load 2B at x+14 (TCP Source Port)
// (012) jeq #0x7b jt 13 jf 26 # TCP Source Port: If 123, goto #13, else #26
// (013) ldh [x + 16] # Load 2B at x+16 (TCP dst port)
// (014) jeq #0x7c jt 25 jf 26 # TCP dst port: If 123, goto #25, else #26
// (015) jeq #0xaf40103 jt 16 jf 26 # If bytes match(10.244.0.3), goto #16, else #26
// (016) ld [30] # Load 4B at 30 (return traffic dest address)
// (017) jeq #0xaf40102 jt 18 jf 26 # If bytes match(10.244.0.2), goto #18, else #26
// (018) ldh [20] # Load 2B at 20 (13b Fragment Offset)
// (019) jset #0x1fff jt 26 jf 20 # Use 0x1fff as a mask for fragment offset; If fragment offset != 0, #20, else #26
// (020) ldxb 4*([14]&0xf) # x = IP header length
// (021) ldh [x + 14] # Load 2B at x+14 (TCP Source Port)
// (022) jeq #0x7c jt 23 jf 26 # TCP Source Port: If 124, goto #23, else #26
// (023) ldh [x + 16] # Load 2B at x+16 (TCP dst port)
// (024) jeq #0x7b jt 25 jf 26 # TCP dst port: If 123, goto #25, else #26
// (025) ret #262144 # MATCH
// (026) ret #0 # NOMATCH

func calculateInstructionsSize(packet *crdv1alpha1.Packet, direction crdv1alpha1.CaptureDirection) int {
count := 0
// load ethertype
count++
Expand Down Expand Up @@ -214,6 +311,31 @@ func calculateInstructionsSize(packet *crdv1alpha1.Packet) int {
// src and dst ip
count += 4

if direction == crdv1alpha1.Both {
count += 3

transPort := packet.TransportHeader
if transPort.TCP != nil {
// load Fragment Offset
count += 3
if transPort.TCP.SrcPort != nil {
count += 2
}
if transPort.TCP.DstPort != nil {
count += 2
}

} else if transPort.UDP != nil {
count += 3
if transPort.UDP.SrcPort != nil {
count += 2
}
if transPort.UDP.DstPort != nil {
count += 2
}
}
}

// ret command
count += 2
return count
Expand Down
Loading
Loading