patch to deduplicate HEP encapsulated packets coming from multiple proxies/agents

[gmaruzz]

Hello Kaian,

I use the HEP listening to see cleartext packets from platforms serving ipsec/wss/tls clients in addition to plain udp/tcp clients

I like to have it easy, so I do not sniff the network, I only display packets coming from the many HEP agents (eg, from the inside of Kamailio, OpenSIPS, FreeSWITCH, etc)

Problem is, when you have many HEP agents sending to one HEP listening server (sngrep in our case), the SIP messages are often duplicated, because sngrep receive one HEP encapsulated packet from the packet sender, and another one (the same) from the packet receiver (eg Proxy1->Proxy2 )

So, HEP content deduplication is needed

I borrowed idea from heplify (the standard HEP listening server in Homer): https://github.com/sipcapture/heplify
(search for deduplicate in main.go: https://github.com/sipcapture/heplify/blob/b5112d417d82591e844b784f391cda9b441d7bd9/main.go )

Being heplify written in GO, it has advanced cache libraries, etc

In C, I tried to make do :), and I added deleting/cleanup of old entries, because is not automatic like in Go library

So, please see attached the patch to sngrep sources, and the additional files needed to compile it all (eg: the cache/hashing stuff)

I use it this way (192.168.1.61 is the ethernet interface of the laptop sngrep is running on) :

/home/gmaruzz/opentelecom/git/sngrep/src/sngrep -N -O traffic.pcap -l 20 -R -E -L udp:192.168.1.61:9060

Albeit it works for me since some time, I'm sure it can be implemented in (many) better ways.

I put it here for you because this feature is very much needed in use cases similar to mine :)

to_kaian.zip

[Kaian]

Hi @gmaruzz !!

Thanks for the patch!

Long time ago, I started working on a version of sngrep using glib-2 library that, if I'm not mistaken, implemented something like this feature.

According to the code, the way to detect a retransmission is based on delta between received times of two packets. If they are duplicated (same payload content) and received in a short period, the second one is discarded.

This is what I implemented in the mentioned branch (it checks 10 ms instead of 5 ms of your code):

sngrep/src/storage/message.c

Lines 296 to 312 in a863151

	gboolean
	msg_is_duplicate(Message *msg)
	{
	if (!msg_is_retransmission(msg))
	return FALSE;
	g_autoptr(GDateTime) orig_ts = g_date_time_new_from_unix_usec(
	msg_get_time(msg_get_retransmission_original(msg))
	);
	g_autoptr(GDateTime) retrans_ts = g_date_time_new_from_unix_usec(
	msg_get_time(msg)
	);
	// Consider duplicate if difference with its original is 10ms or less
	return g_date_time_difference(retrans_ts, orig_ts) > 10000;
	}

Of course, current version of sngrep does not use glib-2 so, those functions can not be used, but just to check if that would be the correct direction: msg is retransmission (same payload) and delta time is small.

Regards!

[gmaruzz]

yep!

longer is retransmission (eg, SIP timers), and must be showed
shorter is duplicated

problem is that they can arrive even not in order, eg you cannot just check two consecutive packets

you must have a large enough cache of timeofarrival<->contenthash to contain let's say 10ms of all traffic

and it only can happen in HEP (not completely sure is true)

in normal network sniffing maybe it can happen with two physical ethernet?
or maybe using many virtual interfaces (they would probably have the exact same received time)?

[Kaian]

problem is that they can arrive even not in order, eg you cannot just check two consecutive packets

This should be no problem. Retransmission logic does not check previous message in dialog, but all received messages in the same dialog prior to the one being checked. Current implementation also checks source and destination addresses of the packet are the same in both original and duplicated message (the actual src and dst of the SIP message, not the source IP of the HEP agent).

you must have a large enough cache of timeofarrival<->contenthash to contain let's say 10ms of all traffic

We store all messages of the dialog in memory (inside call structure), so we can check them until the call is rotated.

in normal network sniffing maybe it can happen with two physical ethernet?

Yep, this is an issue. But this feature will have a setting to be toggled, so the user will only enable this behaviour whenever it knows there is little change to drop a valid packet. The implementation in glib-2 branch was used as a visual toggle, to remove duplicated from call flow, so it was not so aggresive as dropping the packet in capture layer :)

Regards!

[gmaruzz]

sweet!
Thanks a lot, Kaian!