Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace pynids with something better #23

Open
nicomazz opened this issue Sep 6, 2020 · 4 comments
Open

Replace pynids with something better #23

nicomazz opened this issue Sep 6, 2020 · 4 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@nicomazz
Copy link
Collaborator

nicomazz commented Sep 6, 2020

No description provided.

@nicomazz nicomazz added enhancement New feature or request help wanted Extra attention is needed labels Sep 6, 2020
@cothan
Copy link

cothan commented Nov 7, 2020

Hey, I'm interesting replacing pynids.
May I know why do you need pynids?
What pynids has but scapy doesn't have?

@nicomazz
Copy link
Collaborator Author

nicomazz commented Nov 7, 2020

Hi cothan!
Thanks for the interest.
If I remember well, when flower was written there wasn't a nice API to reassembly TCP streams with scrapy, while pynids provided exactly what we needed in only a few lines of python.
From a fast search, it seems it is now possible. Your PR is highly welcomed!
Feel free to write me if you need any help.

@cothan
Copy link

cothan commented Nov 8, 2020

Hi @nicomazz ,

Similar question here: https://stackoverflow.com/questions/2259458/how-to-reassemble-tcp-segment

I have some reasons that it is can not easily be done in in Python, or at least need to use Python Binding wrapper.
Some promising library I found that support TCP reassembly is:

  1. https://github.com/seladb/PcapPlusPlus
  2. https://github.com/mfontanini/libtins

So basically libpcap, libint, and pcap++ are good.
https://pcapplusplus.github.io/docs/benchmark

All of the solution seem a bit complicated, from my little understanding of the code base, a simple tcpflow -r capture.pcap will do the job, the Python code will do the parsing data output from tcpflow.
By looking at some example from: https://github.com/simsong/tcpflow
The file name is: 128.129.130.131.02345-010.011.012.013.45103
We use Python to parse and read IP, port, high level step as follow:

  1. Read report.xml output from tcpflow
  2. Parse report.xml base on tcpflow startime, reorder packet base on port, IP and time. This approach will keep the conversation between IP1:port1 <=> IP2:port2 display in the same box.
  3. Feed to database.

Do you think so ? This will require writing to disk, but straight forward to parse and do TCP reassembly.

@nicomazz
Copy link
Collaborator Author

nicomazz commented Nov 8, 2020

Tbh I'm not a great fan of tcpflow for several reasons:

  1. first, tests in the repo are not passing. Then, compared to the other, the codebase seems a bit neglected.
  2. Passing through the disk seems adding a lot of useless overhead: during a real CTF there is an incredible number of packets (potentially more than 100GB), and doing a filesystem IO operation for each of them is too expensive, both for the time needed and for the lifespan of your SSD maybe.

The two options you've proposed seem great! Both support ipv6, that is a requirement of the new replacement, and they have ready to use examples:

  1. PcapPlusPlus has an example that generates the output in the tcpflow format. Maybe you can change that to avoid writing a file to the filesystem? The great thing here is that both ip v4 and v6 are supported
  2. Also libtins has already a pretty nice interface for what we need. It is basically the same as what we are using now in python.

Let me know if you need any help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants