Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Test DHT with large number of nodes #57

Draft
wants to merge 61 commits into
base: master
Choose a base branch
from
Draft

Conversation

cskiraly
Copy link
Collaborator

@cskiraly cskiraly commented May 21, 2023

The goal of this WIP PR is to collect patches required to use the codebase for large scale DHT emulation, mostly focusing on the DAS use case. Not to be merged, currently just a collection of patches.

Patches currently live in this repository, but they should apply with little changes also in the original nim-eth repo, especially if we merge the code factorisation in status-im/nim-eth#480.

Current approach

  • Take real DHT implementation
    • This or Kim's original
  • Run it over simulated network
    • Switch UDP to in-memory message passing
      • not for this use case, but eventually TCP can easily be added to the simulator through a user-space TCP stack
    • Add configurable delay, loss, and eventual egress/ingress queuing
  • Make it work for large numbers
    • Remove bulk of calculation (e.g. fake encryption)
      • Remove transport encryption
      • Remove signatures
    • Add some time warp
      • Simple time multiplier (e.g. 1 emuSec = 10 sec)
      • add event based simulated time
  • Add upload/download code
    • put/get
    • upload / download traffic pattern

TODO: We might want to still generate the right amount

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
run e.g. as
```
nim c -r -d:debug -d:chronicles_enabled=on -d:chronicles_log_level=TRACE -d:chronicles_sinks=textlines[nocolors,stdout] tests/dht/test_providers.nim >err
```

Signed-off-by: Csaba Kiraly <[email protected]>
Override Chronos DatagramTransport with a fake one

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
no need to name it different, we can override DatagramTransport

Signed-off-by: Csaba Kiraly <[email protected]>
0.0.0.0 vs. 127.0.0.1 creates various issues
As a start, we can index by port only

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
initialize wait for response before sending request.
This is needed in cases where the response arrives before
moving to the next instruction, such as a directly connected
test.

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
ChronoSim, the simulation/emulation wrapper
around nim-chronos.

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
The previous version was based on the assumption that
that the future of waitMessage is triggered in the
event loop before the next message arrives. This seems
to be true with UDP, where a full cycle of the loop
handles only one datagram, but not true with other
transports, such as the in-memory emulation. This
led to the following behavior:
- 1st response arrives, reqId removed, waitMessage called
- 2nd ... last messages arrive, can't find reqId, so dropped
- waitMessage competes, re-adding reqId, but too late

The new version solves this by collecting SPRs explicitly

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
It seems lockfile was generated wrong or not updated,
broken in 76bff75

setting it to version used in nimbus build system based build

Signed-off-by: Csaba Kiraly <[email protected]>
pkg/ is specific to how old nimble handles
packages. Nimble 0.14 (with lock file support)
uses pkg2/

Anyway, pkg or pkg2 are not needed, better remove.

Signed-off-by: Csaba Kiraly <[email protected]>
Make dht compile with new new (0.14+) nimble.

Compilation with Nimble <0.14 can't work, as
the many of the libraries on the dependency
tree are not version tagged. As we can't change
that, two paths remain:
1, use nimbus-build-system (i.e. git submodules)
2, use lock files introduced in nimble 0.14+
First was already working, this fixes second

Signed-off-by: Csaba Kiraly <[email protected]>
this is a minimal implementation, with lots of
work still needed.

Signed-off-by: Csaba Kiraly <[email protected]>
WIP, still not sampling but going through all samples

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
- moving das.nim to root directory
nimble dependencies work strange ....

- adding das target in .nimble file

- adding missing tostring

- copy test-helper to main folder (otherwise compilation errors)

Signed-off-by: Csaba Kiraly <[email protected]>
Signed-off-by: Csaba Kiraly <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant