Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Test DHT with large number of nodes #57

Draft
wants to merge 61 commits into
base: master
Choose a base branch
from
Draft

WIP: Test DHT with large number of nodes #57

wants to merge 61 commits into from

Commits on May 9, 2023

  1. warn and skip test nodes on busy ports 

    TODO: We might want to still generate the right amount

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 9, 2023
    Configuration menu
    Copy the full SHA
    a86a5d6 View commit details
    Browse the repository at this point in the history

  2. increase node count to 1000 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 9, 2023
    Configuration menu
    Copy the full SHA
    3b71e50 View commit details
    Browse the repository at this point in the history

  3. wait more to have better logs 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 9, 2023
    Configuration menu
    Copy the full SHA
    a3092df View commit details
    Browse the repository at this point in the history

  4. log routing table size periodically

    @cskiraly
    cskiraly committed May 9, 2023
    Configuration menu
    Copy the full SHA
    591b270 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. improve tracing of message exchange 

    run e.g. as
```
nim c -r -d:debug -d:chronicles_enabled=on -d:chronicles_log_level=TRACE -d:chronicles_sinks=textlines[nocolors,stdout] tests/dht/test_providers.nim >err
```

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    4b49586 View commit details
    Browse the repository at this point in the history

  2. add 10ms of delay between node initializations 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    ab7edc0 View commit details
    Browse the repository at this point in the history

  3. change revalidateLoop to use [RevalidateMax/2, RevalidateMax] 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    7956898 View commit details
    Browse the repository at this point in the history

  4. Fake DatagramTransport: first compile 

    Override Chronos DatagramTransport with a fake one

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    b1957f4 View commit details
    Browse the repository at this point in the history

  5. store TransportAddress 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    e8b019b View commit details
    Browse the repository at this point in the history

  6. try to add global registry of endpoints 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    2eef69f View commit details
    Browse the repository at this point in the history

  7. make it compile with gcsafe 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    dcdef89 View commit details
    Browse the repository at this point in the history

  8. simplify code changes 

    no need to name it different, we can override DatagramTransport

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    20ff87b View commit details
    Browse the repository at this point in the history

  9. index by port 

    0.0.0.0 vs. 127.0.0.1 creates various issues
As a start, we can index by port only

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    6109cba View commit details
    Browse the repository at this point in the history

  10. add simple send functionality 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    fd317a5 View commit details
    Browse the repository at this point in the history

  11. use 127.0.0.1 for these tests 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    70c2d28 View commit details
    Browse the repository at this point in the history

  12. introduce waitResponse wrapper 

    initialize wait for response before sending request.
This is needed in cases where the response arrives before
moving to the next instruction, such as a directly connected
test.

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    eb08cbf View commit details
    Browse the repository at this point in the history

  13. add on/off for network emulator 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    349127f View commit details
    Browse the repository at this point in the history

  14. fixup: move sendRequest forward 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    8b24458 View commit details
    Browse the repository at this point in the history

  15. remove some echo lines 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    5e0cdd1 View commit details
    Browse the repository at this point in the history

  16. rename callback 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    cd1303f View commit details
    Browse the repository at this point in the history

  17. add recvFrom to fake DatagramTransport 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    3e0c283 View commit details
    Browse the repository at this point in the history

  18. remove some debug output

    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    23733c1 View commit details
    Browse the repository at this point in the history

  19. add 50ms simulated network latency 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 10, 2023
    Configuration menu
    Copy the full SHA
    ba0d671 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2023

  1. test: make nodecount and delays configurable 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    43d71d4 View commit details
    Browse the repository at this point in the history

  2. generalize p2p delay 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    dcb6c24 View commit details
    Browse the repository at this point in the history

  3. add stub for egress queuing 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    349eb01 View commit details
    Browse the repository at this point in the history

  4. introducing ChronoSim 

    ChronoSim, the simulation/emulation wrapper
around nim-chronos.

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    f58f8cb View commit details
    Browse the repository at this point in the history

  5. add logscope 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    eb74441 View commit details
    Browse the repository at this point in the history

  6. chronosim: use const to configure 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    d8b0421 View commit details
    Browse the repository at this point in the history

  7. chronosim: add timewarp 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 11, 2023
    Configuration menu
    Copy the full SHA
    527db66 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2023

  1. add random loss 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 13, 2023
    Configuration menu
    Copy the full SHA
    e934dd7 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. dht: waitMessage: expose timeout as parameter, keeping default 

    defults to ResponseTimeout as before

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed May 17, 2023
    Configuration menu
    Copy the full SHA
    c048362 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2023

  1. encoding: introducing type cipher=aes128 

    Introducing the cipher type to ease changing cipher.
No functional change

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    8f7d342 View commit details
    Browse the repository at this point in the history

  2. encoding: introducing the "nop" cipher 

    remove encryption for faster large scale tests.
Do not use for anything else!

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    50dd3cd View commit details
    Browse the repository at this point in the history

  3. changing async logic for nodes (multi) response 

    The previous version was based on the assumption that
that the future of waitMessage is triggered in the
event loop before the next message arrives. This seems
to be true with UDP, where a full cycle of the loop
handles only one datagram, but not true with other
transports, such as the in-memory emulation. This
led to the following behavior:
- 1st response arrives, reqId removed, waitMessage called
- 2nd ... last messages arrive, can't find reqId, so dropped
- waitMessage competes, re-adding reqId, but too late

The new version solves this by collecting SPRs explicitly

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    d496a47 View commit details
    Browse the repository at this point in the history

  4. chronosim: trace instead of info 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    45e2dea View commit details
    Browse the repository at this point in the history

  5. add long all-to-all test 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    17e55d4 View commit details
    Browse the repository at this point in the history

  6. fixing bearssl version in lockfile and in nimble file 

    It seems lockfile was generated wrong or not updated,
broken in 76bff75

setting it to version used in nimbus build system based build

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    e4862cd View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. fix imports: remove pkg/ 

    pkg/ is specific to how old nimble handles
packages. Nimble 0.14 (with lock file support)
uses pkg2/

Anyway, pkg or pkg2 are not needed, better remove.

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    830247f View commit details
    Browse the repository at this point in the history

  2. update .nimble and .lock files to work 

    Make dht compile with new new (0.14+) nimble.

Compilation with Nimble <0.14 can't work, as
the many of the libraries on the dependency
tree are not version tagged. As we can't change
that, two paths remain:
1, use nimbus-build-system (i.e. git submodules)
2, use lock files introduced in nimble 0.14+
First was already working, this fixes second

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    9665df6 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. just a comment on awaitedNodesMessages 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    6aa2707 View commit details
    Browse the repository at this point in the history

  2. add DHT storage (addValue/getValue) functionality 

    this is a minimal implementation, with lots of
work still needed.

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    c2bb22a View commit details
    Browse the repository at this point in the history

  3. addValue/getValue: add minimal test 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    575a341 View commit details
    Browse the repository at this point in the history

  4. DAS: add minimal DAS test 

    WIP, still not sampling but going through all samples

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    1ed7813 View commit details
    Browse the repository at this point in the history

  5. start sample downloads in parallel 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    15c45be View commit details
    Browse the repository at this point in the history

  6. add sampling instead of getting all segments 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    2a8c7e7 View commit details
    Browse the repository at this point in the history

  7. upload to DHT in parallel 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    479d640 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2023

  1. better sample generation for small samples 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    b277446 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. generate segments systematically 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    0083e1e View commit details
    Browse the repository at this point in the history

  2. das: factorize startSampling procedure 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    2076bcd View commit details
    Browse the repository at this point in the history

  3. DAS: introduce the sample async procedure 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    3af8b67 View commit details
    Browse the repository at this point in the history

  4. DAS: run sampling in parallel from each node 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    6601186 View commit details
    Browse the repository at this point in the history

  5. DAS: code cleanup - variables 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    c768586 View commit details
    Browse the repository at this point in the history

  6. DAS: code cleanup - imports 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    fe38377 View commit details
    Browse the repository at this point in the history

  7. DAS: rename to sampleDA 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    a05c0f8 View commit details
    Browse the repository at this point in the history

  8. DAS: add ratio to stats 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    7b5e9d1 View commit details
    Browse the repository at this point in the history

  9. DASL check samplesize <= blocksize 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    12c4c97 View commit details
    Browse the repository at this point in the history

  10. DAS: better defaults 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    8286d8f View commit details
    Browse the repository at this point in the history

  11. use ChronoSim version of nim-chronos 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    3afc006 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. making this work with nimble 

    - moving das.nim to root directory
nimble dependencies work strange ....

- adding das target in .nimble file

- adding missing tostring

- copy test-helper to main folder (otherwise compilation errors)

Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    d0c777c View commit details
    Browse the repository at this point in the history

  2. add minimal README 

    Signed-off-by: Csaba Kiraly <[email protected]>
    @cskiraly
    cskiraly committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    97d9b37 View commit details
    Browse the repository at this point in the history