Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] hpcSPAdes rebase, cleanup & implementation #1380

Open
wants to merge 102 commits into
base: main
Choose a base branch
from
Open

[DO NOT MERGE] hpcSPAdes rebase, cleanup & implementation #1380

wants to merge 102 commits into from

Commits on Sep 10, 2024

  1. CMAKE: Hook in MPI

    @eodus @asl
    eodus authored and asl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    f285a3c View commit details
    Browse the repository at this point in the history

  2. Add MPI runtime detection

    @asl
    asl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    673dd41 View commit details
    Browse the repository at this point in the history

  3. CMAKE: Install spades-hpc

    @eodus @asl
    eodus authored and asl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0219be9 View commit details
    Browse the repository at this point in the history

  4. Add mpi_console_writer log_writer interface impl

    @eodus @asl
    eodus authored and asl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    0682203 View commit details
    Browse the repository at this point in the history

  5. Factor out mpi log writers as well

    @asl
    asl committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    f4be241 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Add mpi partask core

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    c98a426 View commit details
    Browse the repository at this point in the history

  2. Add MPI test

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b4b9b1f View commit details
    Browse the repository at this point in the history

  3. User-friendly rank reporting

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    10396a4 View commit details
    Browse the repository at this point in the history

  4. Rudimentrary MPI stage manager & MPI stage

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    049271a View commit details
    Browse the repository at this point in the history

  5. Add TestMPI stage

    @eodus @asl
    eodus authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    22fd17f View commit details
    Browse the repository at this point in the history

  6. Separate out SPAdes and hpcSPAdes binaries

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b1485ed View commit details
    Browse the repository at this point in the history

  7. Even better SPAdes / hpcSPAdes separation

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    68e89b4 View commit details
    Browse the repository at this point in the history

  8. read_converter: changable chunk_num

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    5b8f287 View commit details
    Browse the repository at this point in the history

  9. MPI: sequence mapper notifier

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    903dff4 View commit details
    Browse the repository at this point in the history

  10. MPI mismatch_corrector

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    688fdcd View commit details
    Browse the repository at this point in the history

  11. MPI: Gap closer

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b6f8f22 View commit details
    Browse the repository at this point in the history

  12. MPI: pair_info_count

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    0490ca2 View commit details
    Browse the repository at this point in the history

  13. Move implementation to .cpp. Small cleanup while there

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    288cca5 View commit details
    Browse the repository at this point in the history

  14. Allow name ot be overriden

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    c8d6fb3 View commit details
    Browse the repository at this point in the history

  15. Factor out implementation into a separate class

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f3cb336 View commit details
    Browse the repository at this point in the history

  16. Simplify parallel processing

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    adea3d5 View commit details
    Browse the repository at this point in the history

  17. Simplify

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    1fba456 View commit details
    Browse the repository at this point in the history

  18. Ensure the streams are fresh

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ed7fe74 View commit details
    Browse the repository at this point in the history

  19. Add MPI construction stage

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    335c06f View commit details
    Browse the repository at this point in the history

  20. First real MPI construction step: collect k-mer coverage in parallel

    @asl
    asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    dfd6f07 View commit details
    Browse the repository at this point in the history

  21. add comment to DistanceEstimation

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    7739944 View commit details
    Browse the repository at this point in the history

  22. BinRead/Write for PairedIndices

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    60b8dd1 View commit details
    Browse the repository at this point in the history

  23. Distance Estimator MPI

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    e00a2a8 View commit details
    Browse the repository at this point in the history

  24. Distance Estimator MPI stage

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    7d193d9 View commit details
    Browse the repository at this point in the history

  25. DistanceEstimator MPI wrapper

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    a08fd05 View commit details
    Browse the repository at this point in the history

  26. MPI GraphCondensing

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    8ea28ca View commit details
    Browse the repository at this point in the history

  27. remove Seq from extansion index

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    abf09c4 View commit details
    Browse the repository at this point in the history

  28. remove friend from UnbranchingPathExtractor

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    cc1577f View commit details
    Browse the repository at this point in the history

  29. style fix

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    2daac3f View commit details
    Browse the repository at this point in the history

  30. MPI EarlyTipClipper

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    441b189 View commit details
    Browse the repository at this point in the history

  31. MPI Build Extension Index

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    94ae6f7 View commit details
    Browse the repository at this point in the history

  32. comments for MergeKmerFileTask

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    3334aa7 View commit details
    Browse the repository at this point in the history

  33. verification that after split the #buckets the same in all storages

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    cdb191d View commit details
    Browse the repository at this point in the history

  34. make kpostorage const

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    8bc46ff View commit details
    Browse the repository at this point in the history

  35. comment for release_all

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    96c2414 View commit details
    Browse the repository at this point in the history

  36. add comment for Splitter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    dde2892 View commit details
    Browse the repository at this point in the history

  37. resize one time in MergeKMers

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    a6f3691 View commit details
    Browse the repository at this point in the history

  38. VERIFY kmers in kmer storage are sorted and unique

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    d2ad40b View commit details
    Browse the repository at this point in the history

  39. fix warnings

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    9b2a16b View commit details
    Browse the repository at this point in the history

  40. MPI KMerCounter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f148be3 View commit details
    Browse the repository at this point in the history

  41. close stream after use

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f4fa3b6 View commit details
    Browse the repository at this point in the history

  42. constat for contig output stage

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    1955316 View commit details
    Browse the repository at this point in the history

  43. add sync for read conversion

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ff707c0 View commit details
    Browse the repository at this point in the history

  44. MPI ATtipClipper

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b737093 View commit details
    Browse the repository at this point in the history

  45. reduce code dublication

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    556666d View commit details
    Browse the repository at this point in the history

  46. fix pair_info_counter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    d689e1f View commit details
    Browse the repository at this point in the history

  47. detach edge index on load

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    fe27f20 View commit details
    Browse the repository at this point in the history

  48. detach edge index at the end of pair_info_counter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ed7c152 View commit details
    Browse the repository at this point in the history

  49. create lib Spades-MPI

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    61b8318 View commit details
    Browse the repository at this point in the history

  50. separate distEst

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    39b5ece View commit details
    Browse the repository at this point in the history

  51. separate construction_mpi

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    6849808 View commit details
    Browse the repository at this point in the history

  52. separate test_mpi

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    3591808 View commit details
    Browse the repository at this point in the history

  53. update distEst Arhitecture

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    3765393 View commit details
    Browse the repository at this point in the history

  54. make SeqMapperNot in GCMPI as in GC

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    df328d8 View commit details
    Browse the repository at this point in the history

  55. GapCloserBase

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    9ccb1df View commit details
    Browse the repository at this point in the history

  56. Separate MPI gap closer

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    d101a7b View commit details
    Browse the repository at this point in the history

  57. mismatch_corrector with functor

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    47c28e0 View commit details
    Browse the repository at this point in the history

  58. declarate MismatchShallNotPass to hpp

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b5817c4 View commit details
    Browse the repository at this point in the history

  59. separate MPI mismatch_correction

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    6c541d1 View commit details
    Browse the repository at this point in the history

  60. make pair_info_count consistent with master version

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    be0eeee View commit details
    Browse the repository at this point in the history

  61. functor for FillEdgePairFilter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    18acaa2 View commit details
    Browse the repository at this point in the history

  62. separate PairInfoCount MPI

    @olga24912 @asl
    olga24912 authored and asl committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    ccf2e80 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. MapLibFabric

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    520f6f1 View commit details
    Browse the repository at this point in the history

  2. Separate SeqMapperNotifier

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0ebac00 View commit details
    Browse the repository at this point in the history

  3. PerfectHashMapperBuilder MPI

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    28927de View commit details
    Browse the repository at this point in the history

  4. move mpi_kmer_index_builder to mpi dir

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    28da1f1 View commit details
    Browse the repository at this point in the history

  5. separate perfect_hash_map_builder_mpi

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    9735497 View commit details
    Browse the repository at this point in the history

  6. move kmer_extension_index_builder_mpi to mpi dir

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    26da67b View commit details
    Browse the repository at this point in the history

  7. Move partask and mpi stage to hpcSPAdes

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    932636d View commit details
    Browse the repository at this point in the history

  8. run_on_load stage type

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    790d295 View commit details
    Browse the repository at this point in the history

  9. namespace spaces

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    db9db37 View commit details
    Browse the repository at this point in the history

  10. rename function by code style

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ddfa32c View commit details
    Browse the repository at this point in the history

  11. delete mpi from local pipeline

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    3262e35 View commit details
    Browse the repository at this point in the history

  12. separate logger mpi

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    f3772ec View commit details
    Browse the repository at this point in the history

  13. Add time tracer annotations for MPI stage manager. Some cleanup here … 

    …and there
    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    751b782 View commit details
    Browse the repository at this point in the history

  14. Do not overwrite time traces from different nodes

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b4bf0e5 View commit details
    Browse the repository at this point in the history

  15. Cleanup

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0a0f75b View commit details
    Browse the repository at this point in the history

  16. Time tracing for partask

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b833f1c View commit details
    Browse the repository at this point in the history

  17. Better time tracing

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    eb09ca0 View commit details
    Browse the repository at this point in the history

  18. More information

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    864332a View commit details
    Browse the repository at this point in the history

  19. Annotate PathExtend

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    a20187f View commit details
    Browse the repository at this point in the history

  20. A bit more verbosity

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    d4d8a8d View commit details
    Browse the repository at this point in the history

  21. More events + some cleanups

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    70d957c View commit details
    Browse the repository at this point in the history

  22. call process lib func in Mismatch Corrector

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    7a01e58 View commit details
    Browse the repository at this point in the history

  23. fix: allreduce only in sync

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ea29617 View commit details
    Browse the repository at this point in the history

  24. fix: block size/NNodes

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    dbe03b4 View commit details
    Browse the repository at this point in the history

  25. sequence mapper notifier MPI in paired info counter

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ae78af7 View commit details
    Browse the repository at this point in the history

  26. PairedInfoCounter SeqMapNot MPI

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5dbc8fc View commit details
    Browse the repository at this point in the history

  27. split streams on allthreads cnt

    @olga24912 @asl
    olga24912 authored and asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b51eb6c View commit details
    Browse the repository at this point in the history

  28. Move MPI detection down to project

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    2d31adf View commit details
    Browse the repository at this point in the history

  29. Normalize include paths

    @asl
    asl committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    155b203 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Add spades.py bits

    @asl
    asl committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    220ddad View commit details
    Browse the repository at this point in the history

  2. Add hpcSPAdes to list of known projects

    @asl
    asl committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    f9ed062 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. Add SLURM executor

    @asl
    asl committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    fcaca11 View commit details
    Browse the repository at this point in the history

  2. Better job names

    @asl
    asl committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    fcf0c5c View commit details
    Browse the repository at this point in the history

  3. Fix some defaults

    @asl
    asl committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    9573ebf View commit details
    Browse the repository at this point in the history

  4. Run stuff via srun by default

    @asl
    asl committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    a7d7ecf View commit details
    Browse the repository at this point in the history