-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Interleaved computation with communication in halo exchange #881
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cyclic halos seems to be right decision. Two backends - not. Don't we just need a flag in segments (as you did)? Twoi backends seems to be not needed change.
Cyclic halos + special for_each (as you did) should be enough I think.
return; | ||
} | ||
|
||
assert(aligned(dr)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes no sense checking this for one range, not aligned can be two or more ranges, one is always aligned
|
||
assert(aligned(dr)); | ||
|
||
for (auto &s : dr::ranges::segments(dr)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably you wanted local_segments
void fence() { backend.fence(); } | ||
void fence() { backend_.fence(); } | ||
|
||
backend_type& backend(const std::size_t segment_index) { return backend_; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add __attribute__((unused))
to these functoins, but I'm unsure this function is really needed
|
||
static constexpr std::size_t DUAL_SEGMENTS_PER_PROC = 2; | ||
|
||
class DualMpiBackend { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the difference between DualMPiBackend and MpiBackend types? if none, please use one type
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a leftover from when I was experimenting with changing some code in the backend, thanks for pointing it out
|
||
distribution distribution_; | ||
std::size_t size_; | ||
std::vector<dual_dv_segment<dual_distributed_vector>> segments_; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if these 2 lines are the only differences between ordinary and dual vector, then let's pass different template parameters and make segment type a template paramter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
init()
is different, also there are multiple pointers to local segments (std::vector<T *> datas_
) and there are additional member functions that return the local segment currently suited for computation. Although I agree that the two classes might be merged, as long as it's still not fully developed I'd rather keep it split and merge them a bit later when it fully works.
for (std::size_t i = 0; i < DUAL_SEGMENTS_PER_PROC; i++) { | ||
if (size_ > 0) { | ||
datas_.push_back(static_cast<T *>( | ||
backends_[i].allocate(data_size_ * sizeof(value_type)))); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do I understand correctly that you allocate twice as many memory as normal distributed_vector of the same size?
if so - this is unacceptable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this isn't the case -- note the added multiplication by DUAL_SEGMENTS_PER_PROC
:
std::size_t segment_count = comm_size * DUAL_SEGMENTS_PER_PROC;
This results in segment_size_
being smaller, so the memory footprint should only come from more halos.
No description provided.