Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interleaved computation with communication in halo exchange #881

Draft
wants to merge 69 commits into
base: main
Choose a base branch
from

Conversation

lslusarczyk
Copy link
Contributor

No description provided.

Copy link
Contributor Author

@lslusarczyk lslusarczyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cyclic halos seems to be right decision. Two backends - not. Don't we just need a flag in segments (as you did)? Twoi backends seems to be not needed change.

Cyclic halos + special for_each (as you did) should be enough I think.

return;
}

assert(aligned(dr));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes no sense checking this for one range, not aligned can be two or more ranges, one is always aligned


assert(aligned(dr));

for (auto &s : dr::ranges::segments(dr)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably you wanted local_segments

void fence() { backend.fence(); }
void fence() { backend_.fence(); }

backend_type& backend(const std::size_t segment_index) { return backend_; }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add __attribute__((unused)) to these functoins, but I'm unsure this function is really needed


static constexpr std::size_t DUAL_SEGMENTS_PER_PROC = 2;

class DualMpiBackend {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the difference between DualMPiBackend and MpiBackend types? if none, please use one type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a leftover from when I was experimenting with changing some code in the backend, thanks for pointing it out


distribution distribution_;
std::size_t size_;
std::vector<dual_dv_segment<dual_distributed_vector>> segments_;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if these 2 lines are the only differences between ordinary and dual vector, then let's pass different template parameters and make segment type a template paramter

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

init() is different, also there are multiple pointers to local segments (std::vector<T *> datas_) and there are additional member functions that return the local segment currently suited for computation. Although I agree that the two classes might be merged, as long as it's still not fully developed I'd rather keep it split and merge them a bit later when it fully works.

for (std::size_t i = 0; i < DUAL_SEGMENTS_PER_PROC; i++) {
if (size_ > 0) {
datas_.push_back(static_cast<T *>(
backends_[i].allocate(data_size_ * sizeof(value_type))));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do I understand correctly that you allocate twice as many memory as normal distributed_vector of the same size?
if so - this is unacceptable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this isn't the case -- note the added multiplication by DUAL_SEGMENTS_PER_PROC:

std::size_t segment_count = comm_size * DUAL_SEGMENTS_PER_PROC;

This results in segment_size_ being smaller, so the memory footprint should only come from more halos.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants