Add `DEVICEPTR` annotations to data region in driver loop #145

awnawab · 2023-09-07T12:12:26Z

A small update to the DataOffload transformation to mark kernel array arguments as true device-pointers and skip address translation via the OpenACC map. This is needed when using the CUDA backend in the new FIELD_API. The new functionality has been tested here: https://github.com/ecmwf-ifs/ecwam/tree/naan-deviceptr.

NB: This is stacked on top of #142.

github-actions · 2023-09-07T12:16:05Z

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/145/index.html

codecov · 2023-09-07T12:43:10Z

Codecov Report

Merging #145 (3ac1f46) into main (ed7e624) will increase coverage by 0.00%.
Report is 24 commits behind head on main.
The diff coverage is 90.90%.

@@           Coverage Diff            @@
##             main     #145    +/-   ##
========================================
  Coverage   92.09%   92.09%            
========================================
  Files          89       90     +1     
  Lines       16537    16664   +127     
========================================
+ Hits        15229    15347   +118     
- Misses       1308     1317     +9

Flag	Coverage Δ
lint_rules	`96.22% <ø> (ø)`
loki	`92.04% <ø> (+0.01%)`	⬆️
transformations	`91.47% <90.90%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
transformations/transformations/data_offload.py	`92.85% <90.90%> (-0.25%)`	⬇️

... and 2 files with indirect coverage changes

reuterbal

Very nicely done and useful addition, many thanks!

Only real remark is related to naming: I'd prefer a marginally more descriptive naming of that option all the way through from CMake to transformation constructor. Suggestion would be something like assume_deviceptr but better variants do probably exist.

reuterbal · 2023-09-11T19:16:01Z

transformations/transformations/data_offload.py

@@ -27,13 +27,17 @@ class DataOffloadTransformation(Transformation):
    ----------
    remove_openmp : bool
        Remove any existing OpenMP pragmas inside the marked region.
+    deviceptr : bool


It's a little hard to judge from the brief option name what that implies. I would propose to follow the typical "start with a verb" naming scheme, e.g., by calling this assume_deviceptr? That implies imho also the requirement this imposes on the calling code to handle the offload.

Seconded, either assume_deviceptr or use_deviceptr or mark_as_deviceptr (or similar). And ideally propagate change up to cmake layer (explicit is better than implicit 😉 )

reuterbal · 2023-09-11T19:18:31Z

transformations/transformations/data_offload.py

+                if self.deviceptr:
+                    deviceptr = ''
+                    if inargs+outargs+inoutargs:
+                        deviceptr = f'deviceptr({", ".join(inargs+outargs+inoutargs)})'
+                    pragma = Pragma(keyword='acc', content=f'data {deviceptr}')


Suggested change

if self.deviceptr:

deviceptr = ''

if inargs+outargs+inoutargs:

deviceptr = f'deviceptr({", ".join(inargs+outargs+inoutargs)})'

pragma = Pragma(keyword='acc', content=f'data {deviceptr}')

if self.deviceptr:

offload_args = inargs + outargs + inoutargs

if offload_args:

deviceptr = f' deviceptr({", ".join(offload_args)})'

else:

deviceptr = ''

pragma = Pragma(keyword='acc', content=f'data{deviceptr}')

Purely a matter of taste, but I find this a little hard to read with the repeated contracted "sum".

Seconded (but yes, matter of taste), spaces around operators used to be a pylint rule (for a reason).

mlange05

I Agree with all of the previous suggestions, but otherwise GTG. I've not tested this explicitly with any regression tests, but might try CLOUDSC later myself. This should not stop this being merged tho, so GTG from me.

mlange05 · 2023-09-13T07:28:05Z

transformations/transformations/data_offload.py

+                if self.deviceptr:
+                    deviceptr = ''
+                    if inargs+outargs+inoutargs:
+                        deviceptr = f'deviceptr({", ".join(inargs+outargs+inoutargs)})'
+                    pragma = Pragma(keyword='acc', content=f'data {deviceptr}')


Seconded (but yes, matter of taste), spaces around operators used to be a pylint rule (for a reason).

mlange05 · 2023-09-13T07:28:34Z

transformations/transformations/data_offload.py

+                if self.deviceptr:
+                    deviceptr = ''
+                    if inargs+outargs+inoutargs:
+                        deviceptr = f'deviceptr({", ".join(inargs+outargs+inoutargs)})'
+                    pragma = Pragma(keyword='acc', content=f'data {deviceptr}')


Seconded (but yes, matter of taste), spaces around operators used to be a pylint rule (for a reason).

mlange05 · 2023-09-13T07:30:06Z

transformations/transformations/data_offload.py

@@ -27,13 +27,17 @@ class DataOffloadTransformation(Transformation):
    ----------
    remove_openmp : bool
        Remove any existing OpenMP pragmas inside the marked region.
+    deviceptr : bool


Seconded, either assume_deviceptr or use_deviceptr or mark_as_deviceptr (or similar). And ideally propagate change up to cmake layer (explicit is better than implicit 😉 )

awnawab · 2023-09-13T08:09:44Z

Thanks for the feedback @mlange05 and @reuterbal! I'll implement your recommendations later today. @mlange05 re trying this on CLOUDSC, you can see an example of this in action here: https://github.com/ecmwf-ifs/ecwam/tree/naan-deviceptr.

…pointers

reuterbal · 2023-09-13T15:16:21Z

transformations/transformations/data_offload.py

@@ -27,13 +27,17 @@ class DataOffloadTransformation(Transformation):
    ----------
    remove_openmp : bool
        Remove any existing OpenMP pragmas inside the marked region.
+    assume_deviceptr : bool


Docstring and actual argument are out of sync here now. Can we call the argument assume_deviceptr?

Another silly oversight. Thanks for spotting this!

reuterbal · 2023-09-13T15:17:22Z

scripts/loki_transform.py

@@ -92,6 +92,8 @@ def cli(debug):
              help='Run transformation to insert custom data offload regions.')
 @click.option('--remove-openmp', is_flag=True, default=False,
              help='Removes existing OpenMP pragmas in "!$loki data" regions.')
+@click.option('--assume_deviceptr', is_flag=True, default=False,


I think so far we used hyphen instead of underscore for options on the CLI

Nice catch! Corrected 👍

reuterbal

Waiting for CI to confirm but looks good to me now. Many thanks for applying the renaming and apologies that this has turned into such a rat's tail of driving the renaming through the layers...

awnawab requested review from mlange05 and reuterbal September 7, 2023 13:10

reuterbal force-pushed the naan-deviceptr branch from 10d3945 to 30b440f Compare September 8, 2023 19:08

reuterbal requested changes Sep 11, 2023

View reviewed changes

mlange05 approved these changes Sep 13, 2023

View reviewed changes

awnawab force-pushed the naan-deviceptr branch 2 times, most recently from c968f52 to b972d49 Compare September 13, 2023 13:31

DataOffloadTrafo: added option to declare kernel args as true device …

7be1935

…pointers

awnawab force-pushed the naan-deviceptr branch from b972d49 to 219a470 Compare September 13, 2023 14:17

reuterbal reviewed Sep 13, 2023

View reviewed changes

awnawab force-pushed the naan-deviceptr branch 2 times, most recently from 22d6e7a to a41f639 Compare September 13, 2023 15:19

Changed option name to assume_deviceptr

9bc5142

awnawab force-pushed the naan-deviceptr branch from a41f639 to 9bc5142 Compare September 13, 2023 15:31

reuterbal approved these changes Sep 13, 2023

View reviewed changes

Appease linter

3ac1f46

reuterbal merged commit 2322c1b into main Sep 14, 2023
12 checks passed

reuterbal deleted the naan-deviceptr branch September 14, 2023 08:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `DEVICEPTR` annotations to data region in driver loop #145

Add `DEVICEPTR` annotations to data region in driver loop #145

awnawab commented Sep 7, 2023 •

edited

Loading

github-actions bot commented Sep 7, 2023

codecov bot commented Sep 7, 2023 •

edited

Loading

reuterbal left a comment

reuterbal Sep 11, 2023

mlange05 Sep 13, 2023

reuterbal Sep 11, 2023

mlange05 Sep 13, 2023

mlange05 Sep 13, 2023

mlange05 left a comment

mlange05 Sep 13, 2023

mlange05 Sep 13, 2023

mlange05 Sep 13, 2023

awnawab commented Sep 13, 2023

reuterbal Sep 13, 2023

awnawab Sep 13, 2023

reuterbal Sep 13, 2023

awnawab Sep 13, 2023

reuterbal left a comment

Add DEVICEPTR annotations to data region in driver loop #145

Add DEVICEPTR annotations to data region in driver loop #145

Conversation

awnawab commented Sep 7, 2023 • edited Loading

github-actions bot commented Sep 7, 2023

codecov bot commented Sep 7, 2023 • edited Loading

Codecov Report

reuterbal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlange05 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awnawab commented Sep 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reuterbal left a comment

Choose a reason for hiding this comment

Add `DEVICEPTR` annotations to data region in driver loop #145

Add `DEVICEPTR` annotations to data region in driver loop #145

awnawab commented Sep 7, 2023 •

edited

Loading

codecov bot commented Sep 7, 2023 •

edited

Loading