Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for blocked offload of fields to device #83

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Add support for blocked offload of fields to device #83

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c525906
new copy_array interface, new test for blocked copies, new optional b…
wertysas Nov 13, 2024
258a906
added optional BLK_BOUNDS to all SYNC and GET_HOST_DATA routines
wertysas Dec 5, 2024
7be49a3
added support for partial allocations on device
wertysas Dec 16, 2024
8229490
clean of import + tests
wertysas Jan 30, 2025
12d9d54
replaced COPY_ARRAY with COPY_FUNC and removed field_data_module
wertysas Feb 3, 2025
c8eb929
added missing blk_bounds to gang methods, moved realloc logic to crea…
wertysas Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions python_utils/fieldType.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ def __init__ (self, **kwargs):
self.viewRank = self.rank-1
self.viewShape = ','.join ([':'] * (self.rank-1))
self.lbptr = ', '.join (list (map (lambda i: "LBOUNDS(" + str (i+1) + "):", range (0, self.rank))))
self.lbptr_blk = ', '.join([ f"LBOUNDS({i}):" for i in range(1, self.rank)] + ["BLK_BOUNDS(1):"])
self.hst_blk = ':, ' * (self.rank-1) + 'BLK_BOUNDS(1):BLK_BOUNDS(2)'
self.devptr_blk = ':, ' * (self.rank-1) + f'LBOUNDS({self.rank}):LBOUNDS({self.rank}) + BLK_BOUNDS(2)-BLK_BOUNDS(1)'
self.hasView = self.rank > 1
self.ganged = self.rank > 2

Expand Down
9 changes: 6 additions & 3 deletions src/buffer/field_RANKSUFF_gang_module.fypp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ PUBLIC :: ${ftn}$_GANG_${type}$

CONTAINS

SUBROUTINE ${ftn1}$_CREATE_DEVICE_DATA_WRAPPER_HELPER (SELF)
SUBROUTINE ${ftn1}$_CREATE_DEVICE_DATA_WRAPPER_HELPER (SELF, BLK_BOUNDS)
CLASS(${ftn1}$_WRAPPER_HELPER) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we not passing the BLK_BOUNDS through to SELF%PARENT%CREATE_DEVICE_DATA?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching this! I've fixed that now.


IF (ASSOCIATED (SELF%PARENT)) THEN
CALL SELF%PARENT%CREATE_DEVICE_DATA ()
Expand Down Expand Up @@ -93,11 +94,12 @@ CONTAINS
END SUBROUTINE

#:for what in ['HOST', 'DEVICE']
SUBROUTINE ${ftn1}$_GET_${what}$_DATA_WRAPPER_HELPER (SELF, MODE, PTR, QUEUE)
SUBROUTINE ${ftn1}$_GET_${what}$_DATA_WRAPPER_HELPER (SELF, MODE, PTR, QUEUE, BLK_BOUNDS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about passing through BLK_BOUNDS.

CLASS(${ftn1}$_WRAPPER_HELPER) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: MODE
${ft1.type}$, POINTER, INTENT(INOUT) :: PTR(${ft1.shape}$)
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
INTEGER(KIND=JPIM) :: LBOUNDS(${ft1.rank}$)

IF (ASSOCIATED (SELF%PARENT)) THEN
Expand Down Expand Up @@ -250,8 +252,9 @@ CONTAINS

END SUBROUTINE ${ftn}$_GANG_${type}$_DELETE_DEVICE_DATA

SUBROUTINE ${ftn}$_GANG_${type}$_CREATE_DEVICE_DATA (SELF)
SUBROUTINE ${ftn}$_GANG_${type}$_CREATE_DEVICE_DATA (SELF, BLK_BOUNDS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment here about passing through BLK_BOUNDS.

CLASS(${ftn}$_GANG_${type}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
INTEGER (KIND=JPIM) :: JFLD

CALL SELF%${ftn}$_${type}$%CREATE_DEVICE_DATA ()
Expand Down
21 changes: 16 additions & 5 deletions src/core/dev_alloc_module.fypp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,18 +80,24 @@ CONTAINS

#:if defined('USE_BUDDY_MALLOC') or defined('CUDA')

SUBROUTINE ${ft.name}$_DEV_ALLOCATE_HST (DEV, HST, MAP_DEVPTR)
SUBROUTINE ${ft.name}$_DEV_ALLOCATE_HST (DEV, HST, MAP_DEVPTR, BLK_BOUNDS)

${ft.type}$, POINTER :: DEV(${ft.shape}$)
${ft.type}$, POINTER :: HST(${ft.shape}$)
LOGICAL, INTENT(IN) :: MAP_DEVPTR
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

INTEGER :: ILBOUNDS (${ft.rank}$)
INTEGER :: IUBOUNDS (${ft.rank}$)

ILBOUNDS = LBOUND (HST)
IUBOUNDS = UBOUND (HST)

IF ( PRESENT(BLK_BOUNDS) ) THEN
ILBOUNDS(${ft.rank}$) = BLK_BOUNDS(1)
IUBOUNDS(${ft.rank}$) = BLK_BOUNDS(2)
ENDIF

CALL ${ft.name}$_DEV_ALLOCATE_DIM (DEV, UBOUNDS=IUBOUNDS, LBOUNDS=ILBOUNDS, MAP_DEVPTR=MAP_DEVPTR)

END SUBROUTINE ${ft.name}$_DEV_ALLOCATE_HST
Expand Down Expand Up @@ -174,20 +180,25 @@ END SUBROUTINE ${ft.name}$_DEV_DEALLOCATE

#:else

SUBROUTINE ${ft.name}$_DEV_ALLOCATE_HST (DEV, HST, MAP_DEVPTR)
SUBROUTINE ${ft.name}$_DEV_ALLOCATE_HST (DEV, HST, MAP_DEVPTR, BLK_BOUNDS)

USE FIELD_STATISTICS_MODULE

${ft.type}$, POINTER :: DEV(${ft.shape}$)
${ft.type}$, POINTER :: HST(${ft.shape}$)
LOGICAL, INTENT(IN) :: MAP_DEVPTR
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

IF ( PRESENT(BLK_BOUNDS) ) THEN
ALLOCATE (DEV (${ ' '.join([f'LBOUND (HST, {i}):UBOUND (HST, {i}),' for i in range (1, ft.rank)]) + 'BLK_BOUNDS(1):BLK_BOUNDS(2)'}$))
ELSE
#if __INTEL_COMPILER == 1800 && __INTEL_COMPILER_UPDATE == 5
! Bug with Intel 18.0.5.274
ALLOCATE (DEV (${ ', '.join (map (lambda i: 'LBOUND (HST, ' + str (i) + '):UBOUND (HST,' + str (i) + ')', range (1, ft.rank+1))) }$))
! Bug with Intel 18.0.5.274
ALLOCATE (DEV (${ ', '.join (map (lambda i: 'LBOUND (HST, ' + str (i) + '):UBOUND (HST,' + str (i) + ')', range (1, ft.rank+1))) }$))
#else
ALLOCATE (DEV, MOLD=HST)
ALLOCATE (DEV, MOLD=HST)
#endif
ENDIF

!$acc enter data create (DEV)

Expand Down
4 changes: 2 additions & 2 deletions src/core/field_RANKSUFF_data_module.fypp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,9 @@ CONTAINS
INTEGER (KIND=JPIM), INTENT (IN) :: KDIR
INTEGER (KIND=JPIM), OPTIONAL, INTENT (IN) :: QUEUE

PROCEDURE (${ftn}$_COPY_INTF), POINTER :: FUNC
PROCEDURE (${ftn}$_COPY_INTF), POINTER :: FUNC

FUNC => ${ftn}$_COPY_FUNC (HST, DEV)
FUNC => ${ftn}$_COPY_FUNC (HST, DEV)

CALL FUNC (HST, DEV, MAP_DEVPTR, KDIR, QUEUE)

Expand Down
118 changes: 77 additions & 41 deletions src/core/field_RANKSUFF_module.fypp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,17 +426,32 @@ CONTAINS

END SUBROUTINE ${ftn}$_WIPE_OBJECT

SUBROUTINE ${ftn}$_COPY_DATA (SELF, KDIR, QUEUE)
SUBROUTINE ${ftn}$_COPY_DATA (SELF, KDIR, QUEUE, BLK_BOUNDS)

USE FIELD_ABORT_MODULE

CLASS(${ftn}$) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: KDIR
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

INTEGER(KIND=JPIM) :: LB, UB
${ft.type}$, POINTER :: HST_BLK(${ft.shape}$) => NULL()
REAL :: START, FINISH

CALL CPU_TIME(START)
CALL SELF%COPY_FUNC (SELF%PTR, SELF%DEVPTR, SELF%MAP_DEVPTR, KDIR, QUEUE)
CALL CPU_TIME(FINISH)


IF ( .NOT. PRESENT(BLK_BOUNDS) ) THEN
CALL CPU_TIME(START)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These timers should be around this entire conditional.

CALL SELF%COPY_FUNC (SELF%PTR, SELF%DEVPTR, SELF%MAP_DEVPTR, KDIR, QUEUE)
CALL CPU_TIME(FINISH)
ELSE
LB = LBOUND(SELF%PTR, ${ft.rank}$)
UB = UBOUND(SELF%PTR, ${ft.rank}$)
IF ( BLK_BOUNDS(1) < LB .OR. BLK_BOUNDS(2) > UB ) THEN
CALL FIELD_ABORT("BLOCK DIMENSIONS ARE OUT OF RANGE")
END IF
HST_BLK => SELF%PTR(${ft.hst_blk}$)
CALL SELF%COPY_FUNC(HST_BLK, SELF%DEVPTR, SELF%MAP_DEVPTR, KDIR, QUEUE)
END IF
IF (KDIR == NH2D) THEN
CALL SELF%STATS%INC_CPU_TO_GPU_TRANSFER(START, FINISH)
ELSE IF (KDIR == ND2H) THEN
Expand All @@ -445,32 +460,37 @@ CONTAINS

END SUBROUTINE ${ftn}$_COPY_DATA

SUBROUTINE ${ftn}$_GET_HOST_DATA (SELF, MODE, PTR, QUEUE)
SUBROUTINE ${ftn}$_GET_HOST_DATA (SELF, MODE, PTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: MODE

${ft.type}$, POINTER, INTENT(INOUT) :: PTR(${ft.shape}$)
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

INTEGER(KIND=JPIM) :: LBOUNDS(${ft.rank}$)

LBOUNDS=LBOUND(SELF%PTR)
IF (IAND (SELF%GET_STATUS (), NHSTFRESH) == 0) THEN
CALL SELF%COPY_DATA (ND2H, QUEUE)
CALL SELF%COPY_DATA (ND2H, QUEUE, BLK_BOUNDS=BLK_BOUNDS)
CALL SELF%SET_STATUS (IOR (SELF%GET_STATUS (), NHSTFRESH))
ENDIF
PTR (${ft.lbptr}$) => SELF%PTR (${','.join(':' for _ in range(ft.rank))}$)
IF ( PRESENT(BLK_BOUNDS) ) THEN
PTR ( ${ft.lbptr_blk}$) => SELF%PTR (${ft.hst_blk}$)
ELSE
PTR (${ft.lbptr}$) => SELF%PTR (${','.join(':' for _ in range(ft.rank))}$)
END IF
IF (IAND (MODE, NWR) /= 0) THEN
CALL SELF%SET_STATUS (IAND (SELF%GET_STATUS (), NOT (NDEVFRESH)))
ENDIF

END SUBROUTINE ${ftn}$_GET_HOST_DATA

SUBROUTINE ${ftn}$_OWNER_GET_HOST_DATA (SELF, MODE, PTR, QUEUE)
SUBROUTINE ${ftn}$_OWNER_GET_HOST_DATA (SELF, MODE, PTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$_OWNER) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: MODE
${ft.type}$, POINTER, INTENT(INOUT) :: PTR(${ft.shape}$)
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

IF(SELF%GET_STATUS ()==UNALLOCATED)THEN
CALL SELF%CREATE_HOST_DATA ()
Expand All @@ -479,68 +499,79 @@ CONTAINS
CALL SELF%SET_STATUS (NHSTFRESH)
ENDIF
ENDIF
CALL SELF%${ftn}$_GET_HOST_DATA(MODE, PTR, QUEUE)
CALL SELF%${ftn}$_GET_HOST_DATA(MODE, PTR, QUEUE=QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_OWNER_GET_HOST_DATA

SUBROUTINE ${ftn}$_GET_HOST_DATA_RDONLY (SELF, PPTR, QUEUE)
SUBROUTINE ${ftn}$_GET_HOST_DATA_RDONLY (SELF, PPTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
${ft.type}$, POINTER, INTENT(INOUT) :: PPTR(${ft.shape}$)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL SELF%GET_HOST_DATA (NRD, PPTR, QUEUE)
CALL SELF%GET_HOST_DATA (NRD, PPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_GET_HOST_DATA_RDONLY

SUBROUTINE ${ftn}$_SYNC_HOST_RDONLY (SELF, QUEUE)
SUBROUTINE ${ftn}$_SYNC_HOST_RDONLY (SELF, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
${ft.type}$, POINTER :: ZPTR(${ft.shape}$)

CALL SELF%GET_HOST_DATA_RDONLY (ZPTR, QUEUE)
CALL SELF%GET_HOST_DATA_RDONLY (ZPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_SYNC_HOST_RDONLY

SUBROUTINE ${ftn}$_GET_HOST_DATA_RDWR (SELF, PPTR, QUEUE)
SUBROUTINE ${ftn}$_GET_HOST_DATA_RDWR (SELF, PPTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
${ft.type}$, POINTER, INTENT(INOUT) :: PPTR(${ft.shape}$)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL SELF%GET_HOST_DATA (IOR (NRD, NWR), PPTR, QUEUE)
CALL SELF%GET_HOST_DATA (IOR (NRD, NWR), PPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_GET_HOST_DATA_RDWR

SUBROUTINE ${ftn}$_SYNC_HOST_RDWR (SELF, QUEUE)
SUBROUTINE ${ftn}$_SYNC_HOST_RDWR (SELF, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
${ft.type}$, POINTER :: ZPTR(${ft.shape}$)

CALL SELF%GET_HOST_DATA_RDWR (ZPTR, QUEUE)
CALL SELF%GET_HOST_DATA_RDWR (ZPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_SYNC_HOST_RDWR

SUBROUTINE ${ftn}$_CREATE_DEVICE_DATA (SELF)
SUBROUTINE ${ftn}$_CREATE_DEVICE_DATA (SELF, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL DEV_ALLOCATE_HST (DEV=SELF%DEVPTR, HST=SELF%PTR, MAP_DEVPTR=SELF%MAP_DEVPTR)
CALL DEV_ALLOCATE_HST (DEV=SELF%DEVPTR, HST=SELF%PTR, MAP_DEVPTR=SELF%MAP_DEVPTR, BLK_BOUNDS=BLK_BOUNDS)
END SUBROUTINE

SUBROUTINE ${ftn}$_GET_DEVICE_DATA (SELF, MODE, PTR, QUEUE)
SUBROUTINE ${ftn}$_GET_DEVICE_DATA (SELF, MODE, PTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: MODE
${ft.type}$, POINTER, INTENT(INOUT) :: PTR(${ft.shape}$)
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

INTEGER(KIND=JPIM) :: LBOUNDS(${ft.rank}$)

LBOUNDS=LBOUND(SELF%PTR)
IF (.NOT. ASSOCIATED (SELF%DEVPTR)) THEN
CALL SELF%CREATE_DEVICE_DATA
CALL SELF%CREATE_DEVICE_DATA(BLK_BOUNDS=BLK_BOUNDS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should always call CREATE_DEVICE_DATA and inside that routine check whether the device allocation is big enough for the transfer we want to do. The current implementation will fail if we've done a "blocked" transfer for a field and then try and do a non-blocked transfer without wiping the device memory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I have moved the logic to the CREATE_DEVICE_DATA routine now and added the missing delete.

ENDIF
IF (IAND (SELF%GET_STATUS (), NDEVFRESH) == 0) THEN
CALL SELF%COPY_DATA (NH2D, QUEUE)
CALL SELF%COPY_DATA (NH2D, QUEUE, BLK_BOUNDS=BLK_BOUNDS)
CALL SELF%SET_STATUS (IOR (SELF%GET_STATUS (), NDEVFRESH))
ENDIF
PTR (${ft.lbptr}$) => SELF%DEVPTR (${','.join(':' for _ in range(ft.rank))}$)
IF ( PRESENT(BLK_BOUNDS) ) THEN
PTR ( ${ft.lbptr_blk}$) => SELF%DEVPTR (${ft.devptr_blk}$)
ELSE
PTR (${ft.lbptr}$) => SELF%DEVPTR (${','.join(':' for _ in range(ft.rank))}$)
END IF
IF (IAND (MODE, NWR) /= 0) THEN
CALL SELF%SET_STATUS (IAND (SELF%GET_STATUS (), NOT (NHSTFRESH)))
ENDIF
Expand Down Expand Up @@ -573,65 +604,70 @@ CONTAINS

END FUNCTION
#:endif

SUBROUTINE ${ftn}$_OWNER_GET_DEVICE_DATA (SELF, MODE, PTR, QUEUE)
SUBROUTINE ${ftn}$_OWNER_GET_DEVICE_DATA (SELF, MODE, PTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$_OWNER) :: SELF
INTEGER (KIND=JPIM), INTENT(IN) :: MODE
${ft.type}$, POINTER, INTENT(INOUT) :: PTR(${ft.shape}$)
INTEGER (KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

IF(SELF%GET_STATUS ()==UNALLOCATED)THEN
CALL SELF%CREATE_HOST_DATA ()
IF (SELF%HAS_INIT_VALUE) THEN
CALL SELF%CREATE_DEVICE_DATA
CALL SELF%CREATE_DEVICE_DATA(BLK_BOUNDS=BLK_BOUNDS)
SELF%PTR=SELF%INIT_VALUE
CALL SELF%SET_STATUS (NHSTFRESH)
ENDIF
ENDIF
CALL SELF%${ftn}$_GET_DEVICE_DATA(MODE, PTR, QUEUE)
CALL SELF%${ftn}$_GET_DEVICE_DATA(MODE, PTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_OWNER_GET_DEVICE_DATA

SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDONLY (SELF, PPTR, QUEUE)
SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDONLY (SELF, PPTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
${ft.type}$, POINTER, INTENT(INOUT) :: PPTR(${ft.shape}$)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL SELF%GET_DEVICE_DATA (NRD, PPTR, QUEUE)
CALL SELF%GET_DEVICE_DATA (NRD, PPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDONLY

SUBROUTINE ${ftn}$_GET_DEVICE_DATA_WRONLY (SELF, PPTR, QUEUE)
SUBROUTINE ${ftn}$_GET_DEVICE_DATA_WRONLY (SELF, PPTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
${ft.type}$, POINTER, INTENT(INOUT) :: PPTR(${ft.shape}$)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL SELF%SET_STATUS (IOR (SELF%GET_STATUS (), NDEVFRESH))
CALL SELF%GET_DEVICE_DATA (NWR, PPTR, QUEUE)
CALL SELF%GET_DEVICE_DATA (NWR, PPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_GET_DEVICE_DATA_WRONLY

SUBROUTINE ${ftn}$_SYNC_DEVICE_RDONLY (SELF, QUEUE)
SUBROUTINE ${ftn}$_SYNC_DEVICE_RDONLY (SELF, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
${ft.type}$, POINTER :: ZPTR(${ft.shape}$)

CALL SELF%GET_DEVICE_DATA_RDONLY (ZPTR, QUEUE)
CALL SELF%GET_DEVICE_DATA_RDONLY (ZPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_SYNC_DEVICE_RDONLY

SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDWR (SELF, PPTR, QUEUE)
SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDWR (SELF, PPTR, QUEUE, BLK_BOUNDS)
CLASS(${ftn}$) :: SELF
${ft.type}$, POINTER, INTENT(INOUT) :: PPTR(${ft.shape}$)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE

CALL SELF%GET_DEVICE_DATA (IOR (NRD, NWR), PPTR, QUEUE)
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)

CALL SELF%GET_DEVICE_DATA (IOR (NRD, NWR), PPTR, QUEUE, BLK_BOUNDS=BLK_BOUNDS)

END SUBROUTINE ${ftn}$_GET_DEVICE_DATA_RDWR

SUBROUTINE ${ftn}$_SYNC_DEVICE_RDWR (SELF, QUEUE)
SUBROUTINE ${ftn}$_SYNC_DEVICE_RDWR (SELF, QUEUE, BLK_BOUNDS)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to pass the BLK_BOUNDS through here too.

CLASS(${ftn}$) :: SELF
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: QUEUE
INTEGER(KIND=JPIM), OPTIONAL, INTENT(IN) :: BLK_BOUNDS(2)
${ft.type}$, POINTER :: ZPTR(${ft.shape}$)

CALL SELF%GET_DEVICE_DATA_RDWR (ZPTR, QUEUE)
Expand Down
Loading
Loading