Cube mosaicing: Expand capabilities substantially #868

keflavich · 2023-03-09T13:51:34Z

the cube utils mosaic header maker wasn't checking the spectral dimension before

keflavich · 2023-03-13T18:11:13Z

Now branched off of #854

keflavich · 2023-03-13T21:00:34Z

This is largely reimplementing reproject/mosaicking/coadd.py, which is not good.

keflavich · 2023-03-13T21:03:33Z

Some of the functionality I want here needs to be pushed into coadd. Will split this into another PR.

keflavich · 2023-03-13T22:15:38Z

Dang, I thought I was getting somewhere.

Traceback (most recent call last):
  File "<ipython-input-1-349e0fbab013>", line 1, in <module>
    from aces.imaging.mosaic_12m import cs21_cube_mosaicing_minimal_test; cs21_cube_mosaicing_minimal_test()
  File "/orange/adamginsburg/ACES/reduction_ACES/aces/imaging/mosaic_12m.py", line 288, in cs21_cube_mosaicing_minimal_test
    result,_ = mosaic_cubes(cubes,
  File "/blue/adamginsburg/adamginsburg/repos/spectral-cube/spectral_cube/cube_utils.py", line 914, in mosaic_cubes
    output_array, output_footprint = reproject_and_coadd(
  File "/blue/adamginsburg/adamginsburg/repos/reproject/reproject/mosaicking/coadd.py", line 223, in reproject_and_coadd
    array, footprint = reproject_function(
  File "/blue/adamginsburg/adamginsburg/repos/astropy/astropy/utils/decorators.py", line 604, in wrapper
    return function(*args, **kwargs)
  File "/blue/adamginsburg/adamginsburg/repos/reproject/reproject/interpolation/high_level.py", line 121, in reproject_interp
    return _reproject_blocked(
  File "/blue/adamginsburg/adamginsburg/repos/reproject/reproject/utils.py", line 297, in _reproject_blocked
    output_array_dask = da.empty(shape_out, chunks=block_size or "auto")
  File "/blue/adamginsburg/adamginsburg/repos/dask/dask/array/wrap.py", line 60, in wrap_func_shape_as_first_arg
    parsed = _parse_wrap_args(func, args, kwargs, shape)
  File "/blue/adamginsburg/adamginsburg/repos/dask/dask/array/wrap.py", line 30, in _parse_wrap_args
    chunks = normalize_chunks(chunks, shape, dtype=dtype)
  File "/blue/adamginsburg/adamginsburg/repos/dask/dask/array/core.py", line 3106, in normalize_chunks
    raise ValueError(
ValueError: Chunks do not add up to shape. Got chunks=((4,), (5, 80, 101), (5, 100, 101)), shape=(4, 161, 152)

keflavich · 2023-03-13T22:43:52Z

It works with a simple test - but not with blocks yet

keflavich · 2023-09-03T18:31:20Z

I've been using this in production for a while. @e-koch, want to give it a once-over? Looks like I have some failing tests to fix, but I'm not sure if they're also failing on master

codecov · 2023-09-03T20:46:08Z

Codecov Report

Patch has no changes to coverable lines.

❗ Current head e8083b7 differs from pull request most recent head 0cfcd02. Consider uploading reports for the commit 0cfcd02 to get more accurate results

Files Changed	Coverage
spectral_cube/cube_utils.py	`0.00%`

📢 Thoughts on this report? Let us know!.

too

WIP / stash

…(but it doesn't work right yet)

… add a lot of verbosity

…should be semiautomatically excluded

keflavich · 2025-01-24T22:08:23Z

I added some tests in, and they revealed that something is going very weird with weighted mosaicking in at least one case (weights are going to nan when the data are fine).

cube-based reprojection (i.e., fully relying on reproject) is extrapolating in a region where it shouldn't, but that's literally an edge case, so I'm not so worried about it.

keflavich · 2025-01-24T23:28:55Z

I'm now pretty confident there's a bug in reproject, or at least in how it's interacting with cube reprojection.

For one channel in the test case, I get:

weights: [[[nan nan nan nan nan nan]
  [nan nan nan nan nan nan]
  [nan nan nan nan nan nan]
  [nan nan nan nan nan nan]]]
weights_in: [[[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]],

while for the exact same input parameters (same WCS, same everything except it's "array" instead of "weights":

array: [[[2. 2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2. 2.]]]
array_in: [[[2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]]

 [[2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]]

 [[2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]
  [2. 2. 2. 2. 2.]]]

Why are weights going to NaN?

keflavich · 2025-01-24T23:32:00Z

this raises a different issue, which is that weights are implicitly on the same grid as the data, but they're not actually required to be

keflavich · 2025-01-24T23:42:57Z

I fixed the bad weights issue in reproject/5cfa233 / astropy/reproject#474. However, I still don't understand how I encountered the error in the first place - the weight & cube headers are identical.

keflavich · 2025-01-25T02:10:04Z

Checking through errors... there are some horrendous ones. test_mosaic_cubes is resulting in super wrong values:

result.filled_data[:]
<Quantity [[[ 2.68308185e+185, -2.51563734e+252],
            [-9.03978759e+039,  3.04370599e+222],
            [-1.22330481e+051,  2.72896124e+167]],

           [[ 1.51095075e+196, -1.41665604e+263],
            [-5.09066610e+050,  1.71403263e+233],
            [-6.88891886e+061,  1.53678727e+178]],

           [[-2.40002193e+125,  8.57903219e+276],
            [-7.38140426e+135,  6.26917649e+173],
            [ 9.18684009e+002,  1.58752265e-008]],

           [[-1.09033779e+114,  3.89622506e+287],
            [-3.35339603e+124,  2.84810733e+162],
            [ 4.17361142e-009,  7.21216719e-020]]] K>

I think this indicates that an np.empty array is allocated and filled?

This could be associated with memmap; I can't tell yet (waiting on tests to finish).

keflavich · 2025-01-25T14:47:38Z

following up - it's not exclusive to memmap or mode:

FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes[True-100-False0] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes[True-100-False1] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes[True-1-True] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes[True-1-False] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-1-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-1-False-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-100-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-100-False-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-None-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[False-None-False-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-True-cube] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-False-cube] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-False-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-True-cube] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-False-cube] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-False-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-None-True-channel] - AssertionError:
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-None-False-channel] - AssertionError:

keflavich · 2025-01-26T17:05:10Z

Test failures are down to

FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-True-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-1-False-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-True-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-100-False-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-None-True-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_mosaic_cubes_spectral[True-None-False-channel] - RuntimeError: IOLoop is closed
FAILED spectral_cube/tests/test_regrid.py::test_cube_mosaic_weighted - TypeError: not all arguments converted during string formatting

keflavich · 2025-01-26T17:12:04Z

those configurations are impossible to read.

test_mosaic_cubes_spectral[True-None-False-channel]

means

use_memmap = False, data_adv = PosixPath('/private/var/folders/k_/7qh4l0nn72b7qgq15pkd4hw40000gt/T/pytest-of-adam/pytest-146/test_mosaic_cubes_spectral_Tru11/adv.fits'), use_dask = True, spectral_block_size = None, method = 'channel'

test_mosaic_cubes_spectral[True-None-True-channel]

means

use_memmap = True, data_adv = PosixPath('/private/var/folders/k_/7qh4l0nn72b7qgq15pkd4hw40000gt/T/pytest-of-adam/pytest-146/test_mosaic_cubes_spectral_Tru9/adv.fits'), use_dask = True, spectral_block_size = None, method = 'channel'

test_mosaic_cubes_spectral[True-100-False-channel]

means

use_memmap = False, data_adv = PosixPath('/private/var/folders/k_/7qh4l0nn72b7qgq15pkd4hw40000gt/T/pytest-of-adam/pytest-146/test_mosaic_cubes_spectral_Tru7/adv.fits'), use_dask = True, spectral_block_size = 100, method = 'channel'

so the error is just all use_dask examples with channel mode.

e-koch mentioned this pull request Mar 9, 2023

Bugfix: __len__ test for beams #852

Merged

keflavich force-pushed the cubewcs_mosaic branch from e8083b7 to 12ac463 Compare March 13, 2023 18:10

keflavich force-pushed the cubewcs_mosaic branch from 12ac463 to 3719766 Compare March 13, 2023 19:34

keflavich changed the title ~~Cube mosaicing: check spectral dimension~~ Cube mosaicing: Expand capabilities substantially Mar 13, 2023

keflavich force-pushed the cubewcs_mosaic branch from 11ccf6b to 85a8999 Compare March 13, 2023 20:14

keflavich mentioned this pull request Mar 13, 2023

Dask reproject #845

Draft

keflavich force-pushed the cubewcs_mosaic branch from 2f4371f to 944bd2e Compare April 4, 2023 17:28

keflavich mentioned this pull request Jan 30, 2024

Occasional weird zarr error #900

Open

keflavich mentioned this pull request May 30, 2024

Generalize reproject_and_coadd for N-dimensional data, and add option to specify blank pixel value and progress bar astropy/reproject#351

Merged

keflavich added 14 commits October 18, 2024 16:19

enhance the cube header determinator by checking the spectral dimension

7ee6bdd

too

add option to specify header

f988d15

WIP / stash

try to make convolution possible _and_ use memmap'd output

a9ee1ec

add missing import

2571693

need non-array truthiness

a73b1af

import

fe92cc9

import

94d5dde

just raise the exception as is, don't wrap it

eadaa19

refactor cube mosaicing to use reproject.coadd

11aa31c

refactor to allow for direct FITS output

8b26e47

bitpix dtype

d2929f8

getdata doesn't support mode

6665703

force overwrite

fe5fbb9

force overwrite

1d62314

keflavich added 18 commits October 18, 2024 16:19

add flush

7b39880

need to flush occasionally

aaba58a

allow channel-by-channel mosaicing instead of cube-by-cube

c231f0a

add progressbar & frequent flush callback

5385dae

fix some critical typos and enable resuming from partly-written file …

fae1837

…(but it doesn't work right yet)

restructure a bunch of things that are being done too expensively and…

ba2d131

… add a lot of verbosity

more verbosity & parallelism

70155bc

this is very verbose now, but still trying to make verbosity flexible

5ff9f33

allow weights to be included

a9adc6c

typo

7f1e866

add explanations of some choices

8c285e7

bugfix in weight handling

7f7b646

bugfix in weight handling

8dc64ea

when operating on lists of cubes, some may be out of range, and they …

737500a

…should be semiautomatically excluded

need more checks for cubes to be in range

5a73e87

have to account for more corner cases for missing data

0fde323

fix more bugs with keep

15ad955

refactor neighboring channel chooser

10e3ed3

keflavich force-pushed the cubewcs_mosaic branch from 0cfcd02 to 10e3ed3 Compare October 18, 2024 20:20

keflavich mentioned this pull request Jan 26, 2025

Continuum mosaic: Edge/noise problems between fields ACES-CMZ/reduction_ACES#411

Open

keflavich marked this pull request as draft January 26, 2025 17:37

keflavich mentioned this pull request Jan 26, 2025

WIP: Cubewcs mosaic and dask reproject #894

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cube mosaicing: Expand capabilities substantially #868

Cube mosaicing: Expand capabilities substantially #868

keflavich commented Mar 9, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Sep 3, 2023

codecov bot commented Sep 3, 2023 •

edited

Loading

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 25, 2025

keflavich commented Jan 25, 2025

keflavich commented Jan 26, 2025

keflavich commented Jan 26, 2025

Cube mosaicing: Expand capabilities substantially #868

Are you sure you want to change the base?

Cube mosaicing: Expand capabilities substantially #868

Conversation

keflavich commented Mar 9, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Mar 13, 2023

keflavich commented Sep 3, 2023

codecov bot commented Sep 3, 2023 • edited Loading

Codecov Report

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 24, 2025

keflavich commented Jan 25, 2025

keflavich commented Jan 25, 2025

keflavich commented Jan 26, 2025

keflavich commented Jan 26, 2025

codecov bot commented Sep 3, 2023 •

edited

Loading