Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace scipy.signal.medfilt in outlier_detection #8050

Closed
stscijgbot-jp opened this issue Nov 3, 2023 · 4 comments · Fixed by #8033
Closed

Replace scipy.signal.medfilt in outlier_detection #8050

stscijgbot-jp opened this issue Nov 3, 2023 · 4 comments · Fixed by #8033

Comments

@stscijgbot-jp
Copy link
Collaborator

Issue JP-3456 was created on JIRA by Brett Graham:

scipy.signal.medfilt has undefined behavior for nan inputs. These can exist during outlier_detection. Replace the medfilt with an equivalent median filter with defined behavior for nan inputs.

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Brett Graham on JIRA:

A PR is open and awaiting review with a fix to this issue: #8033

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Howard Bushouse on JIRA:

Fixed by #8033

@stscijgbot-jp
Copy link
Collaborator Author

Comment by Melanie Clarke on JIRA:

Tested for NIRSpec IFU with data from jw01335008001_02101, files 1-9, running spec3 with default outlier_detection with v1.12.5 and current dev version on my Mac.  Output data is in ███████████████████████████████████████████

There are differences in the output between the old and new versions, as expected. The changes are mostly at the edges of the IFU slices, where the flat is unreliable and there are many NaN pixels nearby good pixels.  There are a few notable artifacts in the final cube that appear in the v1.12.5 version that do not appear in the new version (see attached image for example, from the second wavelength slice in the cube).  

The fix appears to be working as expected for NIRSpec.

!jw01335008001_02101_s3d.png|width=463,height=206!

@stscijgbot-jp
Copy link
Collaborator Author

Comment by David Law on JIRA:

Also tested this for MIRI MRS and results look good.  Previously the median normalization array was ill behaved near large regions of NaN-valued pixels, and now looks cleaner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant