Add optional flatfield correction to clean_flicker_noise

[stscijgbot-jp]

Issue JP-3829 was created on JIRA by David Law:

As discussed by Paul Goudfrooij in testing for NIRISS data, one of the limitations of the current 1/f correction is that it can fit and remove differences in illumination pattern due to the flatfield.  While this looks fine in the rate file stage, the problem can be apparent in the downstream cal files since the flatfield step will then put the illumination difference back into the data rather than removing it.

In order to work around this, it would be helpful to have an option in the clean_flicker_noise step to apply the flatfield to the image plane data prior to deriving the 1/f signal, and then take the flatfield correction back out again after subtracting the derived 1/f term.

[stscijgbot-jp]

Comment by David Law on JIRA:

For the time being, such an option use_flat_field would be False by default.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Paul Goudfrooij - do you have a sample data set that shows the problem that I can use for testing?

[stscijgbot-jp]

Comment by Paul Goudfrooij on JIRA:

Melanie Clarke - a good example dataset is jw01089001001_10101_00001_nis_uncal.fits. The best options for 1/f-correcting that dataset are method = 'median' and fit_by_channel = True. When doing that at the detector1 stage and then running calwebb_image2 on the resulting rate.fits file to create a 'corrected' flatfielded cal.fits file, and comparing that file with the original flatfielded cal.fits file, you'll see that the 'corrected' cal.fits file will have a skewed background with ~40% peak-to-peak variations. Which disappears when first dividing by the flat field, then 1/f correcting, then multiplying flat field back in.

I placed that data file in /user/goudfroo/to_melanie.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Perfect, thanks Paul Goudfrooij!

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

This change is straightforward, as long as a full-frame flat image is available.  It looks like that’s true for all instruments/modes except for NIRSpec, which generally requires a WCS to be assigned to a cutout of the full spectral image before a flat image can be composed, so I don't see an easy way to get this implemented for that instrument at the detector1 stage.

Here’s what I have implemented:

• Check the ‘apply_flat_field’ parameter. If True, check CRDS or overrides for FLAT reference file type.  For NIRSpec, this is N/A unless the user composes a matching flat image themselves and provides it with the ‘override_flat’ parameter.
• If a matching flat image is found, extract a subarray from it if needed.
• Replace any zeros or NaNs in the flat with a median value to avoid impacting the fit near flat artifacts.
• Divide the draft rate file by the flat image before creating the scene mask.
• Iterate over diff images:
  ** Divide diff by flat
  ** Fit and remove background, fit and remove noise
  ** Add background back in
  ** Multiply the flat back in

Draft PR is here: #9064

Paul Goudfrooij - let me know if you’d like to do some testing before merging.

Testing myself on Paul’s test case, I see improved corrections at ~1% level, especially in the lower right corner.  Without flat handling in the cleaning step, there is a glow in the lower right after flat correction; with flat handling, the glow goes away.  It’s not obvious by eye in the cal image for this data set, but it shows clearly in the residuals.  See attached image (jw01089001001_10101_00001_nis_cal_cfn.png).

Christian Hayes - let me know if NIRSpec is okay with this solution, or if you have ideas on how we might compose a full-frame flat from existing reference data.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Melanie Clarke that sounds fine for NIRSpec.  I don't think there will be a meaningful full-frame flat field for NIRSpec that will be applicable for this purpose.  

The NIRSpec D-flats are full frame and could do some flat-fielding, but there is a non-negligible wavelength dependence in the D-flats that requires knowing what wavelength of light is illuminating each pixel, so I don't think that will work.  The other flat components don't provide full-frame flat-fielding.

[stscijgbot-jp]

Comment by Paul Goudfrooij on JIRA:

Thanks Melanie Clarke - your testing of your implementation looks consistent with my tests done with my own code (which does the same thing as what you describe in your bullet list) so I don't think I need to do more testing. Thanks again.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Thanks for the quick responses, Chris and Paul!  I'll start moving the PR through reviews, then.