Skip to content

Commit

Permalink
Merge pull request #700 from dirac-institute/cleanup_config
Browse files Browse the repository at this point in the history 
Further data reading cleanups
  • Loading branch information
@jeremykubica
jeremykubica authored Sep 10, 2024
2 parents a00cdef + 0b2ef2a commit f967efe
Show file tree
Hide file tree
Showing 6 changed files with 11 additions and 83 deletions.
40 changes: 2 additions & 38 deletions docs/source/user_manual/masking.rst
View file
Original file line number Diff line number Diff line change
@@ -1,42 +1,6 @@
Masking
=======

The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by:
The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by in the standardizers.

* applying the per-image mask
* applying a computed a global mask
* growing the computed mask footprint

As stated previously (see :ref:`Input Files`), KBMOD expects Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`.
These values can be overwritten by providing the :py:attr:`~kbmod.run_search.run_search.flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`. Futhermore, pixels with values that exceed a threshold value will be masked if a flux threshold is provided, see ``mask_threshold`` key in :ref:`Search Parameters`.

Global mask is computed for all images by counting how many individual masks have flagged a pixel for **any** reason in the list of allowed global masking flags, and masking it out if that count surpasses a given threshold. The number of times a pixel has to be flagged to be masked can be set by ``mask_num_images`` parameter (see :ref:`Search Parameters`) and the list of flags to use in global masking can be modified via the ``repeated_flag_keys`` parameter and is by default set to :py:attr:`~kbmod.run_search.run_search.default_repeated_flag_keys`.
Global masking is not applied when the list of allowed masking flags is empty.

After the per-image and global masks are applied to every image, KBMOD grows the mask to nearby pixels. The parameters ``mask_grow`` (see :ref:`Search Parameters`) determines the ammount of the growth.

The provided pixel bitmask uses the following mapping between flag and bitmask values, which corresponds to the Rubin Science Pipelines mask values:

================== =====
Key Value
================== =====
BAD 0
CLIPPED 9
CR 3
CROSSTALK 10
DETECTED 5
DETECTED_NEGATIVE 6
EDGE 4
INEXACT_PSF 11
INTRP 2
NOT_DEBLENDED 12
NO_DATA 8
REJECTED 13
SAT 1
SENSOR_EDGE 14
SUSPECT 7
UNMASKEDNAN 15
================== =====


To overwrite the default mapping set ``mask_bits_dict`` parameter, see :ref:`Search Parameters`.
TODO: Add more detail.
30 changes: 1 addition & 29 deletions docs/source/user_manual/search_params.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,21 +99,6 @@ This document serves to provide a quick overview of the existing parameters and
| ``lh_level`` | 10.0 | The minimum computed likelihood for an |
| | | object to be accepted. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_bits_dict`` | default_mask_bits_dict | A dictionary indicating which masked |
| | | values to consider invalid pixels. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_bit_vector`` | None | A bit vector to use for masking. If |
| | | provided, takes precedence over |
| | | ``flag_keys``. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_grow`` | 10 | Size, in pixels, the mask will be grown|
| | | by. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_num_images`` | 2 | Threshold for number of times a pixel |
| | | needs to be flagged in order to be |
| | | masked in global mask. |
| | | See :ref:`Masking` for more. |
+------------------------+-----------------------------+----------------------------------------+
| ``mask_threshold`` | None | The flux threshold over which a pixel |
| | | is automatically masked. ``None`` |
| | | means no flux-based masking. |
Expand All @@ -123,10 +108,6 @@ This document serves to provide a quick overview of the existing parameters and
| | | computed likelihood above this |
| | | threshold are rejected. |
+------------------------+-----------------------------+----------------------------------------+
| ``mjd_lims`` | None | Limits the search to images taken |
| | | within the given range (or ``None`` |
| | | for no filtering). |
+------------------------+-----------------------------+----------------------------------------+
| ``mom_lims`` | [35.5, 35.5, 2.0, 0.3, 0.3] | Thresholds for the moments of a |
| | | Gaussian fit to the flux, specified as |
| | | ``[xx, yy, xy, x, y]``. |
Expand All @@ -147,10 +128,6 @@ This document serves to provide a quick overview of the existing parameters and
| ``psf_val`` | 1.4 | The value for the standard deviation of|
| | | the point spread function (PSF). |
+------------------------+-----------------------------+----------------------------------------+
| ``psf_file`` | None | The path and filename of a separate |
| | | file containing the per-image PSFs. |
| | | See :ref:`PSF File` for more. |
+------------------------+-----------------------------+----------------------------------------+
| ``repeated_flag_keys`` | default_repeated_flag_keys | The flags used when creating the global|
| | | mask. See :ref:`Masking`. |
+------------------------+-----------------------------+----------------------------------------+
Expand Down Expand Up @@ -182,12 +159,7 @@ This document serves to provide a quick overview of the existing parameters and
| | | if: |
| | | * ``sum`` - (default) Per pixel sum |
| | | * ``median`` - A per pixel median |
| | | * ``mean`` - A per pixel mean |
+------------------------+-----------------------------+----------------------------------------+
| ``time_file`` | None | The path and filename of a separate |
| | | file containing the time when each |
| | | image was taken. See :ref:`Time File` |
| | | for more. |
| | | * ``mean`` - A per pixel mean |\
+------------------------+-----------------------------+----------------------------------------+
| ``track_filtered`` | False | A Boolean indicating whether to track |
| | | the filtered trajectories. Warning |
Expand Down
3 changes: 0 additions & 3 deletions src/kbmod/configuration.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,21 +41,18 @@ def __init__(self):
"known_obj_jpl": False,
"lh_level": 10.0,
"max_lh": 1000.0,
"mjd_lims": None,
"mom_lims": [35.5, 35.5, 2.0, 0.3, 0.3],
"num_obs": 10,
"output_suffix": "search",
"peak_offset": [2.0, 2.0],
"psf_val": 1.4,
"psf_file": None,
"res_filepath": None,
"result_filename": None,
"results_per_pixel": 8,
"save_all_stamps": False,
"sigmaG_lims": [25, 75],
"stamp_radius": 10,
"stamp_type": "sum",
"time_file": None,
"track_filtered": False,
"v_arr": [92.0, 526.0, 256],
"x_pixel_bounds": None,
Expand Down
8 changes: 8 additions & 0 deletions src/kbmod/standardizers/fits_standardizers/kbmodv05.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -155,9 +155,17 @@ def translateHeader(self):
standardizedHeader = {}
obs_datetime = Time(self.primary["DATE-AVG"], format="isot")
standardizedHeader["mjd"] = obs_datetime.mjd
standardizedHeader["mjd_mid"] = obs_datetime.mjd

# these are all optional things
standardizedHeader["filter"] = self.primary["FILTER"]

# If no observatory information is given, default to the Deccam data
# (Cerro Tololo Inter-American Observatory).
standardizedHeader["obs_lon"] = self.primary.get("OBS-LONG", 70.81489)
standardizedHeader["obs_lat"] = self.primary.get("OBS-LAT", -30.16606)
standardizedHeader["obs_elev"] = self.primary.get("OBS-ELEV", 2215.0)

return standardizedHeader

def _standardizeMask(self):
Expand Down
3 changes: 0 additions & 3 deletions tests/test_regression_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -258,8 +258,6 @@ def perform_search(im_stack, res_filename, default_psf):
"im_filepath": "./",
"res_filepath": None,
"result_filename": res_filename,
"time_file": None,
"psf_file": None,
"psf_val": default_psf,
"output_suffix": "",
"v_arr": v_arr,
Expand All @@ -268,7 +266,6 @@ def perform_search(im_stack, res_filename, default_psf):
"num_obs": num_obs,
"do_mask": True,
"lh_level": 25.0,
"mjd_lims": [52130.0, 62130.0],
"sigmaG_lims": [25, 75],
"mom_lims": [37.5, 37.5, 1.5, 1.0, 1.0],
"peak_offset": [3.0, 3.0],
Expand Down
10 changes: 0 additions & 10 deletions tests/test_work_unit.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,6 @@ def setUp(self):
self.config = SearchConfiguration()
self.config.set("im_filepath", "Here")
self.config.set("num_obs", self.num_images)
self.config.set("mask_bits_dict", {"A": 1, "B": 2})
self.config.set("repeated_flag_keys", None)

# Create a fake WCS
self.wcs = make_fake_wcs(200.6145, -7.7888, 500, 700, 0.00027)
Expand Down Expand Up @@ -275,8 +273,6 @@ def test_save_and_load_fits(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

# We throw an error if we try to overwrite a file with overwrite=False
self.assertRaises(FileExistsError, work.to_fits, file_path)
Expand Down Expand Up @@ -333,8 +329,6 @@ def test_save_and_load_fits_shard(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

# We throw an error if we try to overwrite a file with overwrite=False
self.assertRaises(FileExistsError, work.to_fits, file_path)
Expand Down Expand Up @@ -365,8 +359,6 @@ def test_save_and_load_fits_shard_lazy(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])
self.assertEqual(work2.im_stack.img_count(), 0)

work2.load_images()
Expand Down Expand Up @@ -413,8 +405,6 @@ def test_to_from_yaml(self):
# Check that we read in the configuration values correctly.
self.assertEqual(work2.config["im_filepath"], "Here")
self.assertEqual(work2.config["num_obs"], self.num_images)
self.assertDictEqual(work2.config["mask_bits_dict"], {"A": 1, "B": 2})
self.assertIsNone(work2.config["repeated_flag_keys"])

def test_image_positions_to_original_icrs_invalid_format(self):
work = WorkUnit(
Expand Down

0 comments on commit f967efe

Please sign in to comment.