Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-46357: Add calibration pipelines for new IsrTaskLSST for LSSTComCam and LSSTComCamSim #274

Merged
merged 29 commits into from
Oct 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
fe83abf
Move legacy LSSTComCam calibration pipelines to LSSTComCamSim/legacy_…
erykoff Sep 24, 2024
306393a
Move legacy LSSTComCam calibration pipelines to LSSTComCam/legacy_202…
erykoff Sep 24, 2024
baf5097
Add new IsrTaskLSST pipeline for LSSTComCamSim bfk.
erykoff Sep 25, 2024
1610d9e
Add new IsrTaskLSST pipeline for LSSTComCamSim bias.
erykoff Sep 25, 2024
6725d7b
Add new IsrTaskLSST pipeline for LSSTComCamSim biasBootstrap.
erykoff Sep 25, 2024
a16a517
Add new IsrTaskLSST pipeline for LSSTComCamSim dark.
erykoff Sep 25, 2024
3f54ea2
Add new IsrTaskLSST pipeline for LSSTComCamSim darkBootstrap.
erykoff Sep 25, 2024
c571baf
Add new IsrTaskLSST pipeline for LSSTComCamSim defects.
erykoff Sep 25, 2024
993af63
Add new IsrTaskLSST pipeline for LSSTComCamSim flat.
erykoff Sep 25, 2024
1870248
Add new IsrTaskLSST pipeline for LSSTComCamSim flatBootstrap.
erykoff Sep 25, 2024
54b51bc
Add new IsrTaskLSST pipeline for LSSTComCamSim flat (single).
erykoff Sep 25, 2024
cc7cdd3
Add new IsrTaskLSST pipeline for LSSTComCamSim ptc plotting.
erykoff Sep 25, 2024
d6fa1c0
Add new IsrTaskLSST pipeline for LSSTComCamSim ptc.
erykoff Sep 25, 2024
a148eaf
Add new IsrTaskLSST pipeline for LSSTComCamSim cpSky.
erykoff Sep 25, 2024
2cd1355
Update pipeline testing for LSSTComCamSim with IsrTaskLSST.
erykoff Sep 25, 2024
66c0211
Add new IsrTaskLSST pipeline for LSSTComCam bfk.
erykoff Sep 25, 2024
4b56187
Add new IsrTaskLSST pipeline for LSSTComCam bias.
erykoff Sep 25, 2024
6ca3872
Add new IsrTaskLSST pipeline for LSSTComCam biasBootstrap.
erykoff Sep 25, 2024
9ba5f08
Add new IsrTaskLSST pipeline for LSSTComCam dark.
erykoff Sep 25, 2024
52a3746
Add new IsrTaskLSST pipeline for LSSTComCam darkBootstrap.
erykoff Sep 25, 2024
dc75662
Add new IsrTaskLSST pipeline for LSSTComCam defects.
erykoff Sep 25, 2024
faca3a3
Add new IsrTaskLSST pipeline for LSSTComCam flat.
erykoff Sep 25, 2024
cda9c9f
Add new IsrTaskLSST pipeline for LSSTComCam flatBootstrap.
erykoff Sep 25, 2024
67d5364
Add new IsrTaskLSST pipeline for LSSTComCam flat (single chip).
erykoff Sep 25, 2024
842b83e
Add new IsrTaskLSST pipeline for LSSTComCam linearizer.
erykoff Sep 25, 2024
260ba2b
Add new IsrTaskLSST pipeline for LSSTComCam ptc plotting.
erykoff Sep 25, 2024
bbd103e
Add new IsrTaskLSST pipeline for LSSTComCam ptc.
erykoff Sep 25, 2024
6c3e89d
Add new IsrTaskLSST pipeline for LSSTComCam cpSky.
erykoff Sep 25, 2024
ea8e4b6
Update pipeline tests for LSSTComCam IsrTaskLSST pipelines.
erykoff Sep 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions pipelines/LSSTComCam/cpBfk.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,17 @@
description: cp_pipe brighter-fatter kernel calibration construction.
description: LSSTComCam Brighter-Fatter kernel calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBfk.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBfkLSST.yaml
tasks:
cpBfkPtcExtract:
class: lsst.cp.pipe.ptc.PhotonTransferCurveExtractTask
config:
# Restrict the covariance range because we do not have
# dense PTCs for LSSTComCam.
maximumRangeCovariancesAstier: 8
efdSalIndex: 101
cpBfkPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
maximumRangeCovariancesAstier: 8
maximumRangeCovariancesAstierFullCovFit: 8
9 changes: 2 additions & 7 deletions pipelines/LSSTComCam/cpBias.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,4 @@
description: ComCam bias calibration construction
description: LSSTComCam bias calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBias.yaml
tasks:
cpBiasIsr:
class: lsst.ip.isr.IsrTask
config:
doDefect: false
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBiasLSST.yaml
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/cpBiasBootstrap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: LSSTComCam bootstrap bias calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBiasBootstrapLSST.yaml
11 changes: 2 additions & 9 deletions pipelines/LSSTComCam/cpDark.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,4 @@
description: ComCam dark calibration construction
description: LSSTComCam dark calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDark.yaml
tasks:
cpDarkIsr:
class: lsst.ip.isr.IsrTask
config:
doLinearize: false
doCrosstalk: false
doDefect: false
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDarkLSST.yaml
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/cpDarkBootstrap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: LSSTComCam bootstrap dark calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDarkBootstrapLSST.yaml
29 changes: 27 additions & 2 deletions pipelines/LSSTComCam/cpDefects.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,29 @@
description: cp_pipe DEFECT calibration construction from combined images.
description: cp_pipe LSSTComCam defect calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDefects.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDefectsLSST.yaml
tasks:
cpMeasureBiasDefects:
class: lsst.cp.pipe.MeasureDefectsCombinedTask
config:
badPixelsToFillColumnThreshold: 1000
saturatedPixelsToFillColumnThreshold: 20
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect these config options (for all three tasks) to differ between the three cameras?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have left them the same for now, because of cargo culting. But I could see these being different for LSSTCam based on defect research.

saturatedColumnDilationRadius: 2
cpMeasureDarkDefects:
class: lsst.cp.pipe.MeasureDefectsCombinedTask
config:
thresholdType: "VALUE"
darkCurrentThreshold: 3.0
badPixelsToFillColumnThreshold: 1000
saturatedPixelsToFillColumnThreshold: 20
saturatedColumnDilationRadius: 2
cpMeasureFlatDefects:
class: lsst.cp.pipe.MeasureDefectsCombinedWithFilterTask
config:
thresholdType: "VALUE"
fracThresholdFlat: 0.9
badPixelsToFillColumnThreshold: 1000
cpMergeDefects:
class: lsst.cp.pipe.MergeDefectsCombinedTask
config:
edgesAsDefects: true
15 changes: 2 additions & 13 deletions pipelines/LSSTComCam/cpFlat.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,4 @@
description: ComCam flat calibration construction
description: LSSTComCam flat calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlat.yaml
tasks:
cpFlatIsr:
class: lsst.ip.isr.IsrTask
config:
doLinearize: false
doCrosstalk: false
doDefect: false
cpFlatMeasure:
class: lsst.cp.pipe.CpFlatMeasureTask
config:
doVignette: false
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlatLSST.yaml
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/cpFlatBootstrap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: LSSTComCam bootstrap flat construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlatBootstrapSingleChipLSST.yaml
4 changes: 2 additions & 2 deletions pipelines/LSSTComCam/cpFlatSingleChip.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
description: cp_pipe FLAT calibration construction
description: LSSTComCam flat calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlatSingleChip.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlatSingleChipLSST.yaml
13 changes: 11 additions & 2 deletions pipelines/LSSTComCam/cpLinearizer.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
description: cp_pipe linearity calibration construction.
description: cp_pipe LSSTComCam linearizer calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpLinearizer.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpLinearizerLSST.yaml
tasks:
cpLinearizerPtcExtract:
class: lsst.cp.pipe.ptc.PhotonTransferCurveExtractTask
config:
efdSalIndex: 101
cpLinearizerSolve:
class: lsst.cp.pipe.LinearitySolveTask
config:
maxLinearAdu: 120000
2 changes: 1 addition & 1 deletion pipelines/LSSTComCam/cpPlotPtc.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
description: cp_pipe task to plot Photon Transfer Curve dataset
description: cp_pipe LSSTComCam task to plot Photon Transfer Curve dataset
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpPlotPtc.yaml
15 changes: 3 additions & 12 deletions pipelines/LSSTComCam/cpPtc.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,9 @@
description: ComCam Photon-Transfer Curve calibration construction.
description: LSSTComCam Photon-Transfer Curve calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpPtc.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpPtcLSST.yaml
tasks:
cpPtcIsr:
class: lsst.ip.isr.IsrTask
config:
doCrosstalk: false
doDefect: false
cpPtcExtract:
class: lsst.cp.pipe.ptc.PhotonTransferCurveExtractTask
config:
matchExposuresType: TIME
cpPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
ptcFitType: EXPAPPROXIMATION
efdSalIndex: 101
8 changes: 6 additions & 2 deletions pipelines/LSSTComCam/cpSky.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
description: Sky frame generation pipeline definition.
description: Sky frame generation for LSSTComCam pipeline.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpSky.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpSkyLSST.yaml
tasks:
cpSkyIsr:
class: lsst.ip.isr.IsrTaskLSST
config:
crosstalk.doQuadraticCrosstalkCorrection: false
cpSkyImage:
class: lsst.cp.pipe.CpSkyImageTask
config:
Expand Down
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpBfk.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: cp_pipe brighter-fatter kernel calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBfk.yaml
9 changes: 9 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpBias.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
description: ComCam bias calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBias.yaml
tasks:
cpBiasIsr:
class: lsst.ip.isr.IsrTask
config:
doDefect: false
11 changes: 11 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpDark.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
description: ComCam dark calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDark.yaml
tasks:
cpDarkIsr:
class: lsst.ip.isr.IsrTask
config:
doLinearize: false
doCrosstalk: false
doDefect: false
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpDefects.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: cp_pipe DEFECT calibration construction from combined images.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDefects.yaml
15 changes: 15 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpFlat.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
description: ComCam flat calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlat.yaml
tasks:
cpFlatIsr:
class: lsst.ip.isr.IsrTask
config:
doLinearize: false
doCrosstalk: false
doDefect: false
cpFlatMeasure:
class: lsst.cp.pipe.CpFlatMeasureTask
config:
doVignette: false
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpFlatSingleChip.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: cp_pipe FLAT calibration construction
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpFlatSingleChip.yaml
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpLinearizer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: cp_pipe linearity calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpLinearizer.yaml
4 changes: 4 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpPlotPtc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
description: cp_pipe task to plot Photon Transfer Curve dataset
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpPlotPtc.yaml
18 changes: 18 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpPtc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
description: ComCam Photon-Transfer Curve calibration construction.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpPtc.yaml
tasks:
cpPtcIsr:
class: lsst.ip.isr.IsrTask
config:
doCrosstalk: false
doDefect: false
cpPtcExtract:
class: lsst.cp.pipe.ptc.PhotonTransferCurveExtractTask
config:
matchExposuresType: TIME
cpPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
ptcFitType: EXPAPPROXIMATION
15 changes: 15 additions & 0 deletions pipelines/LSSTComCam/legacy_202409/cpSky.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
description: Sky frame generation pipeline definition.
instrument: lsst.obs.lsst.LsstComCam
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpSky.yaml
tasks:
cpSkyImage:
class: lsst.cp.pipe.CpSkyImageTask
config:
# These values roughly equate to the dimensions of a single detector.
# As noted below, the sizes are in millimeters, and correspond to a
# background image of 4096*4096 pixels (4096*0.01=40.96).
# TODO: Revise these values on DM-43321.
largeScaleBackground.xSize: 40.96 # in mm
largeScaleBackground.ySize: 40.96 # in mm
largeScaleBackground.pixelSize: 0.01 # in mm per pixel
33 changes: 31 additions & 2 deletions pipelines/LSSTComCamSim/cpBfk.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,33 @@
description: cp_pipe brighter-fatter kernel calibration construction.
description: LSSTComCamSim Brighter-Fatter kernel calibration construction.
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBfk.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBfkLSST.yaml
tasks:
cpBfkIsr:
class: lsst.ip.isr.IsrTaskLSST
config:
overscanCamera.defaultDetectorConfig.defaultAmpConfig.serialOverscanConfig.fitType: "MEAN"
overscanCamera.defaultDetectorConfig.defaultAmpConfig.doParallelOverscan: false
doCrosstalk: false
doLinearize: false
doDeferredCharge: false
doDefect: false
cpBfkPtcExtract:
class: lsst.cp.pipe.ptc.PhotonTransferCurveExtractTask
config:
maximumRangeCovariancesAstier: 8
useEfdPhotodiodeData: false
auxiliaryHeaderKeys: []
cpBfkPtcSolve:
class: lsst.cp.pipe.ptc.PhotonTransferCurveSolveTask
config:
maximumRangeCovariancesAstier: 8
maximumRangeCovariancesAstierFullCovFit: 8
# TODO DM-46439: This can be renamed back to cpBfkSolve when repos
# are cleaned up for the previous dimensionality error.
cpBfkSolveX:
class: lsst.cp.pipe.BrighterFatterKernelSolveTask
config:
useCovModelSample: true
python: |
config.covModelFluxSample = {"ALL_AMPS": 100000.0}
12 changes: 8 additions & 4 deletions pipelines/LSSTComCamSim/cpBias.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
description: ComCam bias calibration construction
description: LSSTComCamSim bias calibration construction
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBias.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBiasLSST.yaml
tasks:
cpBiasIsr:
class: lsst.ip.isr.IsrTask
class: lsst.ip.isr.IsrTaskLSST
config:
doDefect: false
overscanCamera.defaultDetectorConfig.defaultAmpConfig.serialOverscanConfig.fitType: "MEAN"
overscanCamera.defaultDetectorConfig.defaultAmpConfig.doParallelOverscan: false
doCrosstalk: false
doLinearize: false

11 changes: 11 additions & 0 deletions pipelines/LSSTComCamSim/cpBiasBootstrap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
description: LSSTComCamSim bootstrap bias calibration construction
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpBiasBootstrapLSST.yaml
tasks:
cpBiasBootstrapIsr:
class: lsst.ip.isr.IsrTaskLSST
config:
overscanCamera.defaultDetectorConfig.defaultAmpConfig.serialOverscanConfig.fitType: "MEAN"
overscanCamera.defaultDetectorConfig.defaultAmpConfig.doParallelOverscan: false
doCrosstalk: false
10 changes: 6 additions & 4 deletions pipelines/LSSTComCamSim/cpDark.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
description: ComCam dark calibration construction
description: LSSTComCamSim dark calibration construction
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDark.yaml
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDarkLSST.yaml
tasks:
cpDarkIsr:
class: lsst.ip.isr.IsrTask
class: lsst.ip.isr.IsrTaskLSST
config:
doLinearize: false
overscanCamera.defaultDetectorConfig.defaultAmpConfig.serialOverscanConfig.fitType: "MEAN"
overscanCamera.defaultDetectorConfig.defaultAmpConfig.doParallelOverscan: false
doCrosstalk: false
doLinearize: false
doDefect: false
11 changes: 11 additions & 0 deletions pipelines/LSSTComCamSim/cpDarkBootstrap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
description: LSSTComCam bootstrap dark calibration construction
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- location: $CP_PIPE_DIR/pipelines/_ingredients/cpDarkBootstrapLSST.yaml
tasks:
cpDarkBootstrapIsr:
class: lsst.ip.isr.IsrTaskLSST
config:
overscanCamera.defaultDetectorConfig.defaultAmpConfig.serialOverscanConfig.fitType: "MEAN"
overscanCamera.defaultDetectorConfig.defaultAmpConfig.doParallelOverscan: false
doCrosstalk: false
Loading
Loading