-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add two pipelines to be run on RA during OR3
- A fast observer-support pipeline matching what the single frame runner in the summit rapid analysis runs now - A commissiong validation pipeline that will help inform what changes to make the following night. Detector and Visit level tasks can be run in real time by e.g. Rapid Analysis. The rest of the pipeline is expected to be run as part of the 10am processing.
- Loading branch information
Showing
5 changed files
with
409 additions
and
41 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
191 changes: 191 additions & 0 deletions
191
pipelines/LSSTComCamSim/nightly-validation-ops-rehearsal-3.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| description: | | ||
| DRP-flavored pipeline to support validation | ||
| commissiong validation pipeline that will help inform | ||
| what changes to make the following night. Detector and Visit | ||
| level tasks can be run in real time by e.g. Rapid Analysis. | ||
| The rest of the pipeline is expected to be run as part of the | ||
| 10am processing. | ||
| instrument: lsst.obs.lsst.LsstComCamSim | ||
| imports: | ||
| - $DRP_PIPE_DIR/pipelines/_ingredients/LSSTComCamSim/DRP.yaml | ||
| tasks: | ||
| analyzePreSourceTableCore: | ||
| class: lsst.analysis.tools.tasks.SourceTableVisitAnalysisTask | ||
| config: | ||
| connections.data: preSourceTable_visit | ||
| connections.inputName: preSourceTable_visit | ||
| connections.outputName: preSourceTableCore | ||
| subsets: | ||
| # These 3 steps can be run in real time by Rapid Analysis Framework at USDF | ||
| step1: | ||
| subset: | ||
| - isr | ||
| - characterizeImage | ||
| - calibrate | ||
| - writePreSourceTable | ||
| - transformPreSourceTable | ||
| description: | | ||
| Detector level tasks. TO DO: requests include synthetic source injection, | ||
| subtractImages, detectAndMeasureDiaSources. | ||
| step2a: | ||
| subset: | ||
| - consolidatePreSourceTable | ||
| - consolidateVisitSummary | ||
| - analyzePreSourceTableCore | ||
| - catalogMatchPreVisit | ||
| - astrometricRefCatPreSourceVisit # configure for metrics only | ||
| # TO DO: Add photometricMatchPreVisit, photometricRefCatPreSourceVisit | ||
| description: Visit-level tasks | ||
| nightlyRollup: | ||
| subset: | ||
| - makePreliminaryCcdVisitTable | ||
| - makePreliminaryVisitTable | ||
| - preliminaryVisitCoverageAnalysis | ||
| description: | | ||
| Global tasks that can be run at end of night or multiple times during | ||
| the night, to get a summary of observations taken. | ||
| # The following N steps should be run once at the end of the night | ||
| step2b: | ||
| subset: | ||
| - analyzeMatchedPreVisitCore | ||
| - isolatedStarAssociation | ||
| - gbdesAstrometricFit | ||
| description: | | ||
| Tract-level tasks | ||
| Allowed data query constraints: tract | ||
| GbdesAstrometricFit and isolatedStarAssociation both use PreSources, | ||
| generated by consolidatePreSourceTable, for all visits that overlap a | ||
| tract. | ||
| gbdesAstrometricFit produces solutions per-tract, per-visit | ||
| isolatedStarAssociation produces solutions per-tract. | ||
| step2c: | ||
| subset: | ||
| - fgcmBuildFromIsolatedStars | ||
| - fgcmFitCycle | ||
| - fgcmOutputProducts | ||
| description: | | ||
| Global-level tasks that must not be run with any data query constraints | ||
| Can be run anytime after subset step2b. | ||
| Allowed data query constraints: instrument | ||
| step2d: | ||
| subset: | ||
| - finalizeCharacterization | ||
| - writeRecalibratedSourceTable | ||
| - transformSourceTable | ||
| - consolidateSourceTable | ||
| - updateVisitSummary | ||
| description: | | ||
| Visit-level tasks to re-estimate PSFs and apply | ||
| global calibrations to Source tables. | ||
| Allowed data query constraints: visit | ||
| step2e: | ||
| subset: | ||
| - makeCcdVisitTable | ||
| - makeVisitTable | ||
| description: | | ||
| Global-level tasks that must not be run with any data query constraints | ||
| Can be run anytime after subset step2d. | ||
| Allowed data query constraints: instrument | ||
| Tasks generate one data product per collection. | ||
| make[Ccd]VisitTable produces per-collection summary of the Visits | ||
| and CcdVisits. | ||
| step3: | ||
| subset: | ||
| - makeWarp | ||
| - selectDeepCoaddVisits | ||
| - assembleCoadd | ||
| - detection | ||
| - mergeDetections | ||
| - deblend | ||
| - measure | ||
| - mergeMeasurements | ||
| - forcedPhotCoadd | ||
| - transformObjectTable | ||
| - writeObjectTable | ||
| - consolidateObjectTable | ||
| - healSparsePropertyMaps | ||
| - selectGoodSeeingVisits | ||
| - templateGen | ||
| # analysis_tools tasks | ||
| - analyzeMatchedVisitCore | ||
| - analyzeObjectTableCore | ||
| - analyzeObjectTableExtended | ||
| - catalogMatchTract | ||
| - photometricCatalogMatch | ||
| - photometricRefCatObjectTract | ||
| - plotPropertyMapTract | ||
| - refCatObjectTract | ||
| - validateObjectTableCore | ||
| description: | | ||
| Patch and Tract-level tasks that make coadds and object tables that | ||
| can be run together, but only after the 'step1' and 'step2d' | ||
| subsets. | ||
| These should be run with explicit 'tract' constraints essentially all the | ||
| time, because otherwise quanta will be created for jobs with only partial | ||
| visit coverage. | ||
| step4: | ||
| subset: | ||
| - forcedPhotCcd | ||
| - forcedPhotDiffim | ||
| - getTemplate | ||
| - subtractImages | ||
| - detectAndMeasureDiaSources | ||
| - transformDiaSourceCat | ||
| - writeForcedSourceTable | ||
| description: | | ||
| Image subtraction and forced photometry Tasks that can be run together, | ||
| but only after the 'step1', 'step2d' and 'step3' subsets | ||
| These detector-level tasks should not be run with 'tract' or 'patch' as | ||
| part of the data ID expression if all reference catalogs or diffIm | ||
| templates that cover these detector-level quanta are desired. | ||
| step5: | ||
| subset: | ||
| - drpAssociation | ||
| - drpDiaCalculation | ||
| - forcedPhotCcdOnDiaObjects | ||
| - forcedPhotDiffOnDiaObjects | ||
| - transformForcedSourceTable | ||
| - consolidateAssocDiaSourceTable | ||
| - consolidateFullDiaObjectTable | ||
| - writeForcedSourceOnDiaObjectTable | ||
| - transformForcedSourceOnDiaObjectTable | ||
| - analyzeDiaSourceTableTract | ||
| description: | | ||
| ForcedSource and DIASource association Tasks that can be run together, | ||
| but only after the 'step1', 'step2', 'step3', and 'step4' subsets | ||
| This step includes patch-level aggregation Tasks. These should be run | ||
| with explicit 'tract' constraints in the data query, otherwise quanta | ||
| will be created for jobs with only partial visit coverage. | ||
| step6: | ||
| subset: | ||
| - consolidateDiaSourceTable | ||
| description: | | ||
| Tasks that can be run together, but only after the 'step1', 'step2', | ||
| 'step3', and 'step4' subsets | ||
| This step includes visit-level aggregation tasks. Running without tract | ||
| or patch in the data query is recommended, otherwise the outputs of | ||
| consolidateDiaSourceTable will not contain complete visits. | ||
| This subset is separate from step4 to signal to operators to pause to | ||
| assess unexpected image differencing failures before these aggregation | ||
| steps. Otherwise, if run in the same quantum graph, aggregated data | ||
| products (e.g. diaObjects) would not be created if one or more of the | ||
| expected inputs is missing. | ||
| step7: | ||
| subset: | ||
| - consolidateHealSparsePropertyMaps | ||
| - analyzeObjectTableSurveyCore | ||
| description: | | ||
| Tasks that should be run as the final step that require global inputs, | ||
| and can be run after the 'step3' subset. | ||
| This step has global aggregation tasks to run over all visits, detectors, | ||
| tracts, etc. This step should be run only with the instrument constraint | ||
| in the data query. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| description: | | ||
| A fast pipeline to be run on the rapid analysis framework at | ||
| the summit which feeds RubinTV for realtime observer feedback | ||
| instrument: lsst.obs.lsst.LsstComCamSim | ||
| imports: | ||
| - $DRP_PIPE_DIR/pipelines/_ingredients/LSSTComCamSim/DRP.yaml | ||
| tasks: | ||
| isr: | ||
| class: lsst.ip.isr.IsrTask | ||
| config: | ||
| # Turn off all slow steps in ISR for | ||
| doBrighterFatter: false | ||
| # TO DO: Replace with calibrateImage in a way that we can use the | ||
| # preSources downstream. | ||
| characterizeImage: | ||
| class: lsst.pipe.tasks.characterizeImage.CharacterizeImageTask | ||
| config: | ||
| doApCorr: false | ||
| doDeblend: false | ||
| doMaskStreaks: false | ||
| calibrate: | ||
| class: lsst.pipe.tasks.calibrate.CalibrateTask | ||
| config: | ||
| python: | | ||
| config.measurement.plugins.names = ['base_CircularApertureFlux', | ||
| 'base_PsfFlux', | ||
| 'base_NaiveCentroid', | ||
| 'base_CompensatedGaussianFlux', | ||
| 'base_LocalBackground', | ||
| 'base_SdssCentroid', | ||
| 'base_SdssShape', | ||
| 'base_Variance', | ||
| 'base_Jacobian', | ||
| 'base_PixelFlags', | ||
| 'base_GaussianFlux', | ||
| 'base_SkyCoord', | ||
| 'base_FPPosition', | ||
| 'base_ClassificationSizeExtendedness', | ||
| ] | ||
| config.measurement.slots.shape = "base_SdssShape" | ||
| config.measurement.slots.psfShape = "base_SdssShape_psf" | ||
| config.measurement.plugins['base_PixelFlags'].masksFpAnywhere.remove('STREAK') | ||
| config.measurement.plugins['base_PixelFlags'].masksFpCenter.remove('STREAK') | ||
| config.doApCorr = False | ||
| config.doDeblend = False | ||
| config.astrometry.sourceSelector["science"].doRequirePrimary = False | ||
| config.astrometry.sourceSelector["science"].doIsolated = False | ||
| # TO DO: DM-41189 Get pipeline that will attach an atool here from lauren | ||
| subsets: | ||
| step1: | ||
| subset: | ||
| - isr | ||
| - characterizeImage | ||
| - calibrate | ||
| # TO DO: configure and add writePreSourceTable, transformPreSourceTable | ||
| description: | | ||
| Per-detector tasks that can be run together | ||
| step2a: | ||
| subset: | ||
| - consolidateVisitSummary | ||
| # TO DO: configure and add consolidatePreSourceTable | ||
| nightlyRollup: | ||
| # can be run at end of night or on a timer | ||
| # Need to either clobber OR use a chained collection of timestamped runs | ||
| subset: | ||
| - makePreliminaryCcdVisitTable | ||
| - makePreliminaryVisitTable | ||
| - preliminaryVisitCoverageAnalysis | ||
| description: | | ||
| Global-level tasks |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.