How to kill selfcal worker and get away with it

[KshitijT]

Discussion thread on redesigning the selfcal worker.

[KshitijT]

Currently the selfcal worker does these tasks:

1. Image cross/self-calibrated images with wsclean.
2. Calibrate datasets with CubiCal.
3. Transfer gains to other (generally line datasets).
4. Predict model of deconvolved sources using crystalball (the so-called transfer_model step)

Due to it's ancient, hallowed history, the worker is very difficult to modify - hence the need to rewrite it in some fashion.

The currently-agreed-upon approach is to wait for magicStimela, which will solve this in a trice (enabling a master controller switch over separate calibration and imaging workers).

Till magicStimela arrives, there are the following tasks to be done:

i) Take out transfer_gains and transfer_model and put them in a separate worker (how do we / or do we make this backwards compatible?) .
ii) Improve the documentation on the worker.
iii) Synergies with polcal worker (include @francescaLoi 's branch?)
iv) Add DDFacet capabilities - maybe finally we can have primary beams too !?

Did I miss anything?

Pinging @o-smirnov , @SpheMakh , @francescaLoi and @paoloserra to start with - but this is a open discussion, please join in. :)

[francescaLoi]

Hi @KshitijT,

concerning the polarization part I would like to let you know that we started meeting every two weeks with @o-smirnov , @bennahugo and Feng Gao. He is a post doc researcher who will try to develop DDFacet to image polarization.
This will require several months (or more), therefore I'll suggest to implement polarization imaging in CARACal using wsclean. This is something that I already did in the selfcal worker, but I was wondering if it would be better to do polarization imaging in a separate worker. This worker will be then improved to include DDFacet imaging once this will be ready so that the user can have both options. We could also implement basic functions to obtain polarization intensity, angle and fractional polarization images from Q and U. Also, I would suggest to include RM-synthesis here.
In my mind, this worker will be used only to create Q, U images (plus P, PANG, PERC, FDF, etc) that will be stored in a dedicated directory (something like output/continuum/polarization/).
Of course, if one wants to use total intensity images to perform RM-synthesis the frequency channel resolution of I, Q, U images should be the same, but this is something that the user can easily set in the workflow.

Let me know what do you think about this.

[KshitijT]

@francescaLoi , if DDFacet needs more work to be used for polarisation, then let's use wsclean meanwhile. I agree with you, we can make a separate worker for polarisation which will be easier to modify once DDFacet is ready.