Metadata models for 1.0

[ziw-liu]

recOrder is used in scientific experiments where keeping track of the large sets of parameters is essential to both reliable analysis and future reference.

Due to the complex nature of these parameters, as well as how they are organized in the current implementation, it is difficult to keep track of the state (#262 ) of the experiment, and the attempts to pickle the metadata often introduce bugs (#216).

@ieivanov @mattersoflight @Soorya19Pradeep @talonchandler and I have reached a draft of how the data structure and workflow may look like for these metadata.

Data structure

Metadata at the consecutive stages of an experiment can be each aggregated into a dataclass and have methods for saving, loading, and validating data:

Microscope

Specifications of the microscope used in the experiment:

• Wavelength (nm): float
• Objective NA: float
• Condenser NA: float
• Camera Pixel Size (μm): float
• RI of Immersion Media: float
• RI of Specimen: float
• Magnification: int
• BF Channel: str | Literal(edit: not needed)
• Config Group: str | Literal
• LC Mode: Literal
• LCA Device: str | Literal
• LCB Device: str | Literal
• Swing: float

Calibration

Metadata specified/measured in LC calibration:

• Summary: dict[str]
• Illumination Scheme: Literal
• LC Mode: Literal(edit: redundant)
• LCA: tuple[float]
• LCB: tuple[float]
• State Swings: tuple[float]
• Inst. Matrix: NDArray | list(edit: not needed)
• Black Level: float
• Extinction Ratio: float
• Microscope Parameters ID: str

Acquisition

Orchestration of a multi-dimensional acquisition. This should be handled by the external acquisition engine.

Reconstruction

Parameters used to perform computational reconstruction:

• Data Path: str | Path
• Listening mode: Literal
• GPU: int | None
• Data Axes: list[Literal]
• Background Mode: Literal
• Background Path: str | Path
• 90 Degree Offset: bool
• Depth Model: Literal
• Z Padding: int
• Regularizer: Literal
• Strength: float

Workflow

A typical workflow will operate on the above sets of metadata in a mostly sequential manner:

1. Set up the microscope parameters. Default values should be loaded from system cache directory. The state of the optical system is serialized and saved after changes.
2. Run calibration. Information about the light path is read from the data generated in step 1. Calibration results is serialized and saved in a file.
3. Run acquisition. Acquisition metadata will be saved by the external acquisition engine. recOrder monitors the directory and save the microscope parameters and the calibration metadata files to the acquired data directory.
4. Perform reconstruction. A path to the data directory is provided, and metadata files generated in step 3 is read. Reconstruction parameters are serialized and saved along the reconstructed images.

Discussion and suggestions are most welcome!

[mattersoflight]

@ziw-liu I have some suggestions for adding/renaming parameters and am editing the issue directly to avoid clutter.

One question: why do we need two devices (LCA and LCB) in Microscope metadata? are you thinking of retardance values?

[talonchandler]

Thanks @ziw-liu I think this is a great first iteration. Here are my comments:

1. "Camera Pixel Size (μm): float" and "Magnification: int" can be side by side, because together they determine the object-space X-Y pixel size.
2. "Z-step size (μm): float" needs to be added to microscope parameters. We currently handle this parameter from MM metadata (or from the GUI buttons), but I think it's important to have it next to "Camera Pixel Size" and "Magnification" because these three parameters determine the complete voxel dimensions.
3. Will "BF Channel: str | Literal" as a microscope parameter be necessary for 1.0.0 if we're moving "Acquisitions" elsewhere?
4. I echo @mattersoflight's questions about LC mode, LCA, and LCB in "Microscope Metadata". Do these need to appear in both "Microscope" and "Calibration"?
5. I don't think the "Inst. Matrix: NDArray | list" needs to appear in the calibration metadata, since theres a 1:1 mapping between the instrument matrix and the illumination scheme. I also think that the current handling of the instrument matrix is scattered between recOrder and waveorder...and I think for 1.0.0 we should shoot for all instrument matrix math to be in waveorder.
6. For reconstruction we will need a "Channel Axes: list[Literal]" parameter that routes the channels to reconstruction schemes. For example, the first four channels might be polarization, the fifth and sixth channels fluorescence, and the seventh channel brightfield. Alternatively, we might consider only supporting one reconstruction type at a time, which would simplify this parameter to "Channel Input: list[Bool]". I would personally vote for a "Channel Axes: list[Literal]" design, but we can revisit this.
7. What do you have in mind for "Depth Model"?

[ziw-liu]

why do we need two devices (LCA and LCB) in Microscope metadata? are you thinking of retardance values?

@mattersoflight I think the current implementation writes both the retardance and the voltage values into calibration metadata files in the form of LC_{Device: ("A", "B")}_{State: ("ext", "0",...)}. This is not always the case in the many different data structures we use to keep track of calibrations and involves some internal conversions, which can also be error prone.

recOrder/recOrder/calib/Calibration.py

Lines 892 to 918 in f84f09d

	if self.calib_scheme == "4-State":
	metadata["Calibration"].update(
	{
	"Channel names": [f"State{i}" for i in range(4)],
	"LC retardance": {
	f"LC{i}_{j}": np.around(
	getattr(self, f"lc{i.lower()}_{j}"), decimals=6
	)
	for j in ["ext", "0", "60", "120"]
	for i in ["A", "B"]
	},
	"LC voltage": {
	f"LC{i}_{j}": np.around(
	self.calib.get_voltage(
	getattr(self, f"lc{i.lower()}_{j}")
	),
	decimals=4,
	)
	for j in ["ext", "0", "60", "120"]
	for i in ["A", "B"]
	},
	"Swing_0": np.around(self.swing0, decimals=3),
	"Swing_60": np.around(self.swing60, decimals=3),
	"Swing_120": np.around(self.swing120, decimals=3),
	"Instrument matrix": inst_mat,
	}
	)

I agree that now is a good time to figure out the best way to store and communicate device state parameters and welcome input from the team!

Edit: tagging @ieivanov and @talonchandler. Also feel free to open a separate issue for this sub-topic if needed.

[ziw-liu]

"Z-step size (μm): float" needs to be added to microscope parameters. We currently handle this parameter from MM metadata (or from the GUI buttons), but I think it's important to have it next to "Camera Pixel Size" and "Magnification" because these three parameters determine the complete voxel dimensions.

@talonchandler I'm not sure if z-step information should be part of the microscope parameters, given that this is not necessarily determined on the per-experiment level, but more on the per-acquisition level. So it can be set in the acquisition software (e.g. in the MM MDA window) and read from each of the image metadata to inform reconstruction, unless one would like to use a different value in reconstruction than what's used in acquisition?

[ziw-liu]

Will "BF Channel: str | Literal" as a microscope parameter be necessary for 1.0.0 if we're moving "Acquisitions" elsewhere?

I don't think the "Inst. Matrix: NDArray | list" needs to appear in the calibration metadata, since theres a 1:1 mapping between the instrument matrix and the illumination scheme. I also think that the current handling of the instrument matrix is scattered between recOrder and waveorder...and I think for 1.0.0 we should shoot for all instrument matrix math to be in waveorder.

True! Editing OP.

[ziw-liu]

I echo @mattersoflight's questions about LC mode, LCA, and LCB in "Microscope Metadata". Do these need to appear in both "Microscope" and "Calibration"?

Sorry I did not make it clear. The 'LCA/B Device' field in 'Microscope' is a str or Literal field for the device name in MM, and the 'LCA/B' field in 'Calibration' is an Iterable[float] for the calibrated voltage/retardance values.

The 'LC Mode' is indeed redundant. I think it may be better suited in 'Microscope Parameters'.

[ziw-liu]

For reconstruction we will need a "Channel Axes: list[Literal]" parameter that routes the channels to reconstruction schemes.

I called it 'Data Axes' but maybe 'Channel Axes' is clearer?

[ziw-liu]

What do you have in mind for "Depth Model"?

I think you suggested that we use this term to describe the '2D'/'3D' reconstruction models. We can work on a better name if needed.

[talonchandler]

Regarding storing retardance and voltage values: I think both values are useful, so I vote that we continue to record both in the calibration metadata.

I'm not sure if z-step information should be part of the microscope parameters, given that this is not necessarily determined on the per-experiment level, but more on the per-acquisition level.

I'm really torn on this. I agree that z-step tends to be changed more frequently than other experimental parameters, but I also think that in our 1.0.0 design it "belongs" in the "microscope parameters" tab. I've also had at least two cases (one myself, one with a collaborator), where confusion arose about how z-step information was being passed e.g. "I entered in the x-y pixel size parameters manually, where do I enter the z size parameter?"

Can we do both? Provide a "Z-step size" field and read the z-step size from a file. If there's a mismatch, warn the user and default to the user-provided field?

I called it 'Data Axes' but maybe 'Channel Axes' is clearer?

Okay great, I see. I'm not sure if "Channel Axes" is a particularly good name, either.

We'll need to design these fields with a bit more thought. How should we handle cases like: take the first four channels and send them to this reconstruction, take the next BF channel and send it to that reconstruction, and take the final two fluorescent channels and send them to different reconstructions.

Maybe two fields like "Input Channel Group: list[int]" and "Reconstruction methods: list[Literal]" would be enough. For the example above, Input Groups = [0,0,0,0,1,2,3] and Reconstruction methods = ["QLIPP", "PhaseFromBF", "Fluo", "Fluo"]?

I think you suggested that we use this term to describe the '2D'/'3D' reconstruction models

Roger that...I'll think more on this. I'll get a better sense as I work with wo further.

[ziw-liu]

Can we do both? Provide a "Z-step size" field and read the z-step size from a file. If there's a mismatch, warn the user and default to the user-provided field?

We've also discussed the possibility to display the z-step parsed from the dataset along with other relevant parameters in the reconstruction tab. I was against having a text input for it in recOrder because I thought the user will be forced to type in the value during acquisition (i.e. MDA or pycromanager script) regardless of the recOrder design.

[ieivanov]

My feedback at this stage would be to use parameter names that are as descriptive as possible, and further include a comment line with a longer description for that parameter. That will eliminate confusion that the three of us are already experiencing (e.g. how is LCA Device in Microscope different from LCA in Calibration, or what exactly Depth Model means).

RI of Specimen is something that's hard to know, and is not really a microscope parameter. Did you mean something else Ziwen?

There are also some metadata parameters that are user-provides, some that are calculated/calibrated, and some that are obtained from MM. The description of each parameter should specify its source.

[ziw-liu]

RI of Specimen is something that's hard to know, and is not really a microscope parameter. Did you mean something else Ziwen?

@mattersoflight Added this. Can you explain the use case for such a parameter?

[talonchandler]

@mattersoflight and I discussed this, and it stems from a request from our altos collaborators.

Mismatched refractive indices between the immersion media and the specimen will cause incorrect z scaling. For example, if you are imaging into water with an air/oil objective and you use the piezo to move the sample 5 um, your focal plane will move less/more than 5 um (not to mention the degraded PSF from spherical aberration).

Our intent with these two parameters is to: (1) provide a zeroth-order correction by rescaling the result appropriately, and (2) possibly provide depth-dependent deconvolution (likely a later feature).

@ieivanov We think the most common use case will be n_spec = [1.33] since that's not a bad estimate for most biological samples.

Regarding its placement, I'm leaning towards placing it on the "Microscope Params" tab, but I'm also open to having it on the "Reconstruction" tab.