Add CAREamics CI Script

[melisande-c]

CAREamics compatibility script, for automatic CI testing of CAREamics models in the BMZ.

Compatibility checks description

1. Resource has to be a model type (happens in check_tool_compatibility)
2. Resource must have a model description from v0.5 of the bioimageio.spec, otherwise a failed report is returned.
3. Resource must be tagged with CAREamics or careamics, if not returns a not-applicable report.
4. A CAREamics config must be contained in the attachments, if not returns a failed report.
5. CAREamics must be able to load the model and config using the CAREamics load_from_bmz function, if not returns a failed report.
6. CAREamics must be able to perform inference on the included input array, if not returns a failed report.
7. If all the above steps pass successfully, a passed report is returned.

Notes

The existing uploaded CAREamics models should have their config renamed to careamics.yaml from config.yml, for these checks to pass.

EDIT: Remove resolved questions and notes.

[FynnBe]

Firstly, what is the best way to check the spec version of a ModelDescr object in a way that will satisfy the type checker when accessing attributes?

using isinstance:

from bioimageio.spec import load_model_description
from bioimageio.spec.model import AnyModelDescr  # currently Union[v0_4.ModelDescr, v0_5.ModelDescr]
from bioimageio.spec.model.v0_5 import ModelDescr

# might raise ValueError for invalid or non-model descriptions
model: AnyModelDescr = load_model_description("affable-shark") 

if isinstance(model, ModelDescr):
    # v0_5.ModelDescr
    ...
else:
    # v0_4.ModelDescr
    ...

If you don't want to raise an exception on loading just yet you can also check if isinstance(model, InvalidDescr)

from bioimageio.spec import InvalidDescr, load_description

resource = load_description("affable-shark")
if isinstance(resource, InvalidDescr):
    resource.validation_summary.display()
    raise ValueError("resource is invalid")

... # more type narrowing with `isinstance` like above...

[FynnBe]

Secondly, more practically, how do I access the input tensor in v0.4 of the spec? I tried looking at the source code but I thought maybe I can save time by asking.

.inputs, see https://bioimage-io.github.io/spec-bioimage-io/bioimageio/spec/model/v0_4.html#ModelDescr.inputs

I would encourage you to focus on the model v0_5 implementation (and update any model descriptions still in v0_4 if needed, which is usually very little work, especially as there are not that many yet).
This should make writing your code easier.

v0_5.ModelDescr.inputs dev documentation

[FynnBe]

Finally, for the attachments, the script covers two variations that I encountered through testing a few examples; however, my type checker is complaining:

attachments has changed from model v0_4 to v0_5. With type narrowing you can cover both cases (or just expect a model v0_5 and report an error for v0_4).

[FynnBe]

once we reach our 0.1.0 release (coming soon) we are still a little bit in flux. How do you think we should address this?

As long as CAREamics is not released at all (I'm assuming 0.1.0 will be your first release) I would not run compatibility checks at all. The users will not benefiting from knowing a given model is compatible with CAREamics commit 13497234.
We do not support compatibility checks with some 'lastest' or 'main' version. These would have to be rerun all the time and that does not seem feasible atm. So if a compatbility report exists for a given resource (which includes the version number) the CI will skip checking this combination of resource and tool version again.

edit:

Before this PR is merged, first the following CAREamics should be merged (and a new CAREamics release created):

I guess this is a non-issue if there will be a CAREamics version out.
You can decide which CAREamics versions you'd like to produce compatibility checks with. This can be adapted at any time, fading out testing with old versions and testing with new releases...

[melisande-c]

Thanks for the quick reply!

Yes if I could skip v0.4 altogether that would be ideal since CAREamics can only create v0.5 descriptions. I will try your suggestions and get back to you if I have any questions!

As long as CAREamics is not released at all (I'm assuming 0.1.0 will be your first release)

We actually do have some releases, we are currently on 0.0.4. We are treating our 0th major minor version as an alpha version, so users have an easier installation process, with the understanding there can still be breaking changes up until our 0.1.0 release, but they can still keep track of which version worked for them.

[FynnBe]

We actually do have some releases, we are currently on 0.0.4. We are treating our 0th major minor version as an alpha version, so users have an easier installation process, with the understanding there can still be breaking changes up until our 0.1.0 release, but they can still keep track of which version worked for them.

Great! As long as there is a fixed version we can test with I don't see an issue 👍