Shape prompts and positive/negative points

[dlangenk]

I got a mail asking if mask refinement is also working. I think this would greatly improve the usability of SAM. If we are touching the UI already other functionality could be added as well (don't know about the additional work needed)

• mask refinement by adding positive/negative points
• box inputs
• polygon (convert to binary mask) inputs
• circle (convert to polygon with shapely) inputs

[mzur]

mask refinement by adding positive/negative points

Idea for the refinement UI: Users select Magic SAM and get a suggested mask on hover. If they now press and hold e.g. Ctrl the current suggested mask is frozen and the user can move the cursor elsewhere. They can now left-click (for a positive point) or right-click (for a negative point). Each click updates the current mask. If Ctrl is released, the user can either click inside the current suggested mask to confirm it or they can press Esc to cancel.

polygon (convert to binary mask) inputs

What do you mean with "convert to binary mask"? A bitmap mask? There is no such annotation type in BIIGLE yet.

[dlangenk]

I like the idea of the Ctrl key modifier. But why do you need another click after Ctrl is released. This could just finalize the annotation. If the mask is not to your liking you can just use ⌫ Delete as before.

With the convert to binary mask I meant it the other way around. We can convert a biigle polygon to a binary mask which SAM can handle (but not polygons) as input.

[mzur]

But why do you need another click after Ctrl is released.

IMO the user needs an option to cancel the action (e.g. if they clicked wrong or the mask just won't fit). I think it's counter-intuitive if they have to confirm the (wrong) annotation first before they can delete it.

We can convert a biigle polygon to a binary mask which SAM can handle (but not polygons) as input.

I see, so you draw a rough polygon and SAM should make it more detailed?

[dlangenk]

But why do you need another click after Ctrl is released.

IMO the user needs an option to cancel the action (e.g. if they clicked wrong or the mask just won't fit). I think it's counter-intuitive if they have to confirm the (wrong) annotation first before they can delete it.

I get your point but releasing Ctrl is not really confirming (I mean yes it is technically but not the same as clicking a "I want to confirm this wrong annotation" button). And clicking again would mean another button click in the case that (should) happen more often. I guess I'll think about this tomorrow a bit.

We can convert a biigle polygon to a binary mask which SAM can handle (but not polygons) as input.

I see, so you draw a rough polygon and SAM should make it more detailed?

Yes, the same also for boxes and circles (boxes have an own SAM input, circles need to be transformed to binary masks as SAM input first). Would also be cool to have a refine annotation with SAM button for existing annotations but i don't think this is a priority.

[dlangenk]

I thought a bit about this and I still think that it is better to create the annotation on releasing Ctrl because it means less clicks in the most frequent cases.
A different much more intuitive but more button alternative would be to have a negative and positive SAM button like the paintbrush tool but no wizard hat button anymore. In that case you would click the positive SAM button and create an initial result. Now you either confirm it with a checkmark button (like in the video annotation tool) or you add another positive or negative SAM point. This way there is less memorization of keyboard modifiers but more UI elements as a downside.

[mzur]

I still prefer the idea with Ctrl because it's a more efficient interaction. I think it's ok if the feature is a little hidden (with no extra buttons) because it's something like a "power-user feature". Also, we should not change the existing UI.

What about this: If the user presses Ctrl and enters "positive/negative" mode, the SAM button becomes an "x" and the user can press the button (while still holding Ctrl) to cancel the current annotation. Then we can keep the interaction to create the annotation on Ctrl release.

[dlangenk]

That sounds like a good compromise👍