feat(FieldFormatters): add visual editor

[maxpatiiuk]

Fixing an issue from 2012 😊

Fixes #23

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

Generally visual editor for Field Formatters should act and look very similar to the one for Record formatters since it shares a lot of code with it.

To test:

• Verify that each field formatter type (constant, any character, ...) can be set in the UI, without user-experience issues, and after saving, can be read back correctly
• Verify that a formatter created/edited using a visual editor works in Specify 7 and Specify 6 (in forms and query results)
  • For reference, here is how to configure a field formatter in Specify 6
• P.S: don't try to enter an emoji character like on the screenshot above - Specify 7 back-end does not support emojis in app resources 😢

Documentation on field formatters

[maxpatiiuk]

these cases should not happen - the UI doesn't allow them. but handling them here just in case (and to satisfy TypeScript)

[grantfitzsimmons]

❤️

[grantfitzsimmons]

1. When creating a field format like GiftNumber using the regular expression /^(KUI|KUBI|NHM)$/, I am unable to set it up correctly.

See that I am supposed to put the 'hint' on the left and 'pattern' on the right:

This works as expected when using a regex validator, but when setting it as the pattern it is not recognized properly.

	<format system="true" name="GiftNumber" class="edu.ku.brc.specify.datamodel.Gift" fieldname="giftNumber" title="" default="true">
		<autonumber>edu.ku.brc.af.core.db.AutoNumberGeneric</autonumber>
		<field type="regex" size="4" value="/^(KUI|KUBI|NHM)$/" byyear="true" pattern="KUBI"/>
	</format>

---

2. After modifying other sections of the format, those changes are not made immediately visible for the 'Example Field'. Instead, it uses the previous field format before the edits have been made.

See that it shows "Required Format: KUBI" in the tooltip:

(This tooltip is most often obscured by the browser's "match the requested formatter" error)
https://github.com/specify/specify7/assets/37256050/d34bcb16-85d5-43fc-8c68-56bef2e36cc6

After making it simply numeric:

---

3. What does the 'Auto Numbering' do that the 'Auto-number' checkbox does not?

[maxpatiiuk]

When creating a field format like GiftNumber using the regular expression /^(KUI|KUBI|NHM)$/, I am unable to set it up correctly.

Looks like the expected regex between Sp6 and Sp7 differed.
In Specify 6, the regex is run on each field part separately, so /^ and $/ are necessary
In Specify 7, all regexes are joined together into on regex, thus, having /^ or $/ will break things if there is more than one part in your formatter.

Fixed:

• When reading from .xml, both Specify 7 front-end and back-end will trim any /^ or $/ from the regex pattern to not break the pattern when multiple parts are joined together. Similarly, outer ( and ) are removed if present
• When front-end saves the updated .xml, the syncers will make sure to add /^ and $/, even if not originally specified by the user. Also, if the pattern includes |, leading ( and trailing ) will be added

In practice, that means:

• When using the visual field format editor, it doesn't matter anymore if you put or not put /^ and $/ in a regex field as the value will be normalized automatically
• Backwards compatibility with Specify 6

[maxpatiiuk]

See that I am supposed to put the 'hint' on the left and 'pattern' on the right:

yeah, it's not ideal that 'hint' comes before the 'pattern' as the opposite order seems more intuitive.

The reason I did it this way is because of the column heading.
For all other part types, the 'hint' column is used both for declaring what the format should be, and for proving the validation message
For regex parts, since regex is not "user friendly" enough to put into a regex message, the 'hint' field is used only for validation messages, and the regex pattern itself is a separate field (which also validates that the regex pattern has correct syntax)

Open to ideas on how to make this more intuitive

[maxpatiiuk]

After modifying other sections of the format, those changes are not made immediately visible for the 'Example Field'. Instead, it uses the previous field format before the edits have been made.

Fixed

[maxpatiiuk]

What does the 'Auto Numbering' do that the 'Auto-number' checkbox does not?

Ups, the "Auto Numbering" checkbox at the top is not supposed to be exposed in the UI - removed
(it corresponds to whether .xml had a tag like <autonumber>edu.ku.brc.specify.dbsupport.CollectionAutoNumber</autonumber>, which is an sp6-only construct - sp7 cares about it only in so far as creating sp6-compatiable field formatters)

[grantfitzsimmons]

@maxpatiiuk Can you resolve the merge conflicts?

[lexiclevenger]

Testing instructions

Generally visual editor for Field Formatters should act and look very similar to the one for Record formatters since it shares a lot of code with it.

To test:

• Verify that each field formatter type (constant, any character, ...) can be set in the UI, without user-experience issues, and after saving, can be read back correctly
• Verify that a formatter created/edited using a visual editor works in Specify 7 and Specify 6 (in forms and query results)
  • For reference, here is how to configure a field formatter in Specify 6
• P.S: don't try to enter an emoji character like on the screenshot above - Specify 7 back-end does not support emojis in app resources 😢

---

Super exciting! I have a few recommendations:

1. If no default formatter is selected, the message says "record formatter. I would change this to "field formatter"

1. The default for the "size" field for most field types is 0, and this can be saved as long as there is a value in the "hint" field. The value will be shown in the formatter, but it will make the format invalid if that value is entered in the field. In the Specify 6 field formatter, the "size" value cannot go below 1; I think adding this in 7 would be a good solution.

3. The user can input and save any value at any length in the "hint" field (see screenshot above) for the numeric field type, but this is misleading because "#" is the only character that actually works, and it will only appear as many times as the value in the "size" field. Maybe this field could be read-only since it's not really functional.

I'd like to hear what others have to say about 2 and 3; they don't conflict with the xml editor, and the "Example field" displays their consequences correctly, but I think these solutions would make the visual editor more user-friendly.

[maxpatiiuk]

If no default formatter is selected, the message says "record formatter. I would change this to "field formatter"

Good point. To have the same message be applicable both in the field formatters editor and record formatters editor, I edited it to say:

"Please designate one of the formatters as default"

The default for the "size" field for most field types is 0, and this can be saved as long as there is a value in the "hint" field. The value will be shown in the formatter, but it will make the format invalid if that value is entered in the field. In the Specify 6 field formatter, the "size" value cannot go below 1; I think adding this in 7 would be a good solution.

Made "size" value not go below 1

The user can input and save any value at any length in the "hint" field (see screenshot above) for the numeric field type, but this is misleading because "#" is the only character that actually works, and it will only appear as many times as the value in the "size" field. Maybe this field could be read-only since it's not really functional.

Fixed. "hint" is now readonly for number fields

[lexiclevenger]

Looks good! Another thing I noticed is that clicking on the "edit" button next to a format in schema config often takes you to the wrong one.

Schema.Config_.Gift._.Specify.7.-.Google.Chrome.2024-07-23.11-30-30.mp4

https://fwri1924-field-editor.test.specifysystems.org/specify/schema-config/en/Gift/

[melton-jason]

Not sure if this is supposed to happen but when I set a numeric type to auto-incrementing and put it on the fax field and tried to actually use and save on the form I got an error. Assuming this is intentional, we should probably try to find another solution other than the user getting an error in the forms (maybe some sort of save blocker in the field formatter dialog?)

chrome_oLsql8EBHn.mp4

Can reproduce.
@melton-jason The error says " has no hierarchy field"
Is auto numering only supported for tables with hierarchy fields? Otherwise, can auto-numbering be global? or can it piggy-back on the "parent" table hierarchy (in this case Address is a dependent record of Agent, which is attached to a division)

From #5075 (review) and #5075 (comment)

Yes, looks like auto-numbering right now is only supported for tables with hierarchy fields.

Specifically for auto-numbering, the backend uses the filter_by_colleciton function (so this error is the same one as #4989 and #3564)

specify7/specifyweb/specify/uiformatters.py

Lines 32 to 33 in 1d6b1ee

	def get_autonumber_group_filter(model, collection, format_name: str):
	default = lambda objs: filter_by_collection(objs, collection)

The function has support for a strict argument (which defaults to true if unset), so the easiest solution in this case would just be less strict about scoping (i.e., make the scoping global).

The potential consequence of doing so is that if there are two or more records which are supposed to have differing scopes, they will share a global autonumbering scheme.
I think this solution would work in most cases, but there are times when scoping becomes tricky (artifacts of Specify 6 aside).

Anything besides making the scoping less strict will require some implementation of #5044 (review)

---

or can it piggy-back on the "parent" table hierarchy (in this case Address is a dependent record of Agent, which is attached to a division)

I think Address is a really great example to consider scoping for auto-numbering (and other applications)!
(For general number of cases, maybe not actual feasibility of being useful)

Building on this, Address also has independent relationships in the form of:

• Division -> address
• Institution -> address
• InstitutionNetwork -> address

Because of the dependent relationship of Agent -> addresses, we know that - if desired - auto-numbering of fields within address can even be conceptually restricted to a per-agent basis if desired, though I'd assume we'd try and use the "parent" record's scope if the parent record/scope isn't a hierarchy table.
Should the dependent addresses be included when determining the next number for addresses in independent relationships?

In the case of the independent relationships, should the Division addresses be considered separately from the Institution/InstitutionNetowork addresses? That is, if we add a new Division, do we only consider the addresses of other Divisions or just all addresses in general? (which might be the same in this case, but might not always be the case across all relationships).

A global approach is by far the easiest to implement and think about, at least from a developmental and design standpoint.
As long as it would be expected by most users, that's the route I would take.

[maxpatiiuk]

Yes, looks like auto-numbering right now is only supported for tables with hierarchy fields.

I see. That might match sp6 behavior.
@grantfitzsimmons have you heard of users requesting we expand auto numbering support to more tables?
Otherwise, I will keep the scope of this PR in check and restrict auto numbering in UI to what us allowed by the back-end - we can reconsider once we get a chance to re-engineer the scoping system