Emote modifiers

[Suyooo]

Hi! I've implemented BTTV emote modifiers in Twitch message parsing, so they can be displayed in chat widgets. This is more of a proof of concept right now, hence the Draft PR - I'm submitting this mainly to check whether this is something you want implemented in base slime2, and if yes, what your thoughts on a couple of problems would be.

Currently, this feature works by hardcoding the modifier emotes into the emote map returned by the BTTV service. Then it iterates over the parsed message Parts, removing modifier emotes and applying the styling to the following target emote (ignoring any whitespace-only text-type Parts).
The styling is returned as a string of CSS in the emote-type Part, so widgets can support modifiers by simply assigning that string to the style attribute of their emote elements.
If the style info is not used by the widget, (valid) modifier emotes will be hidden and the target emotes will be unaffected. Widgets could offer an "Enable Emote Modifiers" setting this way, by simply ignoring style info if it is not wanted. (The only exception is z! which works without styling, it just removes the text Part between the emotes.)

KotoBomb h! KotoBomb v! KotoBomb l! KotoBomb r! KotoBomb
c! KotoBomb p! KotoBomb KotoBomb z! KotoBomb
w! KotoBomb w! c! KotoBomb

Some open todos/questions:

• The modifier emote list is currently hardcoded. Would it be better to fetch the BTTV global emotes list, but filter out all non-modifier emotes, to make sure it works fine even if IDs change or future modifiers are still hidden on widgets while unimplemented?
• It would also probably be smarter to make the modifier styles CSS classes and only return the class names to be added to the emote elements, instead of concatenating the style strings in the parser...
• Since this might be useful for YouTube chat as well in the future, would it be useful to implement this as a seperate service that can be used on both sites? The Part type could be possible to share between the chats, I think.
• Should this handle FFZ modifiers as well? These modifier emotes are called effect emotes there, and work a little differently in terms of setting them up (modifier_flags bitmask instead of just by emote name), but are generally also just filters, transforms and animations, so it probably would work just fine.
• The w! modifier works right now by adding blank emotes before and after the scaled emote to make sure that no matter what size the widget renders emotes in, the correct amount of space is reserved. Well, only for square emotes right now - but I think that can be changed by making the blank emotes copies of the widened emote, and adding an internal-only "hidden" modifier to them so they aren't rendered.
• A big problem right now with w! and z! is line breaks. The blank emotes for w! can end up in different lines and mess with the spacing that way, and z! cannot guarantee the emotes will be on the same line, so they might not actually be joined. I can't think of a way to avoid a line break with just CSS.
• The one modifier I have no idea how to implement is emotes overlaid over another ("modifier emotes" in FFZ, and has a PR going in the BTTV repo as o!). The overlaying itself could be just a transform: translateX(-100%), but that would leave the empty space, and result in another possible line break problem.
• Fixing the problems with these modifiers would require additional functionality to be implemented in the widgets instead of pure CSS as it currently is. I'm not sure though, what would be the easiest to implement for widget developers, to reduce the amount of changes they have to make to enable modifiers?
  • For w! and o!, one solution could be to simply add a class to the emotes and having widget developers implement the CSS rules depending on their chosen emote height. This would solve most line break problems (widened emotes could be just a single element, overlaying emotes could use margins), but it wouldn't solve the issue for z!.
  • Another possibility would be to be able to insert zero-width non-breaking spaces between emotes. I can't use a text-type Part as it will be nested in a span - this would require some sort of alternative text Part, which widget developers must render without any surrounding element, so it could be handled by the browser correctly. While that would solve the line break problems, it doesn't help with overlaying emotes for o!.
  • A flexible solution would be to add something like a "combined emotes Part", and widget developers would have to add functionality to render these as a div element with the emotes inside. You could then apply any CSS rules to the outer container - a no-wrap for w! and z!, or a rule that absolutely positions children for o!. I think this might potentially cause troubles with vertical alignment, though, but it seems like the best approach to me.

Looking forward to hearing your thoughts!

[zaytri]

I'll be honest I really don't want to bother implementing emote modifiers from either BTTV or FFZ. BTTV doesn't bother providing any developer documentation on how to implement them, and FFZ does but admits:

I realize that implementing these effects can potentially be a lot of work for a developer, especially if they aren't using a client that renders chat via HTML and CSS. In the future I'd like to potentially offer a server endpoint that prerenders emotes with effects. Unfortunately, we lack the resources to do that at this time. - FrankerFaceZ API Documentation

There's also the question of should these modifiers work for emotes from any service, or just from BTTV/FFZ? FFZ also has premium modifiers for some reason, so it'd have to somehow include a check for that as well. Some of those premium FFZ modifiers also animate a bit too fast for accessibility purposes.

Can they be implemented? Yes. But I just really don't like the idea of these emote services passing the work for the implementation of it on any dev that wants to use their emote services. I would much rather it be like Twitch's emote modifiers where it's pre-rendered on their servers, and then we just display the resulting image.

If anything I'd rather just remove the modifiers from the output entirely so that at the very least they're not shown.

I'll answer your questions anyway because I do have answers even if I don't like the idea of it all:

1. Hardcoding the modifier list is fine
2. Yea would definitely implement it as global CSS classes that developers can easily add
3. That can wait until after YouTube is implemented and released
4. FFZ I talked about up above
5. Hmmm I feel like it's gotta be possible to stretch the emote without using a transform, especially cuz like what happens when an already wide emote gets widened? Does it actually get wider or does it stay the same size?
6. How is z! handled by BTTV in chat normally? If it doesn't handle line breaks in the Twitch chat itself then it's not necessary to implement
7. I really don't trust overlaying emotes because of some of the nasty combinations people do with that. If that were to ever be implemented, I think the best way would be to take the overlaying Emote entirely and put it into the Emote that it's overlaying onto, and then make some CSS class to easily add it via the ::after psuedo-element.
8. Already addressed the possibility of w! and o! as CSS classes above, for z! just removing the text part in between the emotes is fine don't need the zero-width thing at all. Having a separate combinated emotes part feels like too much

[Suyooo]

I understand - my thought was that if several widgets/users wanted modifiers, it would make more sense to implement in slime2 itself so the logic and styling are shared. But yes, if the emote services don't offer any functionality to implement these outside of their own clients, it makes little sense to try to keep up - also since it would mean having to release a update to all widgets every time a new modifier is added, either way it is done.

(I asked on the BTTV side about modifiers, and while they say they'll look into improving documentation on them, they too have no plans for a prerendering endpoint. FFZ also apparently has a whole bunch of additional premium modifiers with more complex animations that are not listed in the documentation at all...)

As for now, how about removing the styling from the PR, but keeping the emote modifier parsing? Valid modifiers (or invalid ones too?) could be removed from the message, but it could still store the information in the emote object as it does right now, except it doesn't apply their effects. Widget developers or users could then decide themselves what to do with that information - leave modifiers unimplemented, offer settings to let users pick which ones to allow, or even style them differently.

In that case, I'd probably change to loading the modifier names from the APIs instead of hardcoding them (filtering out premium FFZ ones), so whenever new ones are added, they are handled and hidden by default without having to update the widget? I think I could also try to still offer the CSS file or something in the guides on the forum then, to help devs and users with getting started if they want to have modifiers...

[zaytri]

Adding a property just to indicate that it's a modifier would be fine. Just note that I won't be pulling this PR in because I'm remaking all of this in the desktop app codebase, but I'll keep it open for reference