UI sound design

[pixelzoom]

From 4/13/21 Q2 planning meeting:

Investigate adding UI-component sounds for the 1.0 release.

Milestones:
start = 4/1/21
end = 5/15/21

[pixelzoom]

4/9/21 Zoom discussion with @jbphet:

• Some standard UI components do not have default sounds - e.g. sliders, which this sim uses heavily.

• Pedagogical vs non-pedagogical sounds. In this sim, sound for the Fourier series is an example of a pedagogical sound, while checkbox sounds are not pedagogical.

• Sound is currently "all on" or "all off". You can't turn off sounds that are potentially distracting without also turning off sounds that are important to learning goals.

• Being able to turn pedagogical vs non-pedagogical sounds on/off separately would require separate audio channels, separate UI controls - definitely some work in tambo and joist, and UX design work.

4/12/21 Zoom discussion with @arouinfar:

• Noted that we had previously decided to turn off "user-interface" category sounds for the 1.0 release to avoid some of the issues below, see Enable UI sounds #48.

• More discussion about pedagogical vs non-pedagogical sounds.

• PhET sims with pedagogical sounds have (so far) had few controls. That's not the case for Fourier, it has many controls, and therefore many non-pedaggocial sounds.

• Can too many non-pedagogical sounds dilute or detract from the effectiveness of pedagocial sounds?

• Having sounds for the amplitude sliders would interfere with the sound for the Fourier series.

• Some standard UI components used by this sim have not had a sound design - e.g. keypad.

• This sounds like a lot of work, and feels like we may be too far into 1.0 to do this.

• Let's discuss at design meeting.

[pixelzoom]

Options:

(a) Continue with the plan we established in #48 - Disable "user-interface" category sounds for the 1.0 release, then address sound design in a future release. (ESTIMATE: No additional time.)

(b) Ignore the distinction between pedagogical vs non-pedagogical sounds, treat all sounds the same. Enable "user-interface" category sounds for 1.0, let them co-exist with Fourier series sound, don't worry about potential impact on usability or learning goals. (ESTIMATE: 2-4 weeks to get audio balanced, fill in the gaps in common-code sound design.)

(c) Continue the discussion about pedagogical vs non-pedagogical sounds, address any UX design and implementation requirements that come out of that, do a full sound design. (ESTIMATE: unknown)

Unless the milestones can all be pushed out by 1-? months, I recommend (a).

[pixelzoom]

4/15/21 design meeting notes:

@kathy-phet suggested "ducking" UI sounds when Fourier sounds checkbox is checked, as in Waves Intro. @pixelzoom is wondering if this is something supported by tambo, or is it something specific to Waves Intro. @arouinfar says to look at https://github.com/phetsims/wave-interference/blob/master/js/common/view/WaveMeterNode.js#L225-L227.

Amplitude sliders would probably not have a "harmonic" sound, probably just click, click, click (like Waves Intro). We also talked about spinners and specific buttons needing specific sounds.

@Kathy would like to do option (b). OK to push milestone out a month or more. Some of @Ashton-Morris's time is available. @jbphet will probably be doing sound implementation for Keypad and other common-code UI components, but that's up for discussion.

Next steps:

• Look at ducking in Waves Intro (@pixelzoom)
• Inventory UI components and what sounds to use (@arouinfar @Ashton-Morris)

[pixelzoom]

user-interface sounds have been enabled in #48.

Ducking of user-interface sounds has been implemented in #55.

[pixelzoom]

Breaking out sound for amplitude sliders into its own issue, see #56. It looks like that's going to be a significant chunk of work. Sliders appear to be lacking in common-code support for sound.

vegas (game) components that need UI sound design are identified in phetsims/vegas#88.

[pixelzoom]

4/22/21 design meeting @arouinfar @kathy-phet @jbphet @jessegreenberg @pixelzoom

• It's time to investigate generalized support for sound in Slider. What do sliders with pedagogical sounds have in common that can be supported in common code? Is there a good default for sliders with non-pedagogical sounds? Continuous vs discrete sliders? etc. @kathy-phet would like @jbphet to work on this, see Slider API for sound sun#697.

• Notes about vegas (game) are in UI sound design needed by fourier-making-waves vegas#88 (comment).

[pixelzoom]

I think we can say that we met the 5/15/21 milestone. We investigated what is needed for Fourier sound design, and identified missing pieces. Here's what was identified:

• sound for the Fourier series, Review sound implementation for Fourier series (6/1/2021) #45
• sound ducking, When sound for the Fourier series is turned on, duck all user-interface sounds. #55
• Slider sound design, Slider API for sound sun#697
• vegas sound design, UI sound design needed by fourier-making-waves vegas#88

If all of these pieces come together, then Fourier should be in good shape for UI sound.

I'll leave this (epic) issue open until the related issues are addressed.

[pixelzoom]

In #56 (comment), we decided not to add sound for sliders in the 1.0 release. So this issue is done. Closing.