[Firefox] No sound heard with keyboard nav until button/checkbox is selected

[Nancy-Salpepi]

Seen while testing phetsims/qa#941, with mac/windows + firefox, if I move a corner or side with keyboard nav no sound is heard. Sound doesn't start with keyboard nav until I press a button or check a checkbox.

This is not a new problem--I see it in the last rc we tested for Quad.

Also, @KatieWoe verified that it is not specific to Quadrilateral. For example, with https://phet-dev.colorado.edu/html/geometric-optics/1.3.0-dev.1/phet/geometric-optics_all_phet.html?screens=1, if I tab to a slider and move it back and forth I don't hear anything.

[jessegreenberg]

Thanks @Nancy-Salpepi, this is another autoplay policy issue. Firefox doesn't count keydown as a relevant gesture to start playing sound. I verified we emit the Display.userGestureEmitter (which tells tambo its OK to start playing sound) from keydown events. But Firefox blocks it.

I do not think there is anything we can do to get around this. We could refer users to Firefox user settings that allow all media right away. Maybe someday PhET could have a "Start" button after the splash screen to start up all media. Could also submit a request to Mozilla to change this.

@jbphet @terracoda @emily-phet @BLFiedler just an FYI for another case of this. Assigning to @BLFiedler to review, comment on next steps, or close.

[jessegreenberg]

Another autoplay policy change was found during this RC from Apple: phetsims/tambo#179

[terracoda]

I sent an email to [email protected]

Dear Apple Devs who care and know about accessibility and inclusion,

Our team at PhET Interactive Simulations designs and implements inclusive features for highly interactive learning tools.

We find that the changing policies at Apple are having general adverse effects on the inclusive experiences we carefully design for learners who use our HTML5 simulations. When learners use the simulations with Apple products like Safari, Safari + VoiceOver, or Safari + iOS VoiceOver, they can can run into trouble.

The most recent issue is that sound no longer plays when a learner moves a slider or movable object in a simulation. The learner must first press a button in the simulation. How is a sonified slider, or moving object any different from a sonified button? All are interactive elements and all make sounds. A learner is choosing to interact, and we feel they should be getting the full experience of the interaction. Learners have a clear option to turn off the sounds if they do not want to hear them.

In addition, this latest change to how Safari handles sounds makes for a very inconsistent user experience in many contexts. At first the slider or moving object won’t make sounds, but then they will make sounds after pressing a button. This is not good design, nor is it inclusive design.

Is there a work around for this issue? We want all our sounds to work when interacting as we have designed them to work.

Sincerely,
Taliesin Smith

PS
I have a few other issues. I invite you to try them out:
With iOS VoiceOver - some toggle buttons no longer work. This is a serious problem in Molecules and Light. I cannot turn the Light Source to an on position. Blind learners cannot use their phones or iPads to access this sim. The sim works fine with MacOS VoiceOver.
With iOS VoiceOver custom grab and drag interactions were also broken, but I just checked again and these custom drag interactions look like they may be working again, see Balloons and Static Electricity and Friction. Thanks for fixing this. They weren’t working for a very long time. I last checked in January 2023. I’ll request these sims be republished with this important feature. This means they have to go through our quality assurance process again which takes up valuable resources for a non-profit.
With MacOS VoiceOver, an annoying sound plays when I activate a custom interaction like the Yellow Balloon in Balloons and Static Electricity. Is there a way to turn the sound off? We have our own sonifications.

Again sincerely,
Taliesin Smith

[terracoda]

Is this issue with Firefox of Safari? Or is Firefox on MacOS?

[jessegreenberg]

Thanks @terracoda! Hopefully they will have some thoughts. This issue is specific to Firefox (both Windows and MacOS).

phetsims/tambo#179 was found only on iOS.

[brettfiedler]

If this is cross sim, it should go into another issue and prioritized for dev attention. Does not block Quad publication (as frustrating as it may be :( )

@jessegreenberg , can you create an issue in the appropriate repo and close this one?

[jessegreenberg]

OK, I created a tambo issue: phetsims/tambo#180

Closing.