When performing a say all on webpages, the screen should scroll along with the speech

[nvaccessAuto]

Reported by mike.reiser on 2010-09-13 20:25
I was showing my sighted friend a news story, and while doing a sayall, he said that it didn't follow on the screen while the speech was going. It would be nice if this could be done so that my friend could follow along while I'm reading.

[nvaccessAuto]

Comment 1 by jteh on 2010-09-13 22:21
Changes:
Milestone changed from None to None

[nvaccessAuto]

Comment 2 by steverep80 on 2015-02-05 20:53
As a low vision user of NVDA, I would also greatly appreciate this enhancement in order to follow along with what I am reading (not to mention for the same reason as pointed out by the originator - reading simultaneously with my sighted wife). g. The only way I see to do that now (as of 2014.4) is to arrow along in browse mode, which is horrible for just sitting back and reading articles or other documents. If this enhancement were added, add-ons such as Focus Highlight might also more easily be able to fix issues with following along during scrolling.

[nvaccessAuto]

Comment 3 by briang1 (in reply to comment 2) on 2015-02-06 08:50
Replying to steverep80:

As a low vision user of NVDA, I would also greatly appreciate this enhancement in order to follow along with what I am reading (not to mention for the same reason as pointed out by the originator - reading simultaneously with my sighted wife). g. The only way I see to do that now (as of 2014.4) is to arrow along in browse mode, which is horrible for just sitting back and reading articles or other documents. If this enhancement were added, add-ons such as Focus Highlight might also more easily be able to fix issues with following along during scrolling.

I agree on this one. I get quite frustrated sometimes when I have an issue on a web site, wheel in a sighted person and when I'm in the area where I encounter a problem, the sighted person cannot see it as the screen has not scrolled to where it is. I imagine there could be some overhead here with speed if the screen has to be scrolled after a while. So I also think it shoud highlight the word its reading as well, so at least the user might be able to show the person ho elping what is going on.
Is the problem with this on various browsers a performance one? I note that it has a recent ticket to refactor the way speech is handled and thought that while this was being condi sidered, this function in browse mode would need to be considered as well.

[LeonarddeR]

I agree there are some quirks that need to be addressed regarding browse mode and sighted use in general. See also #6460 and #6382.

CC @Brian1Gaff @jcsteh @Qchristensen @feerrenrut

[Qchristensen]

There are two issues I think - knowing exactly where the element is that is being read is in the visual representation of a web page, and then deciding how to scroll the page (if at all) when the element being read is not visible.

Using Say all, I found that none of the current browser versions (Firefox, Chrome, IE or Edge) on Windows 10 moved the focus at all during reading.

For a "control", I went and had a look at Word. Doing a Say All on a long document, NVDA moves the focus to the start of each sentence (not line), scrolling the page so the cursor is on the third line from the top if the sentence being read starts off the bottom of the page. Visually, I find that a little distracting, as when it scrolls, I have to look for the first sentence on the third line. There was a slight lag as well

Next, I looked at Notepad. It moves the focus to each line as it reads it. If a line is off screen, it scrolls just enough to bring it on screen - so after the first screen of text, it scrolls every line.

Assuming it can be implemented without too much trouble, I'm in favour of having the ability to follow the text being read.

I think the two main options are:

• Try and follow the current line. Either keep it at the top, middle or bottom of the screen once moving past the first screen.
• Read through the current screen, then scroll one full screen.
  I'm not sure we'd get 100% agreement from users on either of those (or any of those since the first is really three options). Assuming it was all implementable and not too much harder to do the rest after doing one, I'd be tempted to put an option maybe in "object presentation"? something like: "Scroll during say all:" with a drop down list of options: "focus at top of screen, focus at middle of screen, focus at bottom of screen, scroll full screen, and, none" (the last particularly if there are performance issues with implementation. Ideally then use the chosen implementation across different programs.

[Brian1Gaff]

Hard one, as I'm always coming unstuck with this when I ask a sighted person to see if they can find a control I cannot see with other navigation means. Focus highlight seems not to do this. I guess its tying up the place in a virtual buffer with the place on the screen in real time fast enough so the speech does not glitch. Brian

[LeonarddeR]

@Qchristensen commented on 19 jul. 2017 07:09 CEST:

I'm not sure we'd get 100% agreement from users on either of those (or any of those since the first is really three options). Assuming it was all implementable and not too much harder to do the rest after doing one, I'd be tempted to put an option maybe in "object presentation"? something like: "Scroll during say all:" with a drop down list of options: "focus at top of screen, focus at middle of screen, focus at bottom of screen, scroll full screen, and, none" (the last particularly if there are performance issues with implementation. Ideally then use the chosen implementation across different programs.

I don't think this should be limited to say all. IN my opinion, scrolling in say all and manual scrolling with arrow keys/quick nav should look equal. Furthermore, there is indeed some difference in what supports what, see #6382 (comment)

[jcsteh]

As discussed at length in #6382, the exact way we scroll is largely dependent on what the app facilitates and it's difficult to come up with a UX that suits everyone. For the purposes of this issue, I think we can all agree that say all should scroll.

Technical: The reason this doesn't currently happen is that say all calls updateCaret on the TextInfo. However, for virtual buffers, that just updates the buffer's internal caret offset. In order to trigger scrolling, we'd need to somehow have say all (whether directly or indirectly) use _set_selection on the CursorManager. (This is why braille has a CursorManagerRegion.) I guess we might need to special case this in sayAllHandler too.

Note that I'm not as certain we want to move the focus while doing a say all, as that could cause changes to the page on poorly authored sites. At present, it can also cause weirdness with auto focus mode switching (#2039). To avoid issues like this, we should pass REASON_SAYALL to _set_selection and handle it accordingly in the browse mode implementation.

[Qchristensen]

Nothing to add, except we've had a few requests for this still lately.

[Adriani90]

Actually, is it really that important to scroll in real time while sayall? Maybe it would suffice to provide a keystroke which moves the screen to the current virtual cursor without interupting the sayall, so that people can scroll on demand.

[feerrenrut]

@Adriani90 for low vision users this is a fairly important feature.

[gregtatum]

As an added use case here: I'm implementing some features around translating in Firefox, and they will be gated around what is visible in the screen since it costs CPU time to translate the full page. In the case that sayall doesn't actually navigate the page, the text that is not visible on the screen viewport will not be translated, and will be in the original language. The only way to both preserve performance here and provide the translations for the screenreader is to navigate the page while reading the text.

[jcsteh]

Just to clarify @gregtatum's comment, for the purposes of Firefox translations + NVDA, "navigate the page" means "scroll the page".