Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote Access #17580

Merged
merged 207 commits into from
Mar 7, 2025
Merged

Remote Access #17580

merged 207 commits into from
Mar 7, 2025

Conversation

ctoth
Copy link
Contributor

@ctoth ctoth commented Jan 5, 2025

Link to issue number:

#4390 - The initial request for "NVDA Remote functionality

Summary of the issue:

This PR integrates NVDA Remote functionality into core with significant
architectural improvements and modernization. While maintaining protocol compatibility with the existing add-on, it introduces cleaner architecture, type safety, proper event handling, and improved maintainability.

Description of user facing changes

  • Adds a new "Remote" submenu under NVDA's Tools menu with options to:
    • Connect to another computer
    • Disconnect from remote session
    • Mute remote speech
    • Push clipboard content
    • Copy connection link
    • Send Ctrl+Alt+Del
  • New remote settings panel in NVDA Settings dialog with options for:
    • Auto-connect on startup
    • Host/client configuration
    • Connection type (control/be controlled)
    • Server address and port settings
    • Connection key management
    • Audio feedback preferences
  • New keyboard shortcuts for remote control (note these are defaults and can all be changed):
    • F11 to toggle between local and remote control
    • NVDA+Alt+C to push clipboard
    • NVDA+Alt+PageDown to disconnect
    • NVDA+Alt+M to toggle remote mute
  • Audio cues and visual feedback for connection events
  • Comprehensive user documentation added to NVDA User Guide

Description of development approach

The implementation follows a modular architecture:

  1. Core Components:

    • Remote client with session management
    • Secure transport layer using SSL/TLS
    • Protocol serialization for message passing
    • Relay server infrastructure for connection brokering
  2. Integration Points:

    • Hooks into NVDA's speech, braille and input subsystems
    • UI integration via wx menu items and dialog boxes
    • Extension point handling for event propagation
    • Secure desktop support for UAC dialogs
    • Added remaining extension points
  3. Security Considerations:

    • SSL certificate validation and fingerprint verification
    • Channel-based authentication

The code has been ported from the nvdaremote/nvdaremote repository and significantly improved from the version most-recently deployed.

Testing strategy:

We have performed extensive manual testing of the feature both with versions of itself as well as the older add-on.
We have some examples of unit tests hooked up with NVDA's testing infrastructure testing some components which will be supplied, , though we as the community need to write more unit and system tests.

It has been tested with Braille, though we would greatly-appreciate if more Braille and Braille-only users could test it strongly. We wish to improve the Braille experience from the existing baseline in the 2.6.4 add-on.

Known issues with pull request:

  • The URL handler has not yet been hooked up, disabling nvdaremote:// link functionality. We should discuss if we want to embed it as a stand-alone executable or port the functionality to another executable in NVDA. See also NVDA Remote dependency: Bundle url_handler.exe in NVDA #16714
  • Existing add-on translations remain essentially unchanged and will be supplied, which should ease the job for translators
  • The remote functionality attempts to be as self-contained as possible in the remoteClient package, but we may want to move things like the RemoteMenu out into the GUI package. The rationale for leaving everything self-contained is it makes it very easy to completely remove the functionality if desired in a corporate setting.

Code Review Checklist:

  • Documentation:
    • Added detailed user guide section
    • Developer documentation in code comments
    • Help text for UI elements
    • Changelog entry for Remote feature
  • Testing:
    • Unit tests should be added for core components
    • Manual testing of UI and connection flows was performed
  • UX of all users considered:
    • Speech handling preserved
    • Braille display support
    • Audio cues for connection events
    • All strings translatable with many existing translations
    • UI accessibility maintained
  • API is compatible with existing add-ons:
    • New extension points added cleanly
    • Existing hooks preserved
    • Monkeypatching removed from a popular add-on, improving ecosystem stability
  • Security precautions taken:
    • SSL/TLS encryption improvements including removing the old static certificate
    • Certificate validation
    • Channel authentication
    • Secure desktop support
    • UAC handling

Summary by CodeRabbit

Release Notes: NVDA Remote Access Feature

New Features

  • Added comprehensive Remote Access functionality to NVDA
  • Introduced ability to connect to and control another computer running NVDA
  • Supports both relay server and direct connection methods
  • Enables clipboard sharing, remote speech control, and key input transmission

Key Capabilities

  • Connect as master or slave to a remote computer
  • Toggle remote speech muting
  • Push clipboard contents to remote machine
  • Copy remote session link
  • Send Ctrl+Alt+Del to remote computer
  • Secure connection with SSL/TLS encryption

User Interface

  • New Remote Settings panel in NVDA configuration
  • Context menu for managing remote connections
  • Support for nvdaremote:// URL handling

@AppVeyorBot
Copy link

See test results for failed build of commit 460a4efb7f

@amirmahdifard
Copy link

hi, great! but if this will be implimented, tele nvda addon's features should be ported hear because that addon has several features that we need, that nvdaremote doesn't have them. great!

@amirmahdifard
Copy link

@jmdaweb may be you have something hear

@jmdaweb
Copy link

jmdaweb commented Jan 5, 2025

Sorry, but I have nothing to say. An issue was opened in the NVDA Remote repo requesting integration of TeleNVDA features while code refactor was in progress and it was completely ignored. Given our interactions in the past, I prefer waiting for NV Access. For now, I think TeleNVDA will probably reach its end of life and all additional features will be lost. However, seems this pr will have a long discussion and more commits before it's merged, so it's impossible to predict the result.

@amirmahdifard
Copy link

@jmdaweb what! why tele nvda will be discontinued? the features was the best and it's impossible to work with nvda remote with out tele nvda's features. This is not good at all. someone who could, later dipending on nvaccess opinion, could develop the tele nvda's code in to nvda.

@Adriani90
Copy link
Collaborator

Cc: @LeonarddeR

@Aarushb
Copy link

Aarushb commented Jan 5, 2025

To be honest, that is exactly what I was thinking: the tell NVDA assistance addon features. While this itself is a great step, I've migrated to tell for a good while now because it allows for features such as file sharing. I will not derail any further however given that this is pr for remote and not tell, neither is this an issue. I just mentioned it since apparently an issue was opened previously but not responded to. Just having remote itself is also really fantastic, and the features from tell could always be merged in at a later time with proper credit.

@LeonarddeR
Copy link
Collaborator

I guess it makes sense to port whatever is relevant in tele nvda to NVDA core after this pr is merged.
In any case, I wholeheartedly love the fact that this pr is here.
@ctoth Given the draft status of this pr, is it ready for a code review?

@ctoth
Copy link
Contributor Author

ctoth commented Jan 6, 2025

@LeonarddeR We would love feedback! I'm heading to CES this week and so may not be able to address reviews until next week, but absolutely reviews are welcome. We want to make this the best-possible experience for users and devs alike.
I'd generally prefer to minimize any large architectural deltas, though I'm happy to discuss future ideas.

Copy link
Collaborator

@LeonarddeR LeonarddeR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some initial thoughts:

* The URL handler has not yet been hooked up, disabling nvdaremote:// link functionality. We should discuss if we want to embed it as a stand-alone executable or port the functionality to another executable in NVDA. See also [NVDA Remote dependency: Bundle url_handler.exe in NVDA #16714](https://github.com/nvaccess/nvda/issues/16714)

I guess this can be added to nvda_slave, which also deals with the nvda-addon file extension handling.

The rationale for leaving everything self-contained is it makes it very easy to completely remove the functionality if desired in a corporate setting.

I think having a super toggle somewhere that indeed allows you to disable the functionality completely makes sense here.

Regarding the code, I'm noticing several snake cased attribute names here and there, particularly in the gui parts. I'm pretty sure something like code rabbit should be able to list them all, then renaming them with an intelligent IDE shouldn't be to difficult.

@CyrilleB79
Copy link
Collaborator

I can see that #43900 has not yet been triaged; maybe, I hope for you, you have discussed it in an alternative way with NV Access so that this almost 5000 LOC work was not submitted in vain.

I'll wait for NV Access labelling (triaged or concept-approved) before commenting this PR.

@amirmahdifard
Copy link

also, if this will be merged, it should become like nvda it self as wel. In nvda, in addon stor for example, when we press actions button for an addon or press application, all context menu options are toggle options, which is really great. But in nvdaremote, we always have connect and disconnect options, and when connected, the connect option is disabled and disconnect is enabled and vice vercer. This should be a toggle option in core. Tele nvda already done this but not like it should. It used removed and inserted options instead of disabling and enabling. But hear, one of the options connect or disconnect should be completely removed from the code, and only the other be inuse, exactly like the mute option. That is, when we press that option, an if check should be in it's def to check if the remote is connected, it will disconnect and the option name will be changed to connect..., and vice vercer. And also, like tele nvda, the mute option should change it's lable between mute remote and unmute remote. This way, the code is much cleaner and clearer because one of the defs will be removed and the checks will all be aplyed on one def, and also, as we talked before in tele nvda repo, this is more wanted user interface and less confusing. Now that this is coming in to nvda it self, people should have it in the improvements as now this is in the core, and users should not replace it with anything else, like, oh nvdaremote is not good, lets switch to tele nvda because of, 1: user interface. 2: alert before disconnecting the controled computer checkbox to prevent so many problems developed by @cary-rowen in tele nvda repo. 3: the new mute remote computer when controling local computer that if checked, when you switch back to local computer, the remote computer will be muted automaticly. This is useful for many cases but all of these are optional and no body is forced to use or not use them, developed by my self in tele nvda repo. 3: a fue unassighned shortcuts, such as. Send ctrl alt del, open addon options, developed by my self in tele nvda repo. So, please consider everything befor this addition. As always, Thanks!

@SaschaCowley SaschaCowley added the conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review. label Jan 6, 2025
@ctoth ctoth force-pushed the remote branch 2 times, most recently from cacbbbf to 90471a6 Compare January 12, 2025 19:10
@ctoth
Copy link
Contributor Author

ctoth commented Jan 13, 2025

Here are the existing add-on translations:
translations.zip

@amirmahdifard
Copy link

about unmuting remote computer when controling it commit, it should folow tele nvda's code because that is optional checkbox that if checked, controling local computer will mute the remote automaticly and when controling remote, it will unmute the remote automaticly. And if this option is not checked, non of these happen. It is a lot better and useful for many things

@amirmahdifard
Copy link

amirmahdifard commented Jan 13, 2025

@jmdaweb hello, I have a question. If this gets merged in nvda, will it be stil compatable with the nvda remote server relay implimentation made by you? because we and so many people are using that nice server to host their own remote servers. Whel it was compatible with nvda remote and tele nvda, so i just wanted to make sure it is also compatible with this core remote feature as wel.

@ctoth ctoth marked this pull request as ready for review January 13, 2025 16:32
@ctoth ctoth requested review from a team as code owners January 13, 2025 16:32
@LeonarddeR
Copy link
Collaborator

@ctoth you mentioned:

We wish to improve the Braille experience from the existing baseline in the 2.6.4 add-on.

Could you elaborate on this? In what sense do you want to improve the braille experience?

@ctoth
Copy link
Contributor Author

ctoth commented Jan 14, 2025

In what sense do you want to improve the braille experience?

I am not personally a Braille user but I have heard reports of issues where the master and slave displays conflict or both try and show information, or the Master blinks between its own text and the other text with the default 2.6.4 add-on.

I don't know if anything at all is wrong, or if something is wrong and it is specific to certain models, or if nothing is wrong and the current Remote works fine. I do know that we tested this implementation from both connection types, with and without the UAC transition, etc. and it seems to work well.

@seanbudd seanbudd self-requested a review January 15, 2025 01:19
@LeonarddeR
Copy link
Collaborator

If braille is on par with current remote, I guess we can leave it as is and then improve it if need be.

Copy link
Collaborator

@LeonarddeR LeonarddeR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a first cursory review.

Copy link
Member

@SaschaCowley SaschaCowley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a partial preliminary review

ctoth and others added 11 commits March 1, 2025 16:47
- Add comprehensive docstring
- Replace log.error with log.exception for better error tracking
- Move keyConnector cleanup to finally block to ensure proper resource handling
- Remove unused exception variable from catch clause
Add explicit parameter name 'backlog' and explanatory comment to the socket.listen() call to improve code readability and make the purpose of the value 5 more obvious to future maintainers.
- Standardize parameter naming from snake_case to camelCase (bind_host → bindHost)
- Add docstrings to key classes and methods in server.py
- Use more specific is_file() check instead of exists() for certificate fingerprint
- Clean up parameter naming across secureDesktop.py and server.py
@valiant8086
Copy link

Just checking, we will still be able to do that, right? I have a relay server running and on the same machine I have another port to control NVDA but not through the relay server.

# Conflicts:
#	source/config/configSpec.py
#	source/config/profileUpgradeSteps.py
Copy link
Member

@Qchristensen Qchristensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reads well, great work everyone!

@SaschaCowley
Copy link
Member

@coderabbitai resolve

@SaschaCowley
Copy link
Member

@amirmahdifard @Aarushb
Please open new issues for any features you believe should be added to NVDA's Remote feature.

@valiant8086
Assuming you're asking whether we're keeping the ability to host a control server in NVDA - yes, we have no plans to remove that feature.

@seanbudd seanbudd merged commit 495dee3 into nvaccess:master Mar 7, 2025
5 checks passed
@seanbudd
Copy link
Member

seanbudd commented Mar 7, 2025

Very nice work @SaschaCowley and @ctoth

@LeonarddeR LeonarddeR mentioned this pull request Mar 7, 2025
@tgeczy
Copy link

tgeczy commented Mar 7, 2025

This is amazing! Unfortunately self-hosted servers fail in this implementation, so I'm keeping my add-on copy around for just a bit longer, but am glad to see this become part of core. Error log:

Traceback (most recent call last):
  File "wx\core.pyc", line 3427, in <lambda>
  File "gui\__init__.pyc", line 891, in run
  File "remoteClient\client.pyc", line 253, in handleDialogCompletion
  File "remoteClient\client.pyc", line 393, in startControlServer
  File "remoteClient\server.pyc", line 274, in __init__
  File "remoteClient\server.pyc", line 298, in createServerSocket
TypeError: listen() takes no keyword arguments

For non-self-hosted server connections (as to a remote URL), it works without this error popping up, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
conceptApproved Similar 'triaged' for issues, PR accepted in theory, implementation needs review. release/blocking this issue blocks the milestone release
Projects
None yet
Development

Successfully merging this pull request may close these issues.