Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update NVDA Controller client to support speaking SSML sequences #15734

Merged
merged 27 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
1b0ff0a
Add support to speak SSMl to the NVDA controller client
LeonarddeR Oct 26, 2023
6a873da
Small fixups
LeonarddeR Nov 2, 2023
a35099d
Apply suggestions from code review
LeonarddeR Nov 3, 2023
75abc81
Use 32 bit enums since they play better with ctypes defaults
LeonarddeR Nov 4, 2023
2f7499a
Merge remote-tracking branch 'origin/master' into nvdaControllerClient
LeonarddeR Nov 4, 2023
a773456
Use user configured symbol level for -1
LeonarddeR Nov 4, 2023
78bb6b0
Use enum for symbol level
LeonarddeR Nov 4, 2023
499b8d7
Validation fixes
LeonarddeR Nov 4, 2023
c4e3d5f
Add enum comments
LeonarddeR Nov 4, 2023
4597568
Lint
LeonarddeR Nov 4, 2023
0762fa5
Return annotation
LeonarddeR Nov 4, 2023
d1fde7b
Merge remote-tracking branch 'origin/master' into nvdaControllerClient
LeonarddeR Nov 6, 2023
fb78bb9
Remove unnecessary queueHandler imports
LeonarddeR Nov 6, 2023
fb581d8
Add enum docstrings
LeonarddeR Nov 6, 2023
86e6eeb
Update source/NVDAHelper.py
LeonarddeR Nov 6, 2023
f1e8d33
Safer approach
LeonarddeR Nov 6, 2023
9ef20ef
Update docs
LeonarddeR Nov 6, 2023
af3cec1
Fixup remote build
LeonarddeR Nov 7, 2023
991a1dd
Fixup changelog
LeonarddeR Nov 7, 2023
ae1f2e5
Lint
LeonarddeR Nov 7, 2023
5e5b187
Readme fix
LeonarddeR Nov 7, 2023
3f93d21
Update to the C# example
LeonarddeR Nov 7, 2023
9191f1b
update c example
LeonarddeR Nov 7, 2023
41e38c2
Apply suggestions from code review
LeonarddeR Nov 8, 2023
b0a0854
Review actions
LeonarddeR Nov 8, 2023
0cbbab3
Merge remote-tracking branch 'origin/master' into nvdaControllerClient
LeonarddeR Nov 23, 2023
ab3a08b
Expand docs about sync vs async
LeonarddeR Nov 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions extras/controllerClient/readme.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,19 @@
# NVDA Controller Client API 1.0 Documentation
# NVDA Controller Client API 1.1 Documentation

## Introduction

This client API allows an application to communicate with NVDA, in order to do such things as speak text or braille a message.
This client API allows an application to communicate with NVDA 2024.1 and above, in order to do such things as speak text or braille a message.

The client API is implemented as a dll (dynamic link library). The functions in this dll can be called from any programming language that supports looking up and calling of any symbol in a dll (such as ctypes in Python), or by linking to it for languages like C and C++.

## Compatibility notice

Version 1.1 of the controller client was introduced in NVDA 2024.1, offering support to speak SSML speech sequences.
While this version of the client adds more functionality, it is *not compatible* with NVDA 2023.3 and below.
If you do not necessarily rely on the support to speak SSML, you are encouraged to use version 1.0 of the client library that is linked below.

## Security practices

Developers should be aware that NVDA runs on the lock screen and [secure screens](https://www.nvaccess.org/files/nvda/documentation/userGuide.html#SecureScreens).
Before providing information to the end user (e.g. via `nvdaController_speakText`), developers should check if Windows is locked or running on a secure screen to prevent secure data being leaked.

Expand All @@ -18,6 +25,8 @@ You can build locally or download pre-built, details:
- **Latest, in development version:**
- The libraries are built by Appveyor (our CI).
- Downloads are available from the artifacts tab.
- **Latest build of version 1.0 of the library, supporting NVDA 2023.3 and below:**
- Download the `nvda_2023.3_controllerClient.zip` file: [Direct link](https://www.nvaccess.org/files/nvda/releases/2023.3/nvda_2023.3_controllerClient.zip)
- **Build them yourself:**
- Follow the project `readme.txt` for general build requirements/dependencies.
- Run `scons source client`
Expand Down
31 changes: 29 additions & 2 deletions extras/controllerClient/x86/example_python.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,40 @@
res = clientLib.nvdaController_testIfRunning()
if res != 0:
errorMessage = str(ctypes.WinError(res))
ctypes.windll.user32.MessageBoxW(0, "Error: %s" % errorMessage, "Error communicating with NVDA", 0)
ctypes.windll.user32.MessageBoxW(0, f"Error: {errorMessage}", "Error communicating with NVDA", 0)

# Speak and braille some messages
for count in range(4):
clientLib.nvdaController_speakText("This is a test client for NVDA")
clientLib.nvdaController_brailleMessage("Time: %g seconds" % (0.75 * count))
time.sleep(0.625)
clientLib.nvdaController_cancelSpeech()
clientLib.nvdaController_speakText("This is a test client for NVDA!")


# Test SSML output
@ctypes.WINFUNCTYPE(ctypes.c_ulong, ctypes.c_wchar_p)
def onMarkReached(name):
print(f"Reached SSML mark with name: {name}")
return 0


ctypes.cast(clientLib._nvdaController_onSsmlMarkReached, ctypes.POINTER(ctypes.c_void_p)).contents.value = (
ctypes.cast(onMarkReached, ctypes.c_void_p).value
)
ssml = (
'<speak>'
'This is one sentence. '
'<mark name="test" />'
'<prosody pitch="200%">This sentense is pronounced with higher pitch.</prosody>'
'<mark name="test2" />'
'This is a third sentence. '
'<mark name="test3" />'
'This is a fourth sentence. We will stay silent for a second after this one.'
'<break time="1000ms" />'
'<mark name="test4" />'
'This is a fifth sentence. '
'<mark name="test5" />'
'</speak>'
)
clientLib.nvdaController_speakSsml(ssml, 0, 0, False)
clientLib.nvdaController_brailleMessage("Test completed!")
11 changes: 10 additions & 1 deletion nvdaHelper/client/client.cpp
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/*
This file is a part of the NVDA project.
URL: http://www.nvda-project.org/
Copyright 2006-2010 NVDA contributers.
Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License version 2.1, as published by
the Free Software Foundation.
Expand Down Expand Up @@ -47,3 +47,12 @@ BOOL WINAPI DllMain(HINSTANCE hModule,DWORD reason,LPVOID lpReserved) {
}
return TRUE;
}

error_status_t(__stdcall *_nvdaController_onSsmlMarkReached)(const wchar_t*) = nullptr;

error_status_t __stdcall nvdaController_onSsmlMarkReached(const wchar_t* mark) {
if (_nvdaController_onSsmlMarkReached == nullptr) {
return ERROR_CALL_NOT_IMPLEMENTED;
}
return _nvdaController_onSsmlMarkReached(mark);
}
5 changes: 4 additions & 1 deletion nvdaHelper/client/nvdaControllerClient.def
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
;;;
;This file is a part of the NVDA project.
;URL: http://www.nvda-project.org/
;Copyright 2006-2010 NVDA contributers.
;Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.
;This program is free software: you can redistribute it and/or modify
;it under the terms of the GNU Lesser General Public License version 2.1, as published by
;the Free Software Foundation.
Expand All @@ -17,3 +17,6 @@ EXPORTS
nvdaController_speakText
nvdaController_cancelSpeech
nvdaController_brailleMessage
nvdaController_getProcessId
nvdaController_speakSsml
_nvdaController_onSsmlMarkReached
5 changes: 4 additions & 1 deletion nvdaHelper/interfaces/nvdaController/nvdaController.acf
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/*
This file is a part of the NVDA project.
URL: http://www.nvda-project.org/
Copyright 2006-2010 NVDA contributers.
Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License version 2.1, as published by
the Free Software Foundation.
Expand All @@ -20,4 +20,7 @@ interface NvdaController {
[fault_status,comm_status] speakText();
[fault_status,comm_status] cancelSpeech();
[fault_status,comm_status] brailleMessage();
[fault_status,comm_status] getProcessId();
[fault_status,comm_status] speakSsml();
[fault_status,comm_status] onSsmlMarkReached();
}
61 changes: 57 additions & 4 deletions nvdaHelper/interfaces/nvdaController/nvdaController.idl
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/*
This file is a part of the NVDA project.
URL: http://www.nvda-project.org/
Copyright 2006-2010 NVDA contributers.
Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License version 2.1, as published by
the Free Software Foundation.
Expand All @@ -15,7 +15,7 @@ http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html
cpp_quote("/*")
cpp_quote("This file is a part of the NVDA project.")
cpp_quote("URL: http://www.nvda-project.org/")
cpp_quote("Copyright 2006-2010 NVDA contributers.")
cpp_quote("Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.")
cpp_quote("This program is free software: you can redistribute it and/or modify")
cpp_quote("it under the terms of the GNU Lesser General Public License version 2.1, as published by")
cpp_quote("the Free Software Foundation.")
Expand All @@ -26,12 +26,39 @@ cpp_quote("This license can be found at:")
cpp_quote("http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html")
cpp_quote("*/")

typedef enum {
/**
* Indicates that a speech sequence should have normal priority.
*/
SPEECH_PRIORITY_NORMAL = 0,
/**
* Indicates that a speech sequence should be spoken after the next utterance of lower priority is complete.
*/
SPEECH_PRIORITY_NEXT = 1,
/**
* Indicates that a speech sequence is very important and should be spoken right now,
* interrupting low priority speech.
* After it is spoken, interrupted speech will resume.
* Note that this does not interrupt previously queued speech at the same priority.
*/
SPEECH_PRIORITY_NOW = 2
} SPEECH_PRIORITY;

typedef enum {
SYMBOL_LEVEL_NONE = 0,
SYMBOL_LEVEL_SOME = 100,
SYMBOL_LEVEL_MOST = 200,
SYMBOL_LEVEL_ALL = 300,
SYMBOL_LEVEL_CHAR = 1000,
SYMBOL_LEVEL_UNCHANGED = -1
} SYMBOL_LEVEL;

/**
* Allows the controling of NVDA from a remote process
*/
[
uuid(DFF50B99-F7FD-4ca7-A82C-DAEB3E025295),
version(1.0),
version(1.1),
]
interface NvdaController {

Expand All @@ -41,7 +68,7 @@ interface NvdaController {
error_status_t __stdcall testIfRunning();

/**
* Instructs NVDA to speak the given text.
* Instructs NVDA to speak the given text message.
* @param text the text to speak.
*/
error_status_t __stdcall speakText([in,string] const wchar_t* text);
Expand All @@ -57,4 +84,30 @@ interface NvdaController {
*/
error_status_t __stdcall brailleMessage([in,string] const wchar_t* message);

/**
* Retrieves the process identifier (PID) of NVDA's process.
*/
error_status_t __stdcall getProcessId([out] unsigned long* pid);

/**
* Instructs NVDA to speak the given Speech Synthesis Markup Language (SSML).
* @param ssml The ssml to speak.
* @param symbolLevel The symbol verbosity level.
* @param priority The priority of the speech sequence.
* @param asynchronous Whether SSML should be spoken asynchronously.
* If TRUE, returns instantly.
* If FALSE, returns either when the speech sequence is completed or canceled.
*/
error_status_t __stdcall speakSsml(
[in, string] const wchar_t* ssml,
[in, defaultvalue(SYMBOL_LEVEL_UNCHANGED)] const SYMBOL_LEVEL symbolLevel,
[in,defaultvalue(SPEECH_PRIORITY_NORMAL)] const SPEECH_PRIORITY priority,
[in, defaultvalue(TRUE)] const boolean asynchronous
);

/**
* Called by NVDA when a mark in provided SSML is reached after calling speakSsml synchronously.
* @param mark The name of the reached mark.
*/
[callback] error_status_t __stdcall onSsmlMarkReached([in, string] const wchar_t* mark);
};
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/*
This file is a part of the NVDA project.
URL: http://www.nvda-project.org/
Copyright 2006-2010 NVDA contributers.
Copyright 2006-2023 NV Access Limited, Leonard de Ruijter.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2.0, as published by
the Free Software Foundation.
Expand All @@ -13,22 +13,52 @@ This license can be found at:
*/

#include <local/nvdaController.h>
#include <windows.h>

error_status_t(__stdcall *_nvdaController_speakText)(const wchar_t*) = nullptr;

error_status_t(__stdcall *_nvdaController_speakText)(const wchar_t*);
error_status_t __stdcall nvdaController_speakText(const wchar_t* text) {
if (_nvdaController_speakText == nullptr) {
return ERROR_CALL_NOT_IMPLEMENTED;
}
return _nvdaController_speakText(text);
}

error_status_t(__stdcall *_nvdaController_speakSsml)(const wchar_t*, const SYMBOL_LEVEL, const SPEECH_PRIORITY, const boolean) = nullptr;

error_status_t __stdcall nvdaController_speakSsml(const wchar_t* ssml, const SYMBOL_LEVEL symbolLevel, const SPEECH_PRIORITY priority, const boolean asynchronous) {
if (_nvdaController_speakSsml == nullptr) {
return ERROR_CALL_NOT_IMPLEMENTED;
}
return _nvdaController_speakSsml(ssml, symbolLevel, priority, asynchronous);
}

error_status_t(__stdcall *_nvdaController_cancelSpeech)();

error_status_t __stdcall nvdaController_cancelSpeech() {
if (_nvdaController_cancelSpeech == nullptr) {
return ERROR_CALL_NOT_IMPLEMENTED;
}
return _nvdaController_cancelSpeech();
}

error_status_t(__stdcall *_nvdaController_brailleMessage)(const wchar_t*);

error_status_t __stdcall nvdaController_brailleMessage(const wchar_t* text) {
if (_nvdaController_brailleMessage == nullptr) {
return ERROR_CALL_NOT_IMPLEMENTED;
}
return _nvdaController_brailleMessage(text);
}

error_status_t __stdcall nvdaController_getProcessId(unsigned long* pid) {
if (pid == nullptr) {
return ERROR_INVALID_PARAMETER;
}
*pid = GetCurrentProcessId();
return ERROR_SUCCESS;
}

error_status_t __stdcall nvdaController_testIfRunning() {
return 0;
return ERROR_SUCCESS;
}
2 changes: 2 additions & 0 deletions nvdaHelper/local/nvdaHelperLocal.def
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ EXPORTS
_nvdaController_brailleMessage
_nvdaController_cancelSpeech
_nvdaController_speakText
_nvdaController_speakSsml
nvdaController_onSsmlMarkReached
_nvdaControllerInternal_vbufChangeNotify
displayModel_getWindowTextInRect
displayModel_getFocusRect
Expand Down
2 changes: 1 addition & 1 deletion nvdaHelper/local/rpcSrv.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ using namespace std;
typedef RPC_STATUS(RPC_ENTRY *RpcServerRegisterIf3_functype)(RPC_IF_HANDLE,UUID __RPC_FAR*,RPC_MGR_EPV __RPC_FAR*,unsigned int,unsigned int,unsigned int,RPC_IF_CALLBACK_FN __RPC_FAR*,void __RPC_FAR*);

RPC_IF_HANDLE availableInterfaces[]={
nvdaController_NvdaController_v1_0_s_ifspec,
nvdaController_NvdaController_v1_1_s_ifspec,
nvdaControllerInternal_NvdaControllerInternal_v1_0_s_ifspec
};

Expand Down
2 changes: 1 addition & 1 deletion nvdaHelper/local/sconscript
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ localLib=env.SharedLibrary(
nvdaInProcUtilsRPCClientSource,
displayModelRPCClientSource,
'rpcSrv.cpp',
'nvdaController.c',
'nvdaController.cpp',
winIPCUtilsObj,
controllerRPCServerSource,
'nvdaControllerInternal.c',
Expand Down
21 changes: 21 additions & 0 deletions nvdaHelper/remote/nvdaController.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
/*
This file is a part of the NVDA project.
Copyright 2023 NV Access Limited, Leonard de Ruijter.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License version 2.0, as published by
the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
This license can be found at:
http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
*/

#define WIN32_LEAN_AND_MEAN

#include <windows.h>
#include <remote/nvdaController.h>

error_status_t __stdcall nvdaController_onSsmlMarkReached(const wchar_t* mark) {
return ERROR_SUCCESS;
}
1 change: 1 addition & 0 deletions nvdaHelper/remote/sconscript
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ source = [
vbufRPCServerSource,
winIPCUtilsObj,
controllerRPCClientSource,
"nvdaController.cpp",
controllerInternalRPCClientSource,
"sysListView32.cpp",
"winword.cpp",
Expand Down
Loading