-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathnativesteps.txt
213 lines (160 loc) · 9.31 KB
/
nativesteps.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
Steps needed to establish a full audio/visual/data connection to the |pipe| agent.
This is a bit intricate beacuse we manage all the security over the dataChannel,
meaning that we have to create a datachannel first, then upgrade it to support
realtime media too.
A) Pairing -
A users smartphone needs to be paired with a new IoT device this happens by
exchanging X509 certificates. We use a certificate and nonce transfered
over a secure channel to validate this exchange.
(reference javascript is in https://pi.pe/iot/fclaim.html - which uses a scanned QR code to transfer the fingerprint and nonce)
Steps:
1) Create a certificate: The webRTC infastructure will create a new one when needed. - in future we will use PeerConnection.generateCertificate() to have more control over this process.
Create an RTCPeerConnection using the factory or static method available on your platform. This will give you the option to set STUN: and Turn: server URLs and credentials - it is also when you can apply a previously stored client certificate.
2) request a datachannel:
avrelay = pc.createDataChannel('avrelay', {});
3) parse the generated sdp
pc.createOffer()
.then(function (localDesc) {
sdpobj = Phono.sdp.parseSDP(localDesc.sdp);
})
4) calculate the local id
var id = JSON.stringify(sdpobj.contents[0].fingerprint.print);
id = id.split(":").join("");
id = id.split('"').join("");
This example uses the sdp parser defined in https://pi.pe/iot/js/phono.sdp.js
You are encouraged to replicate the functionality in your development language.
5) use the fingerprint/id to connect to a websocket on the rendezvous server -
This will forward messages between the smartphone and the
Open a websocket to https://pi.pe/websocket/?finger={$id}
6) obtain the IoT device's Id and the nonce - this may be a QR, a BTLE beacon
or some other secure back channel.
7) create a sessionID - that will be unique to this run
we use a concatenation of $id-$iotId-$timeStamp
8) calculate the nonsense value this is:
sha256($iotId + ":" + nonce + ":" + $id).tohex().toUpperCase();
9) Now you need to format a message to send to your IoT device
This is json with the following entries:
to: $IoTId
from: $id
session: $session
nonsense: $nonsense
type: "offer"
sdp: sdpObj
10) set the localDescription
pc.setLocalDescription(localDesc);
11) in the sucess call back from setLocalDescription,
send the message you just built
12) attach a method to the onicecandidate event that sends json objects for each of the candidates.
var can_j = Phono.sdp.parseCandidate("a=" + cand.candidate);
var candy = {
"to": $iotId,
"type": 'candidate',
"candidate": can_j,
"session": $session,
"from": $id,
"sdpMLineIndex": cand.sdpMLineIndex,
"nonsense": $nonsense
};
this.ws.send(JSON.stringify(candy));
if candidates arrive before you have completed 11) you should queue them for sending after 11) completes.
13) await messages on the web socket. The messages will be in a similar form to the ones you just sent. Expect 1 or with type='candidate' and a single message type='answer'. You should check that the session matches the one you sent with the offer. If it does not, you should discard the whole message.
14) the candidates you should un-parse back to a string and add to the peer connectio.
if (data.type == 'candidate') {
var jc = {
sdpMLineIndex: data.sdpMLineIndex,
candidate: Phono.sdp.buildCandidate(data.candidate)
};
var nc = "Huh? ";
nc = new RTCIceCandidate(jc);
pc.addIceCandidate(nc);
}
15) the offer you also un-parse back to a long SDP string and set it as the Peerconnection remoteDescription:
if (data.type == 'answer') {
var sdp = Phono.sdp.buildSDP(data.sdp);
var message = {'sdp': sdp, 'type': data.type};
var rtcd = new RTCSessionDescription(message);
pc.setRemoteDescription(rtcd).then(function () {
// step 16;
}).catch( function (e) {
console.log("Set Remote description error " + e);
});
}
16) At this point an onOpen() method should be called for the datachannel avrelay . If this happens, pairing has been sucessful.
17) If this does not happen you can inspect the peerconnection state. Also Looking at the Device Logfiles can be informative. However the device does not send useful error replies in the case of a failed pairing in order to avoid giving an attacker a clue. Messages with invalid to, nonsense, session values will be ignored
B) upgrading to media.
18) once you have a live avrelay dataChannel, you can request the device
'upgrades' the connection.
19) first however you need to gain access to the microphone and give it to the peer connection:
var constraints = {video: false, audio: true};
navigator.mediaDevices.getUserMedia(constraints).then(
function (stream) {
pc.addStream(stream);
console.log("sending upgrade request");
avrelay.send(JSON.stringify({type: "upgrade", time: Date.now()}));
})
20) the onMessage callback for the avrelay datachannel will fire with a 'patch' message. This contains the new info that needs to be patched into the sdp.
avrelay.onmessage = function (evt) {
var message = JSON.parse(evt.data);
if ((message.type == "offer" )
|| (message.type == "answer")) {
if (message.vinfo && message.ainfo) {
addavpatch(message);
}
var patched = Phono.sdp.patch(duct.peerCon.remoteDescription.sdp, message);
var rtcd = new RTCSessionDescription(patched);
pc.setRemoteDescription(rtcd, srd, srdfail);
}
};
addavpatch() adds a set of rules (see fav.html for the static object) and applies the patch.
21) when the new upgraded remote description is sucessfully set, create a new
answer, set it and then send it to the device
pc.createAnswer(function (desc) {
pc.setLocalDescription(desc, function () {
var desc = duct.peerCon.localDescription;
var sdp = Phono.sdp.parseSDP(desc.sdp);
var mess = {type: desc.type, sdp: sdp, tick: Date.now()};
console.log("sending " + JSON.stringify(mess));
avrelay.send(JSON.stringify(mess));
}
}
(Notice that if you have added a local audio stream, this will be enabled here too)
22) Audio and video should now start flowing, causing pc.ontrack() to fire
with 2 new streams, which you should attach to audio and video renderers.
23) You can now manage the activity of the 3 streams. The streams originating from the |pipe| device can be enabled and disabled by sending micon/micoff cameraon/camreaoff commands down the avrelay datachannel.
Video:
if (enable) {
avrelay.send(JSON.stringify({type: "cameraon"}));
video.play();
} else {
avrelay.send(JSON.stringify({type: "cameraoff"}));
video.pause();
}
Microphone:
if (enable) {
vidcc.send(JSON.stringify({type: "micon"}));
audio.play();
} else {
vidcc.send(JSON.stringify({type: "micoff"}));
audio.pause();
}
24) The audio playout to the |pipe| device is a bit more complex since it requires a fresh offer-answer round-trip.
You obtain an audio media stream in the usual way (GetUserMedia) and then add it to the peerconnection:
duct.peerCon.addStream(stream);
or remove it:
var ast = duct.peerCon.getLocalStreams()[0];
var at = ast.getAudioTracks()[0];
ast.removeTrack(at);
duct.peerCon.getSenders()[0].track.stop();
25) Either of these actions will cause peerConnection.onnegotiationneeded() to fire. This can be used to perfrom the O/A - since we already know everything we need to, we can just increment the index of the existing offer using patch:
duct.peerCon.onnegotiationneeded = function () {
var message = {type: "offer", patches: [ {
"action": "increment", "at": "o=-", "field": 2 } ] };
var patched = Phono.sdp.patch(duct.peerCon.remoteDescription.sdp, message);
var rtcd = new RTCSessionDescription(patched);
duct.peerCon.setRemoteDescription(rtcd).then(function (r) {
srd(r)
})};
the existing logic in 21 can be reused to send the new answer to |pipe|
(Note that if you tell |pipe| that you will send audio data, but don't then it will drop the whole peerconnection after about 5 seconds).
C) subsequent connections:
Once paired, if you use the same client certificate to the same IoT device, you can skip the calculation of the nonsense and send a blank value in the offer an candidate messages. This re-uses the trust relationship created by the initial pairing.