Get raw data from MediaStream

[piedshag]

Please excuse me if this is the wrong place to propose this. It would be really useful to have access to the raw data coming out of the MediaStream so it could be processed before being shipped off to a remote peer or displayed to the user. The data could perhaps be emitted in an ondata event which would emit raw data similar to that of the MediaRecorder. Would anyone else find something along these lines useful?

[alvestrand]

There's no such thing as "the raw data". There are many different possible internal representations, and we do not constrain which of them browsers use.
If you want the bitmap, paint it to a canvas via a video element.
If you want a compressed media format, use the MediaStreamRecorder.
I think we have the interfaces we need. What scenario isn't covered?

[piedshag]

I believe the Media recorder only gives you the media when the stream has finished or you end the recorder. I would like the media as it is produced and as far as I am aware that is not possible with the Media recorder.

[alvestrand]

See the "requestData" method in https://rawgit.com/w3c/mediacapture-record/master/MediaRecorder.html#methods - you can get the so-far-available data at any time.

[piedshag]

Thankyou! I should have done some proper research

[alvestrand]

Seems that no action is needed. Closing.

[Pehrsons]

There's also https://w3c.github.io/mediacapture-worker/ being drafted.

[embirico]

Hey @alvestrand, sorry to resuscitate an old thread, but I currently have a use case for getting the raw data, and a scenario that doesn't seem to be currently covered:

I have an application with fairly long-lived streams (e.g. 1 hour long at a time), and I'd like to provide users with a way to capture short clips of the most recent, say, 15 seconds.

Currently it seems that creating a buffer of the most recent data directly from the MediaStream is not supported.
Also, it seems that keeping a buffer of the most recent blobs from MediaRecorder doesn't work, due to the spec stating that

When multiple Blobs are returned (because of timeslice or requestData()), the individual Blobs need not be playable, but the combination of all the Blobs from a completed recording MUST be playable.

What do you think of supporting this scenario, or is there an interface I'm missing? Thank you!

[bc]

@embirico did you end up finding a solution? I have a similar use case

[aboba]

@bc @embirico As Harald has noted, access to raw video is already supported. Therefore the focus for new APIs is on scenarios requiring higher performance than is currently achievable via Canvas or MediaRecorder. We have the following WebRTC-NV use cases for access to raw video (raw audio access is already provided via WebAudio Worklets):
Funny Hats
Machine Learning

So far, there are two specifications under development which could conceivably address these use cases:
Insertable Streams
WebCodecs

In the current Origin Trial, Insertable Streams provides access to encoded video, but only a minor API change would be required to add an insertion point prior to frame encoding (sender) or after frame decoding (receiver). Since raw video is much larger than encoded video, it is not clear that the existing Insertable Streams API would provided sufficient performance for the Machine Learning use case, in particular.

WebCodecs is still early in its incubation, but it may provide access to encoded bitstreams and raw (decoded) video.

[BlobTheKat]

If you want the bitmap, paint it to a canvas via a video element. If you want a compressed media format, use the MediaStreamRecorder. I think we have the interfaces we need. What scenario isn't covered?

Capturing microphone data to perform some processing and/or upload it over network. For some applications implementing WebRTC is more trouble than it's worth and reusing an existing http/websocket interface makes the most sense. The only way to obtain the raw data currently is with a script processor node on an AudioContext which can capture input data. This definitely feels like the wrong way to do it. MediaRecorder is not sufficient as the different supported output formats by different browsers are too restrictive to be useful in most cases

[James-E-A]

If you want the bitmap, paint it to a canvas via a video element.

Won't that still create a round-trip of generation loss?

https://en.wikipedia.org/wiki/Generation_loss#Transcoding

And even if it doesn't, that still only covers video, not audio...

[BlobTheKat]

I think a lot of the problem with more modern web APIs is that they are designed high level, and are uninteroperable with existing lower level APIs. Actually, the problem isn't that these APIs are high level, it's that they were designed quite narrow-mindedly with often a single use-case in mind, take for example

• Service workers, designed exclusively for caching web pages offline (albeit privacy and security concerns fuel most of its prohibitive restrictions)
• MediaStream, designed exclusively for WebRTC, mostly incapable of anything else
• Web workers are a missed potential for a very light and neat way to sandbox untrusted code since they were designed to only work with the same origin. This restriction can easily be bypassed and yet still doesn't provide the sandboxing effect.
• IndexedDB, with its restrictions on operation timing, doesn't prohibit but significantly increases the overhead of any usage pattern that it wasn't intended for.

What all of these have in common is that they were only designed with at most a few use-cases in mind and therefore the specs often unintentionally completely prohibit any more creative use-cases, causing the need for new APIs later on, increasing to already-heavy browser engines

For this specific issue, it will not get solved with a getRawData-like API but rather a fundamental change in perspective when it comes to designing specifications. As a shorter-term solution, MediaRecorder should standardise more formats, including raw-like formats like wav for audio and mov for audio-video. I think it would also be very useful to expose an option similar to WebSocket.binaryType where we could choose between ArrayBuffers or Blobs

TL;DR we suck at making specification and that's why we have issues like this

[fippo]

This particular case was solved a while ago by MediaStreamTrackProcessor no?

[BlobTheKat]

This particular case was solved a while ago by MediaStreamTrackProcessor no?

Neat, didn't see that! Can't wait to start using it for my apps in... ~4 years?

[bradisbell]

@BlobTheKat Yeah, that's assuming it even gets built in another browser before someone decides to kill the spec because there's only one implementation (Chromium). (e.g. #958 ... even if something is marked "at risk", you can bet that nobody is going to bother with it at that point.) At the moment, there's debate as to whether to even allow the track processor for audio. (w3c/mediacapture-transform#29)

This is the nature of it. We benefit from all this platform development, but give up a lot of influence over it in the process.

[youennf]

Safari latest beta supports track processor and video track generator, as standardized. Please have a try.

[BlobTheKat]

Safari latest beta supports track processor and video track generator, as standardized. Please have a try.

It will still be a few years before that version and above have a large enough market share to make this a viable option, not even to mention we're still waiting on firefox...

[James-E-A]

Neat, didn't see that! Can't wait to start using it for my apps in... ~4 years?

It will still be a few years before that version and above have a large enough market share to make this a viable option, not even to mention we're still waiting on firefox...

Just for the comfort of anyone stumbling across this thread in the future, this is being tracked browser-side at https://bugzilla.mozilla.org/show_bug.cgi?id=1749532