Skip to content

Commit 7e346cf

Browse files
jan-ivarjan-ivar
committed
Replace pipeToController with [[transformFrameAlgorithm]] approach
1 parent a991ba0 commit 7e346cf

File tree

1 file changed

+73
-46
lines changed

1 file changed

+73
-46
lines changed

‎index.bs‎

Lines changed: 73 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -93,25 +93,27 @@ partial interface RTCRtpReceiver {
9393
};
9494
</pre>
9595

96-
This allows manipulation of <dfn>encoded frames</dfn> in the media
96+
This API allows manipulation of <dfn>encoded frames</dfn> in the media
9797
pipeline between the processing steps of an {{RTCRtpSender}}'s
9898
underlying <dfn>encoder</dfn> and <dfn>packetizer</dfn>, and/or
9999
between an {{RTCRtpReceiver}}'s underlying <dfn>depacketizer</dfn>
100100
and <dfn>decoder</dfn>.
101101

102102
The [=encoder=] and [=depacketizer=] each have a
103+
<dfn>[[processedFramesQueue]]</dfn> internal slot initialized to
104+
an empty [=queue=], and a
103105
<dfn>[[transformFrameAlgorithm]]</dfn> internal slot initialized to
104106
the <dfn>passthrough algorithm</dfn> which, given an [=encoded frame=]
105107
|frame|, is to return |frame|.
106108

107-
Whenever the [=encoder=] outputs a [=encoded frame|frame=], the user
108-
agent MUST invoke the [=encoder=].[=[[transformFrameAlgorithm]]=] with
109+
Whenever the [=encoder=] outputs an [=encoded frame=], the user
110+
agent MUST invoke the [=encoder=].[=[[transformFrameAlgorithm]]=] on
109111
it and pass the result to the associated [=packetizer=] in place of
110112
the original frame.
111113

112-
Whenever the [=depacketizer=] outputs a [=encoded frame|frame=], the
114+
Whenever the [=depacketizer=] outputs an [=encoded frame=], the
113115
user agent MUST invoke the [=depacketizer=].[=[[transformFrameAlgorithm]]=]
114-
with it and pass the result to the associated [=decoder=] in place of
116+
on it and pass the result to the associated [=decoder=] in place of
115117
the original frame.
116118

117119
## Extension operation ## {#operation}
@@ -120,12 +122,6 @@ At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the followin
120122
1. Initialize [=this=].`[[transform]]` to null.
121123
1. Initialize [=this=].`[[pipeToController]]` to null.
122124
1. Initialize [=this=].`[[frameSource]]` to [=this=]'s [=encoder=] if [=this=] is an {{RTCRtpSender}} or [=this=]'s [=depacketizer=] otherwise.
123-
1. [=Queue a task=] to run the following steps:
124-
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
125-
1. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
126-
1. Let |frameSource| be [=this=].`[[frameSource]]`.
127-
1. [=In parallel=], set |frameSource|.[=[[transformFrameAlgorithm]]=] to the
128-
[=passthrough algorithm=].
129125

130126
### Stream processing ### {#stream-processing}
131127

@@ -139,10 +135,7 @@ For those reasons, streams backpressure is disabled in WebRTC encoded transforms
139135
</p>
140136

141137
The <dfn abstract-op>readEncodedData</dfn> algorithm is given an {{RTCRtpScriptTransformer}}
142-
|transformer| as parameter. It is defined by running the following steps:
143-
1. Wait for a frame to be produced by |transformer|.{{[[frameSource]]}}.
144-
1. Increment |transformer|.`[[lastEnqueuedFrameCounter]]` by <code>1</code>.
145-
1. Let |frame| be the newly produced frame.
138+
|transformer| as parameter, and |frame| as input. It is defined by running the following steps:
146139
1. Set |frame|.`[[owner]]` to |transformer|.{{[[frameSource]]}}.
147140
1. Set |frame|.`[[counter]]` to |transformer|.{{[[lastEnqueuedFrameCounter]]}}.
148141
1. If |frame|.`[[owner]]` is a [=depacketizer=]:
@@ -169,10 +162,8 @@ The <dfn abstract-op>writeEncodedData</dfn> algorithm is given an {{RTCRtpScript
169162
1. Let |data| be |frame|.`[[data]]`.
170163
1. Let |serializedFrame| be [$StructuredSerializeWithTransfer$](|frame|, « |data| »).
171164
1. Let |frameCopy| be [$StructuredDeserializeWithTransfer$](|serializedFrame|, |frame|'s [=relevant realm=]).
172-
1. Enqueue |frameCopy| for processing as if it came directly from the encoded data source, by running one of the following steps:
173-
* If |transformer|.{{[[frameSource]]}} is an [=encoder=], enqueue |frameCopy| to its
174-
associated [=packetizer=], to be processed [=in parallel=].
175-
* If |transformer|.{{[[frameSource]]}} is a [=depacketizer=], enqueue |frameCopy| to its associated [=decoder=], to be processed [=in parallel=].
165+
1. Let |processedFrame| be |frameCopy|'s underlying [=encoded frame=].
166+
1. [=In parallel=], [=queue/enqueue=] |processedFrame| onto |transformer|.{{[[frameSource]]}}.[=[[processedFramesQueue]]=].
176167
1. Return [=a promise resolved with=] undefined.
177168

178169
On the sender side, as part of [$readEncodedData$], frames produced by the [=encoder=] MUST be [=ReadableStream/enqueued=] into |transformer|.{{[[readable]]}} in the [=encoder=]'s output order.
@@ -183,35 +174,28 @@ On the receiver side, as part of [$readEncodedData$], frames produced by the [=d
183174
To ensure the order is respected, the [=depacketizer=] will typically use RTP packet sequence numbers to reorder RTP packets as needed before [=ReadableStream/enqueuing=] frames into |transformer|.{{[[readable]]}}.
184175
As [$writeEncodedData$] ensures that the transform cannot reorder frames, this will be the order expected by the [=decoder=].
185176

186-
## Extension attribute ## {#attribute}
177+
### RTCRtpTransform common processing ### {#transform-processing}
187178

188179
An RTCRtpTransform has a private slot:
189180
* `[[owner]]` of type {{RTCRtpSender}} or {{RTCRtpReceiver}}, initialized to null.
190181

191-
Each RTCRtpTransform has an <dfn abstract-op for=RTCRtpTransform>association steps</dfn> set, which is empty by default.
182+
Each RTCRtpTransform has an <dfn abstract-op for=RTCRtpTransform>association algorithm</dfn>
183+
and a <dfn abstract-op for=RTCRtpTransform>disassociation algorithm</dfn>, both empty by default.
192184

193-
The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter steps are:
194-
1. Return [=this=].`[[transform]]`.
185+
## Extension attribute ## {#attribute}
195186

196-
The `transform` setter steps are:
187+
The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter steps are to
188+
return [=this=].`[[transform]]`. The setter steps are:
197189
1. Let |transform| be the argument to the setter.
198190
1. If |transform| is not null and |transform|.`[[owner]]` is not null, throw a {{InvalidStateError}} and abort these steps.
199191
1. Set |transform|.`[[owner]]` to [=this=].
200-
1. Initialize |newPipeToController| to a new {{AbortController}}.
201-
1. If [=this=].`[[pipeToController]]` is not null, run the following steps:
202-
1. [=AbortSignal/Add=] the [$apply transform$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=].
203-
2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`.
204-
1. Else, run the [$apply transform$] steps.
205-
1. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
192+
1. Let |oldTransform| be [=this=].`[[transform]]`.
193+
1. If |oldTransform| is not null, run |oldTransform|'s [$disassociation algorithm$].
206194
1. Set [=this=].`[[transform]]` to |transform|.
207-
1. Run the steps in the set of [$association steps$] of |transform| with [=this=].
208-
209-
The <dfn abstract-op>apply transform</dfn> steps are defined as:
210-
1. If |newPipeToController|'s [=AbortController/signal=] is [=AbortSignal/aborted=], abort these steps.
211-
1. [=Assert=] that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`.
212-
<!-- FIXME: Fix aborting. -->
213-
<p class="note">TODO: Specify details of how
214-
[=this=].`[[pipeToController]]` replaces the transform in the media pipeline.</p>
195+
1. If |transform| is not null, run |transform|'s [$association algorithm$] with [=this=].
196+
1. If |transform| is null, run the following steps:
197+
1. Let |frameSource| be [=this=].`[[frameSource]]`.
198+
1. [=In parallel=], set |frameSource|.[=[[transformFrameAlgorithm]]=] to the [=passthrough algorithm=].
215199

216200
This algorithm is defined so that transforms can be updated dynamically.
217201
There is no guarantee on which frame will happen the switch from the previous transform to the new transform.
@@ -920,26 +904,69 @@ interface RTCRtpScriptTransform {
920904
};
921905
</pre>
922906

907+
## Internal slots ## {#RTCRtpScriptTransform-internal-slots}
908+
909+
An {{RTCRtpScriptTransform}} object has the following internal slot:
910+
911+
<table class="data" dfn-for="RTCRtpScriptTransform" dfn-type="attribute">
912+
<thead>
913+
<tr>
914+
<th>Internal Slot
915+
<th>Description (<em>non-normative</em>)
916+
</tr>
917+
</thead>
918+
<tbody>
919+
<tr>
920+
<td><dfn>`[[worker]]`</dfn>
921+
<td class="non-normative">The {{Worker}} provided in the constructor.
922+
</tr>
923+
</tbody>
924+
</table>
925+
923926
## Constructor ## {#RTCRtpScriptTransform-constructor}
924927

925928
The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
929+
1. Initialize [=this=]'s internal slot as follows:
930+
: {{RTCRtpScriptTransform/[[worker]]}}
931+
:: |worker|
926932
1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
927-
1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
933+
1. [=Queue a global task=] on the DOM manipulation [=task source=] with |worker|'s
934+
{{WorkerGlobalScope}} to run the following steps:
928935
1. Let |transformerOptions| be the result of [$StructuredDeserializeWithTransfer$](|serializedOptions|, the current Realm).
929936
1. Let |transformer| be the result of [=RTCRtpScriptTransformer/creating=] a
930937
{{RTCRtpScriptTransformer}} with |transformerOptions|.
931-
1. [=Fire an event=] named <dfn event for="DedicatedWorkerGlobalScope">rtctransform</dfn> using {{RTCTransformEvent}} with {{RTCTransformEvent/transformer}} set to |transformer| on |worker|’s global scope.
938+
1. [=Fire an event=] named <dfn event for="DedicatedWorkerGlobalScope">rtctransform</dfn> using {{RTCTransformEvent}} with {{RTCTransformEvent/transformer}} set to |transformer| on |transformer|’s [=relevant global object=].
932939

933940
// FIXME: Describe error handling (worker closing flag true at RTCRtpScriptTransform creation time. And worker being terminated while transform is processing data).
934941

935942
## Algorithms ## {#RTCRtpScriptTransform-algorithms}
936943

937-
Each {{RTCRtpScriptTransform}} has the following set of [$association steps$], given |rtcObject|:
938-
1. Let |transform| be the {{RTCRtpScriptTransform}} object that owns the [$association steps$].
939-
1. Let |frameSource| be |rtcObject|'s `[[frameSource=]]`.
940-
1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
941-
1. Let |transformer| be the {{RTCRtpScriptTransformer}} object associated to |transform|.
944+
Each {{RTCRtpScriptTransform}} has the following [$association algorithm$], given |rtcObject|:
945+
1. Let |transform| be the {{RTCRtpScriptTransform}} object that owns the [$association algorithm$].
946+
1. Let |frameSource| be |rtcObject|'s `[[frameSource]]`.
947+
1. Let |workerGlobalScope| be |transform|.{{[[worker]]}}'s {{WorkerGlobalScope}}.
948+
1. [=Queue a global task=] on the DOM manipulation [=task source=] with |workerGlobalScope|
949+
to run the following steps:
950+
1. Let |transformer| be the {{RTCRtpScriptTransformer}} object associated with |transform|.
942951
1. Set |transformer|.{{[[frameSource]]}} to |frameSource|.
952+
1. [=In parallel=], set |frameSource|.[=[[transformFrameAlgorithm]]=] to the following steps,
953+
given an [=encoded frame=] |frame| as input:
954+
1. [=Queue a global task=] on the DOM manipulation [=task source=] with |workerGlobalScope|
955+
to run the following steps:
956+
1. Let |transformer| be the {{RTCRtpScriptTransformer}} object associated with |transform|.
957+
1. Let |jsFrame| be a new {{RTCEncodedVideoFrame}} from |frame| if |frame| is a video
958+
frame, or a new {{RTCEncodedAudioFrame}} from |frame| otherwise.
959+
1. Invoke [$readEncodedData$] with |transformer| and |jsFrame|.
960+
1. Wait for |frameSource|.[=[[processedFramesQueue]]=] to become non-empty.
961+
1. Return the result of [=dequeueing=] from |frameSource|.[=[[processedFramesQueue]]=].
962+
963+
Each {{RTCRtpScriptTransform}} has the following [$disassociation algorithm$]:
964+
1. Let |transform| be the {{RTCRtpScriptTransform}} object that owns the [$disassociation algorithm$].
965+
1. [=Queue a global task=] on the DOM manipulation [=task source=] with
966+
|transform|.{{[[worker]]}}'s {{WorkerGlobalScope}} to run the following steps:
967+
1. Let |transformer| be the {{RTCRtpScriptTransformer}} object associated with |transform|.
968+
1. [=ReadableStream/cancel=] |transformer|.{{[[readable]]}}.
969+
1. [=WritableStream/abort=] |transformer|.{{[[writable]]}}.
943970

944971
# <dfn interface>RTCRtpScriptTransformer</dfn> interface # {#RTCRtpScriptTransformer-interface}
945972
<pre class="idl">
@@ -959,7 +986,7 @@ interface RTCRtpScriptTransformer : EventTarget {
959986

960987
## Internal slots ## {#RTCRtpScriptTransformer-internal-slots}
961988

962-
An {{RTCRtpScriptTransformer}} object has the following internal slots.
989+
An {{RTCRtpScriptTransformer}} object has the following internal slots:
963990

964991
<table class="data" dfn-for="RTCRtpScriptTransformer" dfn-type="attribute">
965992
<thead>

0 commit comments

Comments
 (0)