Skip to content

Commit

Permalink
Update doc section on custom batch providers
Browse files Browse the repository at this point in the history
  • Loading branch information
funkey committed Mar 17, 2020
1 parent 8de7280 commit b8ea4f4
Show file tree
Hide file tree
Showing 4 changed files with 63 additions and 36 deletions.
32 changes: 20 additions & 12 deletions docs/_sources/custom_providers.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,28 @@ override :meth:`prepare<gunpowder.BatchFilter.prepare>` and/or
class ExampleFilter(BatchFilter):
def prepare(self, request):
pass
# create a new request for this node's dependencies
dependencies = BatchRequest()
# [...]
return dependencies
def process(self, batch, request):
pass
# create a new batch for this node's output
output = Batch()
# [...]
return output
``prepare`` and ``process`` will be called in an alternating fashion.
``prepare`` is called first, when a ``BatchRequest`` is passed upstream through
the filter. Your filter is free to change the request in any way it needs to,
for example, by increasing the requested sizes. After ``prepare``, ``process``
will be called with a batch going downstream, which is the upstream's response
to the request you modified in ``prepare``. In ``process``, your filter should
make all necessary changes to the batch and ensure it meets the original
downstream request earlier communicated to ``prepare`` (given as ``request``
parameter in ``process`` for convenience).

For an example of a batch filter changing both the spec going upstream and the
batch going downstream, see :class:`ElasticAugment<gunpowder.ElasticAugment>`.
the filter. Your filter has to formulate a new request, stating the
dependencies needed by this filter, e.g., by increasing the requested sizes of
existing arrays in the request.
After ``prepare``, ``process`` will be called with a batch going downstream,
which is the upstream's response to the your request you returned in
``prepare``. In ``process``, your filter has to create a new batch with
expected outputs and ensure it meets the original downstream request earlier
communicated to ``prepare`` (given as ``request`` parameter in ``process`` for
convenience).

For a simple example of a batch filter following this scheme, see
the source code of :class:`DownSample<gunpowder.DownSample>`.
32 changes: 20 additions & 12 deletions docs/build/custom_providers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,28 @@ override :meth:`prepare<gunpowder.BatchFilter.prepare>` and/or
class ExampleFilter(BatchFilter):
def prepare(self, request):
pass
# create a new request for this node's dependencies
dependencies = BatchRequest()
# [...]
return dependencies
def process(self, batch, request):
pass
# create a new batch for this node's output
output = Batch()
# [...]
return output
``prepare`` and ``process`` will be called in an alternating fashion.
``prepare`` is called first, when a ``BatchRequest`` is passed upstream through
the filter. Your filter is free to change the request in any way it needs to,
for example, by increasing the requested sizes. After ``prepare``, ``process``
will be called with a batch going downstream, which is the upstream's response
to the request you modified in ``prepare``. In ``process``, your filter should
make all necessary changes to the batch and ensure it meets the original
downstream request earlier communicated to ``prepare`` (given as ``request``
parameter in ``process`` for convenience).

For an example of a batch filter changing both the spec going upstream and the
batch going downstream, see :class:`ElasticAugment<gunpowder.ElasticAugment>`.
the filter. Your filter has to formulate a new request, stating the
dependencies needed by this filter, e.g., by increasing the requested sizes of
existing arrays in the request.
After ``prepare``, ``process`` will be called with a batch going downstream,
which is the upstream's response to the your request you returned in
``prepare``. In ``process``, your filter has to create a new batch with
expected outputs and ensure it meets the original downstream request earlier
communicated to ``prepare`` (given as ``request`` parameter in ``process`` for
convenience).

For a simple example of a batch filter following this scheme, see
the source code of :class:`DownSample<gunpowder.DownSample>`.
33 changes: 22 additions & 11 deletions docs/custom_providers.html
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@
<li class="toctree-l3"><a class="reference internal" href="api.html#array">Array</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#graph">Graph</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#node">Node</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#edge">Edge</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#arraykey">ArrayKey</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#graphkey">GraphKey</a></li>
</ul>
Expand Down Expand Up @@ -151,6 +152,7 @@
</li>
<li class="toctree-l2"><a class="reference internal" href="api.html#image-processing-nodes">Image Processing Nodes</a><ul>
<li class="toctree-l3"><a class="reference internal" href="api.html#downsample">DownSample</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#upsample">UpSample</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#intensityscaleshift">IntensityScaleShift</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#normalize">Normalize</a></li>
</ul>
Expand All @@ -173,6 +175,7 @@
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="api.html#training-and-prediction-nodes">Training and Prediction Nodes</a><ul>
<li class="toctree-l3"><a class="reference internal" href="api.html#stack">Stack</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#torch-train">torch.Train</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#torch-predict">torch.Predict</a></li>
<li class="toctree-l3"><a class="reference internal" href="api.html#tensorflow-train">tensorflow.Train</a></li>
Expand Down Expand Up @@ -273,23 +276,31 @@
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ExampleFilter</span><span class="p">(</span><span class="n">BatchFilter</span><span class="p">):</span>

<span class="k">def</span> <span class="nf">prepare</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
<span class="k">pass</span>
<span class="c1"># create a new request for this node&#39;s dependencies</span>
<span class="n">dependencies</span> <span class="o">=</span> <span class="n">BatchRequest</span><span class="p">()</span>
<span class="c1"># [...]</span>
<span class="k">return</span> <span class="n">dependencies</span>

<span class="k">def</span> <span class="nf">process</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">batch</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
<span class="k">pass</span>
<span class="c1"># create a new batch for this node&#39;s output</span>
<span class="n">output</span> <span class="o">=</span> <span class="n">Batch</span><span class="p">()</span>
<span class="c1"># [...]</span>
<span class="k">return</span> <span class="n">output</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">prepare</span></code> and <code class="docutils literal notranslate"><span class="pre">process</span></code> will be called in an alternating fashion.
<code class="docutils literal notranslate"><span class="pre">prepare</span></code> is called first, when a <code class="docutils literal notranslate"><span class="pre">BatchRequest</span></code> is passed upstream through
the filter. Your filter is free to change the request in any way it needs to,
for example, by increasing the requested sizes. After <code class="docutils literal notranslate"><span class="pre">prepare</span></code>, <code class="docutils literal notranslate"><span class="pre">process</span></code>
will be called with a batch going downstream, which is the upstream’s response
to the request you modified in <code class="docutils literal notranslate"><span class="pre">prepare</span></code>. In <code class="docutils literal notranslate"><span class="pre">process</span></code>, your filter should
make all necessary changes to the batch and ensure it meets the original
downstream request earlier communicated to <code class="docutils literal notranslate"><span class="pre">prepare</span></code> (given as <code class="docutils literal notranslate"><span class="pre">request</span></code>
parameter in <code class="docutils literal notranslate"><span class="pre">process</span></code> for convenience).</p>
<p>For an example of a batch filter changing both the spec going upstream and the
batch going downstream, see <a class="reference internal" href="api.html#gunpowder.ElasticAugment" title="gunpowder.ElasticAugment"><code class="xref py py-class docutils literal notranslate"><span class="pre">ElasticAugment</span></code></a>.</p>
the filter. Your filter has to formulate a new request, stating the
dependencies needed by this filter, e.g., by increasing the requested sizes of
existing arrays in the request.
After <code class="docutils literal notranslate"><span class="pre">prepare</span></code>, <code class="docutils literal notranslate"><span class="pre">process</span></code> will be called with a batch going downstream,
which is the upstream’s response to the your request you returned in
<code class="docutils literal notranslate"><span class="pre">prepare</span></code>. In <code class="docutils literal notranslate"><span class="pre">process</span></code>, your filter has to create a new batch with
expected outputs and ensure it meets the original downstream request earlier
communicated to <code class="docutils literal notranslate"><span class="pre">prepare</span></code> (given as <code class="docutils literal notranslate"><span class="pre">request</span></code> parameter in <code class="docutils literal notranslate"><span class="pre">process</span></code> for
convenience).</p>
<p>For a simple example of a batch filter following this scheme, see
the source code of <a class="reference internal" href="api.html#gunpowder.DownSample" title="gunpowder.DownSample"><code class="xref py py-class docutils literal notranslate"><span class="pre">DownSample</span></code></a>.</p>
</div>


Expand Down
Loading

0 comments on commit b8ea4f4

Please sign in to comment.