Embed Notebooks in the Docs (mosaicml#1116)

* Configured `nbsphinx` to embed sphinx notebooks in the docs site
* Symlink the `examples/` folder in `docs/source/examples`, so notebooks can be included via docs
* Rename the `notebooks` folder to `examples`, to match the structure on the docs site. (Notebooks really are examples, anyways)
* Refactored some notebooks to clean up the headings -- there should only be one h1 (the title); otherwise, it will show up in the sidebar on the left
* Replaced `scripts/clean_notebooks.py` with the `nbstripout` pre-commit hook
* Removed the embedded image from the "custom speedup methods" notebook; instead, uploaded it to the google storage bucket, and linked to it from there, as changing this file was causing the "large file" pre-commit hook to fail

.ci/Jenkinsfile

@@ -191,7 +191,7 @@ void scheduleJob(jobs, String image, buildArgs) {
     // jobs: The list of jobs. Modified in-place.
     // buildArgs: The build args matrix
 
-    String markers = 'not notebooks and not daily and not remote'
+    String markers = 'not daily and not remote'
     Boolean isLintImage = false
     Boolean isVisionImage = false
     Boolean isGpu = false

.pre-commit-config.yaml

@@ -89,4 +89,10 @@ repos:
           - --comment-style
           - "#"
         types: [python]
+  - repo: https://github.com/kynan/nbstripout
+    rev: 0.5.0
+    hooks:
+      - id: nbstripout
+        types:
+          - "jupyter"
 exclude: .ci\/release_tests\/.*

Makefile

@@ -25,6 +25,6 @@ test-dist-gpu:
 	$(PYTHON) -m composer.cli.launcher -n $(WORLD_SIZE) --master_port $(MASTER_PORT) $(EXTRA_LAUNCHER_ARGS) -m $(PYTEST) -m gpu $(EXTRA_ARGS)
 
 clean-notebooks:
-	$(PYTHON) scripts/clean_notebooks.py -i notebooks/*.ipynb
+	$(PYTHON) scripts/clean_notebooks.py -i examples/*.ipynb
 
 .PHONY: test test-gpu test-dist test-dist-gpu clean-notebooks

README.md

@@ -97,7 +97,7 @@ You can use Composer's speedup methods in two ways:
 * Through a standalone **Functional API** (similar to `torch.nn.functional`) that allows you to integrate them into your existing training code.
 * Using Composer's built-in **Trainer**, which is designed to be performant and automatically takes care of many of the low-level details of using speedup methods.
 
-### Example: Functional API [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/Composer_Functional.ipynb)
+### Example: Functional API [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/Composer_Functional.ipynb)
 
 Integrate our speed-up methods into your training loop with just a few lines of code, and see the results. Here we easily apply [BlurPool](https://docs.mosaicml.com/en/stable/method_cards/blurpool.html) and SqueezeExcite:
 
@@ -116,9 +116,9 @@ my_model = cf.apply_squeeze_excite(my_model)
 ```
 <!-- end_example_1 --->
 
-For more examples, see the [Composer Functional API Colab notebook](https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/Composer_Functional.ipynb) and [Functional API guide](https://docs.mosaicml.com/en/latest/functional_api.html).
+For more examples, see the [Composer Functional API Colab notebook](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/Composer_Functional.ipynb) and [Functional API guide](https://docs.mosaicml.com/en/latest/functional_api.html).
 
-### Example: Trainer [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/up_and_running_with_composer.ipynb)
+### Example: Trainer [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/up_and_running_with_composer.ipynb)
 
 For the best experience and the most efficient possible training, we recommend using Composer's built-in trainer, which automatically takes care of the low-level details of using speedup methods and provides useful abstractions that facilitate rapid experimentation.
 
@@ -307,7 +307,7 @@ Through this, our methods can modify:
  - optimizer (Sharpness Aware Minimization)
  - training dynamics (layer freezing, selective backprop, etc.)
 
-Easily [add your own methods](https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/custom_method_tutorial.ipynb) or callbacks to instrument any part of the training loop.
+Easily [add your own methods](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/custom_method_tutorial.ipynb) or callbacks to instrument any part of the training loop.
 
 # 🧐 Why shouldn’t I use Composer?
 
@@ -327,19 +327,19 @@ Here's some resources actively maintained by the Composer community to help you
 </thead>
 <tbody>
   <tr>
-    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/up_and_running_with_composer.ipynb" target="_blank" rel="noopener noreferrer">Getting started with our Trainer</a></td>
+    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/up_and_running_with_composer.ipynb" target="_blank" rel="noopener noreferrer">Getting started with our Trainer</a></td>
     <td>An interactive Colab Notebook aimed at teaching users about our Trainer</td>
   </tr>
   <tr>
-    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/Composer_Functional.ipynb" target="_blank" rel="noopener noreferrer">Getting started with our Functional API</a></td>
+    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/Composer_Functional.ipynb" target="_blank" rel="noopener noreferrer">Getting started with our Functional API</a></td>
     <td>An interactive Colab Notebook aimed at teaching users about our Functional API</td>
   </tr>
   <tr>
-    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/custom_method_tutorial.ipynb" target="_blank" rel="noopener noreferrer">Building Speedup Methods</a></td>
+    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/custom_method_tutorial.ipynb" target="_blank" rel="noopener noreferrer">Building Speedup Methods</a></td>
     <td>An interactive Colab Notebook aimed at teaching users about building speedup methods on top of Composer</td>
   </tr>
   <tr>
-    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/notebooks/nlp_notebook_tutorial.ipynb" target="_blank" rel="noopener noreferrer">Training BERTs with Composer</a></td>
+    <td><a href="https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/nlp_notebook_tutorial.ipynb" target="_blank" rel="noopener noreferrer">Training BERTs with Composer</a></td>
     <td>An interactive Colab Notebook aimed at helping users learn how to train BERT models with Composer!</td>
   </tr>
   <tr>

docs/source/conf.py

@@ -18,7 +18,9 @@
 import inspect
 import json
 import os
+import shutil
 import sys
+import tempfile
 import textwrap
 import types
 import warnings
@@ -33,9 +35,26 @@
 from docutils import nodes
 from docutils.nodes import Element
 from git.repo.base import Repo
+from pypandoc.pandoc_download import download_pandoc
 from sphinx.ext.autodoc import ClassDocumenter, _
 from sphinx.writers.html5 import HTML5Translator
 
+if not shutil.which("pandoc"):
+    # Install pandoc if it is not installed.
+    # Pandoc is required by nbconvert but it is not included in the pypandoc pip package
+    with tempfile.TemporaryDirectory() as tmpdir:
+        # if root on linux, use the "/bin" folder, since "~/bin" = "/root/bin" is not in the path by default
+        # similar on osx -- use /Applications instead of "~/Applications" = "/root/Applications"
+        target_folder = None
+        if os.getuid() == 0:
+            if sys.platform == "linux":
+                target_folder = "/bin"
+            elif sys.platform == "darwin":
+                target_folder = "/Applications/pandoc"
+            # Not handling windows; nobody uses root on windows lol
+
+        download_pandoc(version='2.18', download_folder=tmpdir, targetfolder=target_folder, delete_installer=True)
+
 sys.path.insert(0, os.path.abspath('..'))
 
 log = sphinx.util.logging.getLogger(__name__)
@@ -68,6 +87,7 @@
     'sphinx.ext.doctest',
     'sphinx_panels',
     'sphinxcontrib.images',
+    'nbsphinx',
 ]
 
 
@@ -93,6 +113,23 @@ def _get_commit_sha() -> str:
 
 _COMMIT_SHA = _get_commit_sha()
 
+# Don't show notebook output in the docs
+nbsphinx_execute = 'never'
+
+notebook_path = "mosaicml/composer/blob/" + _COMMIT_SHA + "/{{ env.doc2path(env.docname, base=None) }}"
+
+# Include an "Open in Colab" link at the beginning of all notebooks
+nbsphinx_prolog = f"""
+
+.. tip::
+
+    This tutorial is available as a `Jupyter notebook <https://github.com/{notebook_path}>`_.
+
+    ..  image:: https://colab.research.google.com/assets/colab-badge.svg
+        :target: https://colab.research.google.com/github/{notebook_path}
+        :alt: Open in Colab
+"""
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 source_suffix = ['.rst', '.md']

docs/source/examples

@@ -0,0 +1 @@
+../../examples

docs/source/getting_started/installation.rst

@@ -155,6 +155,6 @@ Besides easily running our built-in algorithms, Composer also features:
 Next steps
 ----------
 
-* Try :doc:`/getting_started/notebooks` to see some of our speed-ups with notebooks on Colab.
+* Try our :doc:`/examples/getting_started` tutorial on Colab.
 * See :doc:`/trainer/using_the_trainer` for more details on our trainer.
 * Read :doc:`/getting_started/welcome_tour` for a tour through the library.

docs/source/index.rst

@@ -38,16 +38,30 @@ If you have any questions, please feel free to reach out to us on `Twitter`_, `E
 
 Composer is part of the broader Machine Learning community, and we welcome any contributions, pull requests, and issues.
 
-
 .. toctree::
    :hidden:
    :maxdepth: 1
    :caption: Getting Started
 
    getting_started/installation.rst
-   getting_started/notebooks.rst
    getting_started/welcome_tour.rst
 
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+   :caption: Examples
+
+
+   examples/getting_started.ipynb
+   examples/functional_api.ipynb
+   examples/medical_image_segmentation.ipynb
+   examples/custom_speedup_methods.ipynb
+   examples/nlp_models.ipynb
+   examples/ffcv_dataloaders.ipynb
+   examples/huggingface_models.ipynb
+   examples/migrate_from_ptl.ipynb
+
+
 .. toctree::
    :hidden:
    :maxdepth: 1

examples/.gitignore

@@ -1 +1,3 @@
 datasets/
+data/
+wandb/