Add github-action-markdown-link-check

.github/workflows/check_markdown_links.yaml

@@ -0,0 +1,14 @@
+name: Check Markdown links
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  markdown-link-check:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@master
+    - uses: gaurav-nelson/github-action-markdown-link-check@v1
+      with:
+        use-quiet-mode: 'yes'

README.md

@@ -21,7 +21,7 @@ This repository contains a variety of Determined examples that are not actively
 | [cifar10\_pytorch\_inference](computer_vision/cifar10_pytorch_inference)     | CIFAR-10                     | PyTorch                                  |
 | [cifar10\_tf\_keras](computer_vision/cifar10_tf_keras)                       | CIFAR-10                     | TensorFlow (tf.keras)                    |
 | [fasterrcnn\_coco\_pytorch](computer_vision/fasterrcnn_coco_pytorch)         | Penn-Fudan Dataset           | PyTorch                                  |
-| [mmdetection\_pytorch](computer_vision/mmdetection_pytorch)                  | COCO                         | PyTorch                                  |
+| [mmdetection](model_hub/mmdetection)                  | COCO                         | PyTorch                                  |
 | [detr\_coco\_pytorch](computer_vision/detr_coco_pytorch)                     | COCO                         | PyTorch                                  |
 | [deformabledetr\_coco\_pytorch](computer_vision/deformabledetr_coco_pytorch) | COCO                         | PyTorch                                  |
 | [iris\_tf\_keras](computer_vision/iris_tf_keras)                             | Iris Dataset                 | TensorFlow (tf.keras)                    |

blog/python_sdk_demo/README.md

@@ -42,4 +42,4 @@ python determined_sdk_demo.py
 - [Wesley Turner](https://github.com/wes-turner)
 - [Kevin Musgrave](https://github.com/KevinMusgrave)
 
-The code in the `medmnist_model` directory is based on the [`determined_medmnist_e2e`](https://github.com/ighodgao/determined_medmnist_e2e) repo by [Isha Ghodgaonkar](ighodgao).
+The code in the `medmnist_model` directory is based on the [`determined_medmnist_e2e`](https://github.com/ighodgao/determined_medmnist_e2e) repo by [Isha Ghodgaonkar](https://github.com/ighodgao).

computer_vision/byol_pytorch/README.md

@@ -3,7 +3,7 @@
 This example shows how to perform self-supervised image classifier training with BYOL using
 Determined's PyTorch API.  This example is based on the [byol-pytorch](https://github.com/lucidrains/byol-pytorch/tree/master/byol_pytorch) package.
 
-Original BYOL paper: https://arxiv.org/abs/2006.0
+Original BYOL paper: https://arxiv.org/abs/2006.07733
 
 Code and configuration details also sourced from the following BYOL implementations:
   - (JAX, paper authors) https://github.com/deepmind/deepmind-research/tree/master/byol

computer_vision/cifar10_pytorch/README.md

@@ -2,7 +2,7 @@
 
 This example shows how to build a simple CNN on the CIFAR-10 dataset using
 Determined's PyTorch API. This example is adapted from this [Keras CNN
-example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
+example](https://github.com/keras-team/keras/blob/keras-2/examples/cifar10_cnn.py).
 
 ## Files
 * **model_def.py**: The core code for the model. This includes building and compiling the model.

computer_vision/cifar10_pytorch/model_def.py

@@ -1,6 +1,6 @@
 """
 CNN on Cifar10 from Keras example:
-https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py
+https://github.com/keras-team/keras/blob/keras-2/examples/cifar10_cnn.py
 """
 import os
 import tempfile

computer_vision/cifar10_tf_keras/README.md

@@ -2,7 +2,7 @@
 
 This example shows how to build a simple CNN on the CIFAR-10 dataset using
 Determined's tf.keras API. This example is adapted from this [Keras CNN
- example](https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py).
+ example](https://github.com/keras-team/keras/blob/keras-2/examples/cifar10_cnn.py).
 
 ## Files
 * **model_def.py**: Organizes the model and data-loaders into the Determined TFKerasTrial API.

computer_vision/cifar10_tf_keras/cifar_model.py

@@ -1,6 +1,6 @@
 """
 Original CIFAR-10 CNN Keras model code from:
-https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py.
+https://github.com/keras-team/keras/blob/keras-2/examples/cifar10_cnn.py.
 """
 
 import numpy as np

computer_vision/cifar10_tf_keras/model_def.py

@@ -10,7 +10,7 @@
     https://docs.determined.ai/latest/reference/api/keras.html
     https://www.tensorflow.org/guide/keras
 
-Based on: https://github.com/fchollet/keras/blob/master/examples/cifar10_cnn.py
+Based on: https://github.com/keras-team/keras/blob/keras-2/examples/cifar10_cnn.py
 
 """
 from typing import Generator, List, Tuple

deepspeed/cifar10_cpu_offloading/README.md

@@ -1,6 +1,6 @@
 # DeepSpeed CPU Offloading example
 This example is adapted from the 
-[CIFAR example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/cifar) 
+[CIFAR example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/training/cifar) 
 repository. It is intended to show how to configure 
 [ZeRO Stage 3 with CPU offloading](https://www.deepspeed.ai/tutorials/zero/) for a simple CNN network.
 

deepspeed/cifar10_moe/README.md

@@ -1,6 +1,6 @@
 # DeepSpeed CIFAR Example
 This example is adapted from the 
-[CIFAR example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/cifar) 
+[CIFAR example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/training/cifar) 
 repository. It is intended to demonstrate a simple usecase of DeepSpeed with Determined.
 
 ## Files

deepspeed/deepspeed_dcgan/README.md

@@ -1,6 +1,6 @@
 # DeepSpeed CIFAR Example
 This example is adapted from the
-[DCGAN example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/gan)
+[DCGAN example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/training/gan)
 repository. It is intended to demonstrate a simple usecase of DeepSpeed with Determined.
 
 ## Files

deepspeed/pipeline_parallelism/README.md

@@ -1,6 +1,6 @@
 # DeepSpeed CIFAR Example
 This example is adapted from the 
-[pipeline parallelism example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/pipeline_parallelism) 
+[pipeline parallelism example in the DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples/tree/master/training/pipeline_parallelism) 
 repository. It is intended to demonstrate a simple usecase of DeepSpeed's PipelineEngine with Determined.
 
 ## Files

gan/cyclegan/README.md

@@ -6,7 +6,7 @@ Determined implementation of [CycleGAN](https://github.com/eriklindernoren/PyTor
 
 - **determined_model_def.py** is the user model definition file for Determined-managed training. 
   This file is ported from **cyclegan.py** to implement the 
-  [Determined Pytorch API](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial).
+  [Determined Pytorch API](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial).
 - **const.yaml** is the user configuration for Determined-managed training.
 - **startup-hook.sh** is the startup bash script that is used for downloading and 
   extracting the training data.
@@ -56,5 +56,5 @@ hyperparameters:
 * The throughput is unstable due to inter-node communication when the global batch size 
   is 64 and the aggregation frequency is 1. Use a larger batch size or a larger aggregation 
   frequency to increase the scaling efficiency of the throughput. See 
-  [Effective Distributed Training](https://docs.determined.ai/latest/topic-guides/effective-distributed-training.html#effective-distributed-training)
+  [Distributed Training Performance Optimization](https://docs.determined.ai/latest/model-dev-guide/dtrain/dtrain-introduction.html#performance-optimization)
   for details.

gan/cyclegan/cyclegan.py

@@ -1,6 +1,6 @@
 """
 This file is not used in the Determined-managed training. We port this file to use the
-Determined Pytorch API (https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial).
+Determined Pytorch API (https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial).
 """
 import argparse
 import os

gan/gan_mnist_pytorch/README.md

@@ -2,7 +2,7 @@
 
 This example demonstrates how to build a simple GAN on the MNIST dataset using
 Determined's PyTorch API. This example is adapted from this [PyTorch Lightning GAN
-example](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/generative_adversarial_net.py).
+example](https://github.com/Lightning-AI/pytorch-lightning/blob/master/examples/pytorch/domain_templates/generative_adversarial_net.py).
 
 ## Files
 * **model_def.py**: The core code for the model. This includes building and compiling the model.

gan/pix2pix_tf_keras/README.md

@@ -40,6 +40,6 @@ The following image was created using a model trained with `const.yaml`.
 
 ## Results
 The following plots show differences in performance made when utilizing GPUs with the Determined distributed system.
-![Cumulative Batches vs. Time](./images/batches_vs_time.png)
-![Training Loss vs. Time](./images/training_loss_vs_time.png)
-![Validation Loss vs. Time](./images/validation_loss_vs_time.png)
+![Cumulative Batches vs. Time](./images/batches_vs_time.jpg)
+![Training Loss vs. Time](./images/training_loss_vs_time.jpg)
+![Validation Loss vs. Time](./images/validation_loss_vs_time.jpg)

model_hub/huggingface/language-modeling/README.md

@@ -4,9 +4,9 @@ The examples here mirror the [language-modeling examples](https://github.com/hug
 You can finetune GPT and GPT-2 with the causal language model (CLM); ALBERT, BERT, DistilBERT, and RoBERTa with the masked language model(MLM); and XLNet with the permutation language model (PLM).
 
 ## Files
-* **clm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for CLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
-* **mlm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for MLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
-* **plm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for PLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **clm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for CLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
+* **mlm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for MLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
+* **plm_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for PLM. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 
 ### Configuration Files
 * **clm_config.yaml**: Experiment configuration for finetuning on WikiText-2 with GPT2.  

model_hub/huggingface/multiple-choice/README.md

@@ -2,7 +2,7 @@
 This example mirrors the [multiple choice example](https://github.com/huggingface/transformers/tree/master/examples/pytorch/multiple-choice) from the original huggingface transformers repo for finetuning on the SWAG dataset.
 
 ## Files
-* **swag_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for named-entity recognition but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **swag_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for named-entity recognition but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 * **data.py**: Modifies ``DataCollatorForMultipleChoice`` to work with PyTorchTrial.
 
 ### Configuration FIles

model_hub/huggingface/question-answering/README.md

@@ -2,9 +2,9 @@
 The examples here mirror the [question answering examples](https://github.com/huggingface/transformers/tree/master/examples/pytorch/question-answering) from the original huggingface transformers repo.
 
 ## Files
-* **qa_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for question-answering on SQuAD. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **qa_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for question-answering on SQuAD. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 * **data.py**: data pre and post-processing for question answering.
-* **qa_beam_search_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for question-answering with beam search. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **qa_beam_search_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for question-answering with beam search. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 * **data_beam_search.py**: data pre and post-processing for question answering with beam search.
 
 ### Configuration Files

model_hub/huggingface/text-classification/README.md

@@ -4,7 +4,7 @@ The two examples here mirror the [text-classification examples](https://github.c
 ## GLUE Benchmark
 
 ### Files
-* **glue_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **glue_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for text classification but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 
 #### Configuration Files
 * **glue_config.yaml**: Experiment configuration for finetuning on GLUE datasets with Bert.  
@@ -32,7 +32,7 @@ Using the provided experiment config `glue_config.yaml` yields a Matthew's Corre
 ## XNLI Dataset
 
 ### Files
-* **xnli_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for text classification on XNLI but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **xnli_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for text classification on XNLI but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 
 #### Configuration Files
 * **xnli_config.yaml**: Experiment configuration for finetuning on the XNLI dataset with Bert.

model_hub/huggingface/token-classification/README.md

@@ -2,7 +2,7 @@
 This example mirrors the [token-classification example](https://github.com/huggingface/transformers/tree/master/examples/pytorch/token-classification) from the original huggingface transformers repo for named-entity recognition.
 
 ## Files
-* **ner_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/reference/api/pytorch.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for named-entity recognition but otherwise the behavior is the same as the [BaseTransformerTrial class](../model_hub/transformers/_trial.py).
+* **ner_trial.py**: The [PyTorchTrial definition](https://docs.determined.ai/latest/model-dev-guide/api-guides/apis-howto/api-pytorch-ug.html#pytorch-trial) for this example. A few class methods are overwritten and specialized for named-entity recognition but otherwise the behavior is the same as the [BaseTransformerTrial class](https://github.com/determined-ai/determined/blob/main/model_hub/model_hub/huggingface/_trial.py).
 * **ner_utils.py**: Utility functions for NER largely extracted from [run_ner.py](https://github.com/huggingface/transformers/tree/master/examples/pytorch/token-classification/run_ner.py) to separate example code from Determined code.
 
 ### Configuration Files