Add knowledge_distillation_tutorial #2514

AlexandrosChrtn · 2023-07-31T16:14:49Z

Includes a Knowledge distillation tutorial.
From this tutorial one can learn:

How to modify model classes to extract hidden representations and use them for further calculations
How to modify regular train loops in pytorch to include additional losses on top of, for example, cross-entropy for classification
How to improve the performance of lightweight models, using more complex models as teachers

pytorch-bot · 2023-07-31T16:14:51Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/2514

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit cafd91e with merge base e7c86fd ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-07-31T16:14:53Z

Hi @AlexandrosChrtn!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

netlify · 2023-07-31T16:16:09Z

❌ Deploy Preview for pytorch-tutorials-preview failed.

Name	Link
🔨 Latest commit	`79b7467`
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-tutorials-preview/deploys/64c7ddfb7a63f7000843ae3c

AlexandrosChrtn · 2023-07-31T20:39:00Z

@svekars Hello! I get this when deploying, looks related to recent changes in requirements.txt

ERROR: No matching distribution found for scipy==1.11.1 (from -r requirements.txt (line 50))
Error installing pip dependencies
Build was terminated: dependency_installation script returned non-zero exit code: 1
Failing build: Failed to install dependencies

svekars · 2023-08-07T18:20:11Z

@AlexandrosChrtn if you are referring to the Netlify error, it can be safely ignored for now. GitHub Actions checks are the ones we are looking at.

svekars · 2023-08-07T18:22:11Z

Can you please add a better description for this tutorial describing how the PyTorch community benefits from this tutorial?

AlexandrosChrtn · 2023-08-08T18:45:40Z

@svekars Added a short description with bullets. Let me know if you need anything else!

albanD

Sounds good to me!

albanD · 2023-08-14T18:49:22Z

beginner_source/knowledge_distillation_tutorial.py

+])
+
+# Data preprocessing for CIFAR-10
+transform_test = transforms.Compose([


Why not use the same as train one?

albanD · 2023-08-14T18:52:35Z

beginner_source/knowledge_distillation_tutorial.py

+            soft_targets = nn.functional.softmax(teacher_logits / T, dim=-1)
+            soft_prob = nn.functional.log_softmax(student_logits / T, dim=-1)
+
+            # Calculate the soft targets loss. Scaled by T**2 as suggested by the authors of the paper


Which paper?

albanD

Thanks for the update.
That sounds good to me cc @svekars

svekars

Thanks so much, @AlexandrosChrtn. I added some editorial suggestions. A couple of things, in addition to that:

Can you some headings that would better break your tutorial to sections (I made some suggestions in my review)
Please break lines at 79 symbols where possible so that it's easier to edit in the future.

Let me know if you have any questions.

beginner_source/knowledge_distillation_tutorial.py

svekars · 2023-08-14T21:52:27Z

beginner_source/knowledge_distillation_tutorial.py

+import torchvision.transforms as transforms
+import torchvision.datasets as datasets
+
+# Ignore warnings


can you add an explanation to the comment why we ignore them

svekars · 2023-08-15T15:20:36Z

beginner_source/knowledge_distillation_tutorial.py

+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+######################################################################
+# CIFAR-10 is a popular image dataset with 10 classes. Our objective is to predict one of the following classes for each input image.


Can you please add a heading 2 here?

beginner_source/knowledge_distillation_tutorial.py

svekars · 2023-08-15T18:28:09Z

Also, make sure to check the html output here and fix the spellcheck by enclosing CosineLoss in double backticks (``).

Co-authored-by: Svetlana Karslioglu <[email protected]>

beginner_source/knowledge_distillation_tutorial.py

Co-authored-by: Svetlana Karslioglu <[email protected]>

AlexandrosChrtn · 2023-08-18T11:53:45Z

Thanks everyone for the insightful suggestions! @svekars I think I fixed everything based on your proposals, except for the conclusion part, where it feels repetitive to state everything the user learned, becuase we do so in the beginning, however I can still add a few lines if you think it's best. I added sections and removed the lines about filtering warnings. The only reason they were there at the first place is because I saw them in other tutorials (I speculate people add them because of deprecation warnings due to version mismatches and what not).

svekars

Final cleanup

beginner_source/knowledge_distillation_tutorial.py

Editorial clean up

* Add knowledge_distillation_tutorial --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

Add knowledge_distillation_tutorial

79b7467

facebook-github-bot added the cla signed label Jul 31, 2023

Merge branch 'main' into knowledge-distillation-tutorial

7606376

AlexandrosChrtn and others added 2 commits August 8, 2023 16:47

Include description above and fix pyspelling

137b313

Merge branch 'main' into knowledge-distillation-tutorial

5618628

svekars requested review from malfet and albanD August 14, 2023 15:08

Merge branch 'main' into knowledge-distillation-tutorial

e260d4c

albanD approved these changes Aug 14, 2023

View reviewed changes

AlexandrosChrtn and others added 2 commits August 15, 2023 13:24

Merge branch 'main' into knowledge-distillation-tutorial

4f6eec8

review fixes + pyspelling

a569ee9

albanD reviewed Aug 15, 2023

View reviewed changes

svekars reviewed Aug 15, 2023

View reviewed changes

Merge branch 'main' into knowledge-distillation-tutorial

0632312

AlexandrosChrtn and others added 8 commits August 16, 2023 15:32

Update beginner_source/knowledge_distillation_tutorial.py

a5d8221

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

b7185df

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

86fa7a0

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

4f3570a

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

5881348

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

fbbb40d

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

9078639

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

f66b0ee

Co-authored-by: Svetlana Karslioglu <[email protected]>

AlexandrosChrtn and others added 11 commits August 16, 2023 23:02

Update beginner_source/knowledge_distillation_tutorial.py

035c541

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

fd8b9b4

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

788aedf

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

de2d4ef

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

56b7002

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

f44a32d

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

1ced800

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

1113cf1

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

d166fff

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

196637a

Co-authored-by: Svetlana Karslioglu <[email protected]>

Update beginner_source/knowledge_distillation_tutorial.py

1d32f4f

Co-authored-by: Svetlana Karslioglu <[email protected]>

svekars reviewed Aug 16, 2023

View reviewed changes

beginner_source/knowledge_distillation_tutorial.py Outdated Show resolved Hide resolved

AlexandrosChrtn and others added 5 commits August 17, 2023 14:06

Apply suggestions from code review

bccd00a

Co-authored-by: Svetlana Karslioglu <[email protected]>

review fixes

abcdd60

review fixes

bf79878

review fixes

beec23c

final review fixes

c79d298

svekars approved these changes Aug 22, 2023

View reviewed changes

Apply suggestions from code review

cafd91e

Editorial clean up

svekars merged commit 827843f into pytorch:main Aug 22, 2023
18 checks passed

mikaylagawarecki pushed a commit to mikaylagawarecki/tutorials that referenced this pull request Sep 18, 2023

Add knowledge_distillation_tutorial (pytorch#2514)

9bf92c9

* Add knowledge_distillation_tutorial --------- Co-authored-by: Svetlana Karslioglu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add knowledge_distillation_tutorial #2514

Add knowledge_distillation_tutorial #2514

AlexandrosChrtn commented Jul 31, 2023 •

edited

Loading

pytorch-bot bot commented Jul 31, 2023 •

edited

Loading

facebook-github-bot commented Jul 31, 2023

netlify bot commented Jul 31, 2023

AlexandrosChrtn commented Jul 31, 2023

svekars commented Aug 7, 2023

svekars commented Aug 7, 2023

AlexandrosChrtn commented Aug 8, 2023

albanD left a comment

albanD Aug 14, 2023

albanD Aug 14, 2023

albanD left a comment

svekars left a comment

svekars Aug 14, 2023

svekars Aug 15, 2023

svekars commented Aug 15, 2023 •

edited

Loading

AlexandrosChrtn commented Aug 18, 2023

svekars left a comment

Add knowledge_distillation_tutorial #2514

Add knowledge_distillation_tutorial #2514

Conversation

AlexandrosChrtn commented Jul 31, 2023 • edited Loading

pytorch-bot bot commented Jul 31, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/2514

✅ No Failures

facebook-github-bot commented Jul 31, 2023

Action Required

Process

netlify bot commented Jul 31, 2023

❌ Deploy Preview for pytorch-tutorials-preview failed.

AlexandrosChrtn commented Jul 31, 2023

svekars commented Aug 7, 2023

svekars commented Aug 7, 2023

AlexandrosChrtn commented Aug 8, 2023

albanD left a comment

Choose a reason for hiding this comment

albanD Aug 14, 2023

Choose a reason for hiding this comment

albanD Aug 14, 2023

Choose a reason for hiding this comment

albanD left a comment

Choose a reason for hiding this comment

svekars left a comment

Choose a reason for hiding this comment

svekars Aug 14, 2023

Choose a reason for hiding this comment

svekars Aug 15, 2023

Choose a reason for hiding this comment

svekars commented Aug 15, 2023 • edited Loading

AlexandrosChrtn commented Aug 18, 2023

svekars left a comment

Choose a reason for hiding this comment

AlexandrosChrtn commented Jul 31, 2023 •

edited

Loading

pytorch-bot bot commented Jul 31, 2023 •

edited

Loading

svekars commented Aug 15, 2023 •

edited

Loading