[ layer ] Mixed precision forwarding / backwarding for bn layer @open sesame 03/07 10:42 #2462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
skykongkong8
requested review from
myungjoo,
jijoongmoon,
again4you,
jaeyun-jung,
leemgs,
wooksong,
helloahn,
kparichay,
gichan-jang,
anyj0527,
zhoonit,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
djeong20,
EunjuYang and
a team
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2462. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/. |
skykongkong8
force-pushed
the
bn_layer_fp16
branch
from
412da0f to
11006c0
Compare
taos-ci
approved these changes
Feb 6, 2024
There was a problem hiding this comment.
@skykongkong8, 💯 All CI checkers are successfully verified. Thanks.
djeong20
reviewed
Feb 6, 2024
nntrainer/layers/bn_layer.cpp
Outdated
Comment on lines
179
to
182
| TensorDim mu_dim = mu.getDim(); | ||
| mu_dim.setDataType(ml::train::TensorDim::DataType::FP32); | ||
| Tensor mu32(mu_dim, true); | ||
| mu32.copyData(mu); |
There was a problem hiding this comment.
could be reduced?
Tensor getFloatTensor(Tensor input) {
Tensor output({input.getFormat(), ml::train::TensorDim::DataType::FP32}, true);
ouput.copyData(input);
return output;
}
Tensor mu32 = getFloatTensor(mu);
Tensor var32 = getFloatTensor(var);
Tensor gamma = getFloatTensor(gamma);
...
nntrainer/layers/bn_layer.cpp
Outdated
| @@ -213,42 +306,151 @@ void BatchNormalizationLayer::calcDerivative(RunLayerContext &context) { | |||
|
|
|||
| Tensor &t_reduced = context.getTensor(wt_idx[BNParams::t_reduced]); | |||
| Tensor &t_full = context.getTensor(wt_idx[BNParams::t_full]); | |||
| if (deriv.getDataType() == ml::train::TensorDim::DataType::FP16) { | |||
| #ifdef ENABLE_FP16 | |||
| TensorDim gamma_dim = gamma.getDim(); | |||
- According to recent papers in mixed precision, when it comes to computation of statistics in bn layer, we should use fp32 values - input(fp16) <-> BN_layer(fp16 weight, but compute with fp32 and re-cast to fp16) <-> output(fp16) - Although we need a bulky code block for this, I believe we can revisit here for cleaner code when TensorV2 becomes official **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
bn_layer_fp16
branch
from
11006c0 to
534d25c
Compare
skykongkong8
changed the title
skykongkong8
force-pushed
the
bn_layer_fp16
branch
from
534d25c to
bd50fbe
Compare
- For mixed precision training computation of BN layer, we have been declaring temporal single-precision Tensors, compute it with Tensor ops, and save the result in previous half-precision. - To remove redundant code, declare temporal function and reuse. - We should revisit here fore even clearer code when TensorV2 refactorization is finished **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
bn_layer_fp16
branch
from
bd50fbe to
f48e585
Compare
taos-ci
approved these changes
Feb 26, 2024
There was a problem hiding this comment.
@skykongkong8, 💯 All CI checkers are successfully verified. Thanks.
DonghakPark
changed the title
EunjuYang
reviewed
Mar 7, 2024
| /** | ||
| * This calculates dgamma tensor. | ||
| */ | ||
| Tensor dgamma = context.getWeightGrad(wt_idx[BNParams::gamma]); |
There was a problem hiding this comment.
is it Tensor &dgamma = context.getWeightGrad(wt_idx[BNParams::gamma]); ?
taos-ci
approved these changes
Mar 7, 2024
There was a problem hiding this comment.
@skykongkong8, 💯 All CI checkers are successfully verified. Thanks.
jijoongmoon
requested changes
Apr 18, 2024
There was a problem hiding this comment.
I dont think this is the right way to enable the mixed precision. please consider #2455
|
This PR is no longer needed. Close. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Self evaluation: