Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MXNet inference] Fix implementation + extend features #376

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
2dc4c85
MXNet inference for classification models using Gluon API
valentina-kustikova Jan 31, 2023
0bce56e
Fix loading label file and logging predictions
valentina-kustikova Feb 1, 2023
67f1c94
Code cleanings
valentina-kustikova Feb 2, 2023
f55f91a
Fix formatting
valentina-kustikova Feb 2, 2023
3587f4d
Inference implementation using import Gluon (model should be saved in…
valentina-kustikova Feb 5, 2023
5451033
Validation table for MXNet (Gluon Model zoo)
valentina-kustikova Feb 5, 2023
a902651
Update list of supported MXNet models + readme for inference script
valentina-kustikova Feb 6, 2023
532f3c7
Fix line break
valentina-kustikova Feb 6, 2023
ef9da57
Fix arguments + update inference readme
valentina-kustikova Feb 7, 2023
1bdff6f
Labels for ImageNet and Cifar-10 required by MXNet
valentina-kustikova Feb 7, 2023
559e8f4
Prepare benchmark to infer MXNet models (requires to be tested)
valentina-kustikova Feb 10, 2023
d827f72
Becnhmark readme fixes
valentina-kustikova Feb 10, 2023
1a0eca1
Bug fixes (checked for alexnet)
valentina-kustikova Feb 17, 2023
dcab1db
Coding style bug fixes
valentina-kustikova Feb 17, 2023
f2155bd
Fix logging format
valentina-kustikova Feb 19, 2023
bf72f88
Validation results for InceptionV3
valentina-kustikova Feb 21, 2023
817bacd
Fix coding style
valentina-kustikova Feb 21, 2023
1e4ae84
Fix coding style
valentina-kustikova Feb 21, 2023
dd0e1d4
Add smoke test for MXNet inference
valentina-kustikova Feb 21, 2023
cb4e025
Fix smoke test for MXNet inference
valentina-kustikova Feb 22, 2023
08a9658
Fix config for MXNet smoke test
valentina-kustikova Feb 22, 2023
ffce6cc
Fix readme (add MXNet)
valentina-kustikova Feb 22, 2023
85eb12d
Fix space
valentina-kustikova Feb 22, 2023
3a06458
Replace mxnet.gluon by gluoncv
valentina-kustikova Feb 23, 2023
6b355db
Fix smoke test
valentina-kustikova Feb 23, 2023
e7fc95c
Update valudation table
valentina-kustikova Feb 24, 2023
6d1ae6e
Fix valudation table
valentina-kustikova Feb 24, 2023
4191d6a
Add checklist for supported MXNet models
valentina-kustikova Feb 25, 2023
6677a29
Merge branch 'master' into mxnet-inference
valentina-kustikova Feb 25, 2023
d2a05be
Fix formatting in checklist
valentina-kustikova Feb 25, 2023
e06509e
Add validated models
valentina-kustikova Feb 25, 2023
06391ae
Fix list of available MXNet models
valentina-kustikova Feb 25, 2023
4c44514
Update validation table (add int8 models)
valentina-kustikova Feb 26, 2023
b0432b6
Update validation table (add some models)
valentina-kustikova Feb 26, 2023
e8b7d55
Fix formatting
valentina-kustikova Feb 26, 2023
829d1a9
Merge branch 'master' into mxnet-inference
valentina-kustikova Mar 3, 2023
62e574b
Fix requirements for smoke test + GPU -> NVIDIA GPU
valentina-kustikova Mar 3, 2023
f8cd899
Print output when number of iterations equals 1
valentina-kustikova Mar 4, 2023
1b67c03
Add notes to the validation results
valentina-kustikova Mar 4, 2023
f638536
Remove space
valentina-kustikova Mar 4, 2023
7fb5c88
Merge branch 'master' into mxnet-inference
valentina-kustikova Mar 6, 2023
dfd5fd4
Fix preparing output for classification task
valentina-kustikova Mar 9, 2023
00bad66
Fix coding style
valentina-kustikova Mar 9, 2023
2e5bb26
Fix import + code cleanings + support saving model
valentina-kustikova Mar 18, 2023
b6a9e07
Fix readme
valentina-kustikova Mar 18, 2023
d7ea596
Fix smoke test
valentina-kustikova Mar 18, 2023
5255732
Fix coding style + update readme
valentina-kustikova Mar 18, 2023
a10dcd0
Print traceback
valentina-kustikova Mar 19, 2023
636f16b
Merge branch 'master' into mxnet-inference
valentina-kustikova Mar 19, 2023
ff371d2
Merge
valentina-kustikova Apr 28, 2023
5dcc5c5
Revert
valentina-kustikova Apr 28, 2023
3f0885b
Merge remote-tracking branch 'upstream/master' into mxnet-inference
valentina-kustikova Apr 28, 2023
cff84b6
Add hybrid flag
valentina-kustikova Apr 28, 2023
1a35f3c
Fix codeing style
valentina-kustikova Apr 28, 2023
e516ea7
Remove input_name and input_shape from report parameters + replace fi…
valentina-kustikova May 1, 2023
55fca5a
Fix converter (add parameters head)
valentina-kustikova May 1, 2023
3050e0a
Merge remote-tracking branch 'origin/mxnet-inference' into mxnet-infe…
valentina-kustikova May 1, 2023
0681e5d
Fix coding style
valentina-kustikova May 1, 2023
ee0899b
Fix coding style
valentina-kustikova May 1, 2023
ced99b8
Fix table beautication
valentina-kustikova May 2, 2023
94a781d
Merge conflicts
valentina-kustikova May 8, 2023
8090fad
Fix converter multiframework data
valentina-kustikova May 13, 2023
6862618
Code cleanings
valentina-kustikova May 14, 2023
e3929f4
Move common functions to postprocessing_data.py
valentina-kustikova May 14, 2023
be00d15
Move common functions for async mode to postprocessing_data.py
valentina-kustikova May 14, 2023
d0aac8e
Move computing and printing performance metrics
valentina-kustikova May 15, 2023
82b2f5c
Fix sync mode + fix coding style
valentina-kustikova May 15, 2023
536f85a
Code cleanings
valentina-kustikova May 15, 2023
bde34ec
Merge conflicts
valentina-kustikova May 15, 2023
4821c1c
Remove unreported parameters
valentina-kustikova May 15, 2023
15a0cf1
Code cleanings
valentina-kustikova May 15, 2023
4c0c472
Fix report parameters
valentina-kustikova May 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,9 @@ Intel® Optimization for TensorFlow, TensorFlow Lite, OpenCV, MXNet и PyTorch
осуществляется последовательный и независимый запуск запросов.
Запуск очередного запроса выполняется после завершения предыдущего.
Для каждого запроса осуществляется замер времени его выполнения.
Для множества полученных времен определяется стандартное среднеквадратичное
отклонение, и отбрасываются времена, выходящие за пределы трех стандартных
отклонений от среднего времени вывода. Результирующий набор времен
Для множества полученных времен определяется стандартное среднеквадратичное
отклонение, и отбрасываются времена, выходящие за пределы трех стандартных
отклонений от среднего времени вывода. Результирующий набор времен
используется для вычисления показателя латентности.
Остальные показатели вычисляются для всего множества времен.

Expand Down
14 changes: 1 addition & 13 deletions src/benchmark/frameworks/config_parser/test_reporter.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,19 +62,7 @@ def prepare_framework_params(self):
match_parameter_description['num_inter_threads'] = 'Inter threads'
match_parameter_description['num_intra_threads'] = 'Intra threads'

match_parameter_description['mean'] = 'Mean'
match_parameter_description['input_scale'] = 'Scale'
match_parameter_description['layout'] = 'Layout'
match_parameter_description['input_shape'] = 'Input shape'
match_parameter_description['input_name'] = 'Input name'
match_parameter_description['output_names'] = 'Output names'
# duplicate because pytorch launcher does not match common template. To be fixed
match_parameter_description['output_name'] = 'Output name'
maslovaz marked this conversation as resolved.
Show resolved Hide resolved
match_parameter_description['normalize'] = 'Normalization flag'
match_parameter_description['std'] = 'Standard deviation'
match_parameter_description['channel_swap'] = 'Channel swap'
match_parameter_description['swapRB'] = 'Channel swap'
match_parameter_description['crop'] = 'Crop'
match_parameter_description['hybridize'] = 'Hybridization flag'
match_parameter_description['model_type'] = 'Model type'
match_parameter_description['inference_mode'] = 'Inference mode'

Expand Down
11 changes: 10 additions & 1 deletion src/benchmark/frameworks/mxnet/mxnet_parameters_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ def parse_parameters(self, curr_test):
CONFIG_FRAMEWORK_DEPENDENT_TAG = 'FrameworkDependent'
CONFIG_FRAMEWORK_DEPENDENT_INPUT_NAME_TAG = 'InputName'
CONFIG_FRAMEWORK_DEPENDENT_INPUT_SHAPE_TAG = 'InputShape'
CONFIG_FRAMEWORK_DEPENDENT_HYBRIDIZE_TAG = 'Hybridize'
CONFIG_FRAMEWORK_DEPENDENT_NORMALIZE_TAG = 'Normalize'
CONFIG_FRAMEWORK_DEPENDENT_MEAN_TAG = 'Mean'
CONFIG_FRAMEWORK_DEPENDENT_STD_TAG = 'Std'
Expand All @@ -18,6 +19,9 @@ def parse_parameters(self, curr_test):
CONFIG_FRAMEWORK_DEPENDENT_INPUT_NAME_TAG)[0].firstChild
_input_shape = dep_parameters_tag.getElementsByTagName(
CONFIG_FRAMEWORK_DEPENDENT_INPUT_SHAPE_TAG)[0].firstChild
_hybridize = dep_parameters_tag.getElementsByTagName(
CONFIG_FRAMEWORK_DEPENDENT_HYBRIDIZE_TAG)[0].firstChild
print(_hybridize.data)
_normalize = dep_parameters_tag.getElementsByTagName(
CONFIG_FRAMEWORK_DEPENDENT_NORMALIZE_TAG)[0].firstChild
_mean = dep_parameters_tag.getElementsByTagName(
Expand All @@ -30,6 +34,7 @@ def parse_parameters(self, curr_test):
return MXNetParameters(
input_name=_input_name.data if _input_name else None,
input_shape=_input_shape.data if _input_shape else None,
hybridize=_hybridize.data if _hybridize else None,
normalize=_normalize.data if _normalize else None,
mean=_mean.data if _mean else None,
std=_std.data if _std else None,
Expand All @@ -38,9 +43,11 @@ def parse_parameters(self, curr_test):


class MXNetParameters(FrameworkParameters):
def __init__(self, input_name, input_shape, normalize, mean, std, channel_swap):
def __init__(self, input_name, input_shape, hybridize, normalize,
mean, std, channel_swap):
self.input_name = None
self.input_shape = None
self.hybridize = None
self.normalize = None
self.mean = None
self.std = None
Expand All @@ -50,6 +57,8 @@ def __init__(self, input_name, input_shape, normalize, mean, std, channel_swap):
self.input_name = input_name
if self._parameter_is_not_none(input_shape):
self.input_shape = input_shape
if self._parameter_is_not_none(hybridize):
self.hybridize = hybridize
if self._parameter_is_not_none(normalize):
self.normalize = normalize
if self._parameter_is_not_none(mean):
Expand Down
5 changes: 5 additions & 0 deletions src/benchmark/frameworks/mxnet/mxnet_process.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ def _fill_command_line(self):
common_params = MXNetProcess._add_optional_argument_to_cmd_line(
common_params, '--input_name', input_name)

hybridize = self._test.dep_parameters.hybridize
if hybridize == 'True':
common_params = MXNetProcess._add_flag_to_cmd_line(
common_params, '--hybrid')

normalize = self._test.dep_parameters.normalize
if normalize == 'True':
common_params = MXNetProcess._add_flag_to_cmd_line(
Expand Down
1 change: 1 addition & 0 deletions src/configs/benchmark_configuration_file_template.xml
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@
<FrameworkDependent>
<InputName></InputName>
<InputShape></InputShape>
<Hybridize></Hybridize>
<Normalize></Normalize>
<Mean></Mean>
<Std></Std>
Expand Down
Loading