Sourcery refactored main branch #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
| eval_result = val.eval_model(pred_result, dummy_model, dataloader, task) | ||
| return eval_result | ||
| return val.eval_model(pred_result, dummy_model, dataloader, task) |
There was a problem hiding this comment.
Function run refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
Comment on lines
-48
to
+51
| assert not (device.type == 'cpu' and args.half), '--half only compatible with GPU export, i.e. use --device 0' | ||
| assert ( | ||
| device.type != 'cpu' or not args.half | ||
| ), '--half only compatible with GPU export, i.e. use --device 0' | ||
|
|
There was a problem hiding this comment.
Lines 48-149 refactored with the following changes:
- Simplify logical expression using De Morgan identities (
de-morgan) - Move assignment closer to its usage within a block (
move-assign-in-block) - Merge dictionary updates via the union operator [×2] (
dict-assign-update-to-union) - Merge nested if conditions (
merge-nested-ifs)
Comment on lines
-39
to
+47
| # Create a 4D blob from a frame. | ||
| blob = cv2.dnn.blobFromImage(input_image, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False) | ||
| # Create a 4D blob from a frame. | ||
| blob = cv2.dnn.blobFromImage(input_image, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False) | ||
|
|
||
| # Sets the input to the network. | ||
| net.setInput(blob) | ||
| # Sets the input to the network. | ||
| net.setInput(blob) | ||
|
|
||
| # Runs the forward pass to get output of the output layers. | ||
| output_layers = net.getUnconnectedOutLayersNames() | ||
| outputs = net.forward(output_layers) | ||
| # print(outputs[0].shape) | ||
|
|
||
| return outputs | ||
| # Runs the forward pass to get output of the output layers. | ||
| output_layers = net.getUnconnectedOutLayersNames() | ||
| return net.forward(output_layers) |
There was a problem hiding this comment.
Function pre_process refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
This removes the following comments ( why? ):
# print(outputs[0].shape)
Comment on lines
-35
to
+41
| # Create a 4D blob from a frame. | ||
| blob = cv2.dnn.blobFromImage(input_image, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False) | ||
| # Create a 4D blob from a frame. | ||
| blob = cv2.dnn.blobFromImage(input_image, 1/255, (INPUT_WIDTH, INPUT_HEIGHT), [0,0,0], 1, crop=False) | ||
|
|
||
| # Sets the input to the network. | ||
| net.setInput(blob) | ||
|
|
||
| # Run the forward pass to get output of the output layers. | ||
| outputs = net.forward(net.getUnconnectedOutLayersNames()) | ||
| return outputs | ||
| # Sets the input to the network. | ||
| net.setInput(blob) | ||
|
|
||
| return net.forward(net.getUnconnectedOutLayersNames()) |
There was a problem hiding this comment.
Function pre_process refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
This removes the following comments ( why? ):
# Run the forward pass to get output of the output layers.
| if not p6: | ||
| self.strides = [8, 16, 32] | ||
| else: | ||
| self.strides = [8, 16, 32, 64] | ||
| self.strides = [8, 16, 32, 64] if p6 else [8, 16, 32] |
There was a problem hiding this comment.
Function yolox.__init__ refactored with the following changes:
- Replace if statement with if expression (
assign-if-exp) - Swap if/else branches of if expression to remove negation (
swap-if-expression)
Comment on lines
-87
to
+98
| if self.index < self.length: | ||
| for i in range(self.batch_size): | ||
| assert os.path.exists(self.img_list[i + self.index * self.batch_size]), 'not found!!' | ||
| img = cv2.imread(self.img_list[i + self.index * self.batch_size]) | ||
| img = precess_image(img, self.input_h, 32) | ||
| self.calibration_data[i] = img | ||
|
|
||
| self.index += 1 | ||
| return np.ascontiguousarray(self.calibration_data, dtype=np.float32) | ||
| else: | ||
| if self.index >= self.length: | ||
| return np.array([]) | ||
| for i in range(self.batch_size): | ||
| assert os.path.exists(self.img_list[i + self.index * self.batch_size]), 'not found!!' | ||
| img = cv2.imread(self.img_list[i + self.index * self.batch_size]) | ||
| img = precess_image(img, self.input_h, 32) | ||
| self.calibration_data[i] = img | ||
|
|
||
| self.index += 1 | ||
| return np.ascontiguousarray(self.calibration_data, dtype=np.float32) |
There was a problem hiding this comment.
Function DataLoader.next_batch refactored with the following changes:
- Swap if/else branches (
swap-if-else-branches) - Remove unnecessary else after guard condition (
remove-unnecessary-else)
| args = parser.parse_args() | ||
| return args | ||
| return parser.parse_args() |
There was a problem hiding this comment.
Function parse_args refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
Comment on lines
-65
to
+66
| sys.exit('%s is not a valid directory' % args.imgs_dir) | ||
| sys.exit(f'{args.imgs_dir} is not a valid directory') | ||
| if not os.path.isfile(args.annotations): | ||
| sys.exit('%s is not a valid file' % args.annotations) | ||
| sys.exit(f'{args.annotations} is not a valid file') |
There was a problem hiding this comment.
Function check_args refactored with the following changes:
- Replace interpolated string formatting with f-string [×2] (
replace-interpolation-with-fstring)
Comment on lines
-105
to
+104
|
|
There was a problem hiding this comment.
Function generate_results refactored with the following changes:
- Swap if/else branches of if expression to remove negation (
swap-if-expression)
Comment on lines
-144
to
+153
| print('Serialized the TensorRT engine to file: %s' % args.model) | ||
| print(f'Serialized the TensorRT engine to file: {args.model}') | ||
|
|
||
| model_prefix = args.model.replace('.trt', '').split('/')[-1] | ||
| results_file = 'results_{}.json'.format(model_prefix) | ||
| results_file = f'results_{model_prefix}.json' |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring) - Replace call to format with f-string (
use-fstring-for-formatting)
| raise ValueError("Unsupported data type: %s" % dtype) | ||
| raise ValueError(f"Unsupported data type: {dtype}") |
There was a problem hiding this comment.
Function build_engine_from_onnx refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring) - Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-174
to
+179
| engine_path = args.model.replace('.onnx', '-int8-{}-{}-minmax.trt'.format(args.batch_size, args.num_calib_batch)) | ||
| engine_path = args.model.replace( | ||
| '.onnx', | ||
| f'-int8-{args.batch_size}-{args.num_calib_batch}-minmax.trt', | ||
| ) | ||
|
|
||
|
|
||
| with open(engine_path, 'wb') as f: | ||
| f.write(engine.serialize()) | ||
| print('Serialized the TensorRT engine to file: %s' % engine_path) | ||
| print(f'Serialized the TensorRT engine to file: {engine_path}') |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting) - Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring)
| args = parser.parse_args() | ||
| return args | ||
| return parser.parse_args() |
There was a problem hiding this comment.
Function parse_args refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
Comment on lines
-54
to
+55
| sys.exit('%s is not a valid directory' % args.imgs_dir) | ||
| sys.exit(f'{args.imgs_dir} is not a valid directory') | ||
| if not os.path.exists(args.visual_dir): | ||
| print("Directory {} does not exist, create it".format(args.visual_dir)) | ||
| print(f"Directory {args.visual_dir} does not exist, create it") |
There was a problem hiding this comment.
Function check_args refactored with the following changes:
- Replace interpolated string formatting with f-string (
replace-interpolation-with-fstring) - Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-109
to
+108
| cv2.imwrite("{}".format(os.path.join(visual_dir, image_names[j])), image) | ||
| cv2.imwrite(f"{os.path.join(visual_dir, image_names[j])}", image) |
There was a problem hiding this comment.
Function generate_results refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-91
to
+98
| line = name + " " + "{:0.4f}".format(mAP1) + " " + "{:0.4f}".format(mAP2) + "\n" | ||
| line = ( | ||
| f"{name} " | ||
| + "{:0.4f}".format(mAP1) | ||
| + " " | ||
| + "{:0.4f}".format(mAP2) | ||
| + "\n" | ||
| ) | ||
|
|
There was a problem hiding this comment.
Function quant_sensitivity_save refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation)
Comment on lines
-35
to
+44
| if isinstance(m, quant_nn.QuantConv2d) or \ | ||
| isinstance(m, quant_nn.QuantConvTranspose2d): | ||
| if isinstance( | ||
| m, (quant_nn.QuantConv2d, quant_nn.QuantConvTranspose2d) | ||
| ): | ||
| # print(m) | ||
| # print(m._weight_quantizer._amax) | ||
| weight_amax = m._weight_quantizer._amax.detach().cpu().numpy() | ||
| # print(weight_amax) | ||
| print(k) | ||
| ones = np.ones_like(weight_amax) | ||
| print("zero scale number = {}".format(np.sum(weight_amax == 0.0))) | ||
| print(f"zero scale number = {np.sum(weight_amax == 0.0)}") |
There was a problem hiding this comment.
Function zero_scale_fix refactored with the following changes:
- Merge isinstance calls (
merge-isinstance) - Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-148
to
+152
| export_file = args.quant_weights.replace('.pt', '_bs{}.onnx'.format(args.export_batch_size)) # filename | ||
| export_file = args.quant_weights.replace( | ||
| '.pt', f'_bs{args.export_batch_size}.onnx' | ||
| ) | ||
|
|
There was a problem hiding this comment.
Lines 148-148 refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-71
to
79
| print("Skip Layer {}".format(k)) | ||
| print(f"Skip Layer {k}") | ||
| continue | ||
| if ( | ||
| args.calib is True | ||
| and cfg.ptq.sensitive_layers_skip is True | ||
| and k in cfg.ptq.sensitive_layers_list | ||
| ): | ||
| print(f"Skip Layer {k}") | ||
| continue |
There was a problem hiding this comment.
Function qat_init_model_manu refactored with the following changes:
- Replace call to format with f-string [×2] (
use-fstring-for-formatting) - Merge nested if conditions (
merge-nested-ifs)
Comment on lines
-51
to
+64
|
|
||
| config = builder.create_builder_config() | ||
|
|
||
| # If it is a dynamic onnx model , you need to add the following. | ||
| # profile = builder.create_optimization_profile() | ||
| # profile.set_shape("input_name", (batch, channels, min_h, min_w), (batch, channels, opt_h, opt_w), (batch, channels, max_h, max_w)) | ||
| # config.add_optimization_profile(profile) | ||
|
|
||
|
|
||
| parser = trt.OnnxParser(network, TRT_LOGGER) | ||
| config.max_workspace_size = GiB(1) | ||
|
|
||
| if not os.path.exists(onnx_file): | ||
| quit('ONNX file {} not found'.format(onnx_file)) | ||
| quit(f'ONNX file {onnx_file} not found') |
There was a problem hiding this comment.
Function build_engine refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
Comment on lines
-85
to
+98
| # Use calibration files from validation dataset if no cache exists | ||
| else: | ||
| if not calib_data: | ||
| raise ValueError("ERROR: Int8 mode requested, but no calibration data provided. Please provide --calibration-data /path/to/calibration/files") | ||
|
|
||
| elif calib_data: | ||
| calib_files = get_calibration_files(calib_data, max_calib_size) | ||
|
|
||
| else: | ||
| raise ValueError("ERROR: Int8 mode requested, but no calibration data provided. Please provide --calibration-data /path/to/calibration/files") | ||
|
|
||
| # Choose pre-processing function for INT8 calibration | ||
| preprocess_func = preprocess_yolov6 | ||
|
|
||
| int8_calibrator = ImageCalibrator(calibration_files=calib_files, | ||
| batch_size=calib_batch_size, | ||
| cache_file=calib_cache) | ||
| return int8_calibrator | ||
| return ImageCalibrator( | ||
| calibration_files=calib_files, | ||
| batch_size=calib_batch_size, | ||
| cache_file=calib_cache, | ||
| ) |
There was a problem hiding this comment.
Function get_int8_calibrator refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable) - Merge else clause's nested if statement into elif (
merge-else-if-into-elif) - Lift code into else after jump in control flow (
reintroduce-else) - Swap if/else branches (
swap-if-else-branches)
This removes the following comments ( why? ):
# Use calibration files from validation dataset if no cache exists
Comment on lines
-123
to
+129
| if len(calibration_files) == 0: | ||
| if not calibration_files: | ||
| raise Exception("ERROR: Calibration data path [{:}] contains no files!".format(calibration_data)) | ||
|
|
||
| if max_calibration_size: | ||
| if len(calibration_files) > max_calibration_size: | ||
| logger.warning("Capping number of calibration images to max_calibration_size: {:}".format(max_calibration_size)) | ||
| random.seed(42) # Set seed for reproducibility | ||
| calibration_files = random.sample(calibration_files, max_calibration_size) | ||
| if max_calibration_size and len(calibration_files) > max_calibration_size: | ||
| logger.warning("Capping number of calibration images to max_calibration_size: {:}".format(max_calibration_size)) | ||
| random.seed(42) # Set seed for reproducibility | ||
| calibration_files = random.sample(calibration_files, max_calibration_size) |
There was a problem hiding this comment.
Function get_calibration_files refactored with the following changes:
- Simplify sequence length comparison (
simplify-len-comparison) - Merge nested if conditions (
merge-nested-ifs)
| logger.debug("{} - OptProfile {} - Min {} Opt {} Max {}".format(inp.name, i, _min, _opt, _max)) | ||
| logger.debug(f"{inp.name} - OptProfile {i} - Min {_min} Opt {_opt} Max {_max}") |
There was a problem hiding this comment.
Function add_profiles refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
| if all([inp.shape[0] > -1 for inp in inputs]): | ||
| if all(inp.shape[0] > -1 for inp in inputs): |
There was a problem hiding this comment.
Function create_optimization_profiles refactored with the following changes:
- Replace unneeded comprehension with generator (
comprehension-to-generator)
Comment on lines
-166
to
+179
| with trt.Builder(TRT_LOGGER) as builder, \ | ||
| builder.create_network(network_flags) as network, \ | ||
| builder.create_builder_config() as config, \ | ||
| trt.OnnxParser(network, TRT_LOGGER) as parser: | ||
| with (trt.Builder(TRT_LOGGER) as builder, builder.create_network(network_flags) as network, builder.create_builder_config() as config, trt.OnnxParser(network, TRT_LOGGER) as parser): | ||
|
|
||
| config.max_workspace_size = 2**30 # 1GiB | ||
|
|
||
| # Set Builder Config Flags | ||
| for flag in builder_flag_map: | ||
| if getattr(args, flag): | ||
| logger.info("Setting {}".format(builder_flag_map[flag])) | ||
| logger.info(f"Setting {builder_flag_map[flag]}") | ||
| config.set_flag(builder_flag_map[flag]) | ||
|
|
||
| # Fill network atrributes with information by parsing model | ||
| with open(args.onnx, "rb") as f: | ||
| if not parser.parse(f.read()): | ||
| print('ERROR: Failed to parse the ONNX file: {}'.format(args.onnx)) | ||
| print(f'ERROR: Failed to parse the ONNX file: {args.onnx}') |
There was a problem hiding this comment.
Function main refactored with the following changes:
- Replace call to format with f-string [×2] (
use-fstring-for-formatting)
| LOGGER.info("Model Summary: {}".format(get_model_info(model, self.img_size))) | ||
| LOGGER.info(f"Model Summary: {get_model_info(model, self.img_size)}") |
There was a problem hiding this comment.
Function Evaler.init_model refactored with the following changes:
- Replace call to format with f-string (
use-fstring-for-formatting)
| if task == 'val' or task == 'test': | ||
| if task in ['val', 'test']: |
There was a problem hiding this comment.
Function Evaler.check_thres refactored with the following changes:
- Replace multiple comparisons of same variable with
inoperator (merge-comparisons)
Comment on lines
-439
to
+520
| # https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/ | ||
| x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, | ||
| 21, 22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, | ||
| 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, | ||
| 59, 60, 61, 62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, | ||
| 80, 81, 82, 84, 85, 86, 87, 88, 89, 90] | ||
| return x | ||
| return [ | ||
| 1, | ||
| 2, | ||
| 3, | ||
| 4, | ||
| 5, | ||
| 6, | ||
| 7, | ||
| 8, | ||
| 9, | ||
| 10, | ||
| 11, | ||
| 13, | ||
| 14, | ||
| 15, | ||
| 16, | ||
| 17, | ||
| 18, | ||
| 19, | ||
| 20, | ||
| 21, | ||
| 22, | ||
| 23, | ||
| 24, | ||
| 25, | ||
| 27, | ||
| 28, | ||
| 31, | ||
| 32, | ||
| 33, | ||
| 34, | ||
| 35, | ||
| 36, | ||
| 37, | ||
| 38, | ||
| 39, | ||
| 40, | ||
| 41, | ||
| 42, | ||
| 43, | ||
| 44, | ||
| 46, | ||
| 47, | ||
| 48, | ||
| 49, | ||
| 50, | ||
| 51, | ||
| 52, | ||
| 53, | ||
| 54, | ||
| 55, | ||
| 56, | ||
| 57, | ||
| 58, | ||
| 59, | ||
| 60, | ||
| 61, | ||
| 62, | ||
| 63, | ||
| 64, | ||
| 65, | ||
| 67, | ||
| 70, | ||
| 72, | ||
| 73, | ||
| 74, | ||
| 75, | ||
| 76, | ||
| 77, | ||
| 78, | ||
| 79, | ||
| 80, | ||
| 81, | ||
| 82, | ||
| 84, | ||
| 85, | ||
| 86, | ||
| 87, | ||
| 88, | ||
| 89, | ||
| 90, | ||
| ] |
There was a problem hiding this comment.
Function Evaler.coco80_to_coco91_class refactored with the following changes:
- Inline variable that is immediately returned (
inline-immediately-returned-variable)
This removes the following comments ( why? ):
# https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/
| with open(txt_path + '.txt', 'a') as f: | ||
| with open(f'{txt_path}.txt', 'a') as f: |
There was a problem hiding this comment.
Function Inferer.infer refactored with the following changes:
- Use f-string instead of string concatenation (
use-fstring-for-concatenation)
| outside = p1[1] - h - 3 >= 0 # label fits outside box | ||
| outside = p1[1] - h >= 3 |
There was a problem hiding this comment.
Function Inferer.plot_box_and_label refactored with the following changes:
- Simplify numeric comparison (
simplify-numeric-comparison)
This removes the following comments ( why? ):
# label fits outside box
Sourcery Code Quality Report✅ Merging this PR will increase code quality in the affected files by 0.05%.
Here are some functions in these files that still need a tune-up:
Legend and ExplanationThe emojis denote the absolute quality of the code:
The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request. Please see our documentation here for details on how these metrics are calculated. We are actively working on this report - lots more documentation and extra metrics to come! Help us improve this quality report! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Branch
mainrefactored by Sourcery.If you're happy with these changes, merge this Pull Request using the Squash and merge strategy.
See our documentation here.
Run Sourcery locally
Reduce the feedback loop during development by using the Sourcery editor plugin:
Review changes via command line
To manually merge these changes, make sure you're on the
mainbranch, then run:Help us improve this pull request!