Code doesn't work correctly and results don't replicate

[viggykrish]

The code errors out in ope.py on line 226

key=self.with_emb(appearance)

I fixed this to the following

key=self.with_emb(appearance, query_pos_emb)

The code now runs but the results are not very good. Can you take a look?

[djukicn]

@viggykrish Thanks for your question. There is indeed a bug in line 226, thanks for noticing. However, this part of the code is reached only when --pre_norm flag is not used. This corresponds to Post-LN transformer that is more difficult to train and is not used in our work (the code is there just for completeness and that's why the bug went unnoticed).

Can you make sure that you used the settings as used in train_few_shot.sh for training or as in README for evaluation and let me know whether the results are any better?

[viggykrish]

Thank you for getting back so quickly. I also updated another line in evaluate.py to

model.load_state_dict(state_dict,strict=False)

Otherwise I get the following runtime error.

Exception has occurred: RuntimeError
Error(s) in loading state_dict for LOCA:
Missing key(s) in state_dict: "backbone.backbone.conv1.weight", "backbone.backbone.bn1.weight", "backbone.backbone.bn1.bias", "backbone.backbone.bn1.running_mean", "backbone.backbone.bn1.running_var", "backbone.backbone.layer1.0.conv1.weight", "backbone.backbone.layer1.0.bn1.weight", "backbone.backbone.layer1.0.bn1.bias", "backbone.backbone.layer1.0.bn1.running_mean", "backbone.backbone.layer1.0.bn1.running_var", "backbone.backbone.layer1.0.conv2.weight", "backbone.backbone.layer1.0.bn2.weight", "backbone.backbone.layer1.0.bn2.bias", "backbone.backbone.layer1.0.bn2.running_mean", "backbone.backbone.layer1.0.bn2.running_var", "backbone.backbone.layer1.0.conv3.weight", "backbone.backbone.layer1.0.bn3.weight", "backbone.backbone.layer1.0.bn3.bias", "backbone.backbone.layer1.0.bn3.running_mean", "backbone.backbone.layer1.0.bn3.running_var", "backbone.backbone.layer1.0.downsample.0.weight", "backbone.backbone.layer1.0.downsample.1.weight", "backbone.backbone.layer1.0.downsample.1.bias", "backbone.backbone.layer1.0.downsample.1.running_mean", "backbone.backbone.layer1.0.downsample.1.running_var", "backbone.backbone.layer1.1.conv1.weight", "backbone.backbone.layer1.1.bn1.weight", "backbone.backbone.layer1.1.bn1.bias", "backbone.backbone.layer1.1.bn1.running_mean", "backbone.backbone.layer1.1.bn1.running_var", "backbone.backbone.layer1.1.conv2.weight", "backbone.backbone.layer1.1.bn2.weight", "backbone.backbone.layer1.1.bn2.bias", "backbone.backbone.layer1.1.bn2.running_mean", "backbone.backbone.layer1.1.bn2.running_var", "backbone.backbone.layer1.1.conv3.weight", "backbone.backbone.layer1.1.bn3.weight", "backbone.backbone.layer1.1.bn3.bias", "backbone.backbone.layer1.1.bn3.running_mean", "backbone.backbone.layer1.1.bn3.running_var", "backbone.backbone.layer1.2.conv1.weight", "backbone.backbone.layer1.2.bn1.weight", "backbone.backbone.layer1.2.bn1.bias", "backbone.backbone.layer1.2.bn1.running_mean", "backbone.backbone.layer1.2.bn1.running_var", "backbone.backbone.layer1.2.conv2.weight", "backbone.backbone.layer1.2.bn2.weight", "backbone.backbone.layer1.2.bn2.bias", "backbone.backbone.layer1.2.bn2.running_mean", "backbone.backbone.layer1.2.bn2.running_var", "backbone.backbone.layer1.2.conv3.weight", "backbone.backbone.layer1.2.bn3.weight", "backbone.backbone.layer1.2.bn3.bias", "backbone.backbone.layer1.2.bn3.running_mean", "backbone.backbone.layer1.2.bn3.running_var", "backbone.backbone.layer2.0.conv1.weight", "backbone.backbone.layer2.0.bn1.weight", "backbone.backbone.layer2.0.bn1.bias", "backbone.backbone.layer2.0.bn1.running_mean", "backbone.backbone.layer2.0.bn1.running_var", "backbone.backbone.layer2.0.conv2.weight", "backbone.backbone.layer2.0.bn2.weight", "backbone.backbone.layer2.0.bn2.bias", "backbone.backbone.layer2.0.bn2.running_mean", "backbone.backbone.layer2.0.bn2.running_var", "backbone.backbone.layer2.0.conv3.weight", "backbone.backbone.layer2.0.bn3.weight", "backbone.backbone.layer2.0.bn3.bias", "backbone.backbone.layer2.0.bn3.running_mean", "backbone.backbone.layer2.0.bn3.running_var", "backbone.backbone.layer2.0.downsample.0.weight", "backbone.backbone.layer2.0.downsample.1.weight", "backbone.backbone.layer2.0.downsample.1.bias", "backbone.backbone.layer2.0.downsample.1.running_mean", "backbone.backbone.layer2.0.downsample.1.running_var", "backbone.backbone.layer2.1.conv1.weight", "backbone.backbone.layer2.1.bn1.weight", "backbone.backbone.layer2.1.bn1.bias", "backbone.backbone.layer2.1.bn1.running_mean", "backbone.backbone.layer2.1.bn1.running_var", "backbone.backbone.layer2.1.conv2.weight", "backbone.backbone.layer2.1.bn2.weight", "backbone.backbone.layer2.1.bn2.bias", "backbone.backbone.layer2.1.bn2.running_mean", "backbone.backbone.layer2.1.bn2.running_var", "backbone.backbone.layer2.1.conv3.weight", "backbone.backbone.layer2.1.bn3.weight", "backbone.backbone.layer2.1.bn3.bias", "backbone.backbone.layer2.1.bn3.running_mean", "backbone.backbone.layer2.1.bn3.running_var", "backbone.backbone.layer2.2.conv1.weight", "backbone.backbone.layer2.2.bn1.weight", "backbone.backbone.layer2.2.bn1.bias", "backbone.backbone.layer2.2.bn1.running_mean", "backbone.backbone.layer2.2.bn1.running_var", "backbone.backbone.layer2.2.conv2.weight", "backbone.backbone.layer2.2.bn2.weight", "backbone.backbone.layer2.2.bn2.bias", "backbone.backbone.layer2.2.bn2.running_mean", "backbone.backbone.layer2.2.bn2.running_var", "backbone.backbone.layer2.2.conv3.weight", "backbone.backbone.layer2.2.bn3.weight", "backbone.backbone.layer2.2.bn3.bias", "backbone.backbone.layer2.2.bn3.running_mean", "backbone.backbone.layer2.2.bn3.running_var", "backbone.backbone.layer2.3.conv1.weight", "backbone.backbone.layer2.3.bn1.weight", "backbone.backbone.layer2.3.bn1.bias", "backbone.backbone.layer2.3.bn1.running_mean", "backbone.backbone.layer2.3.bn1.running_var", "backbone.backbone.layer2.3.conv2.weight", "backbone.backbone.layer2.3.bn2.weight", "backbone.backbone.layer2.3.bn2.bias", "backbone.backbone.layer2.3.bn2.running_mean", "backbone.backbone.layer2.3.bn2.running_var", "backbone.backbone.layer2.3.conv3.weight", "backbone.backbone.layer2.3.bn3.weight", "backbone.backbone.layer2.3.bn3.bias", "backbone.backbone.layer2.3.bn3.running_mean", "backbone.backbone.layer2.3.bn3.running_var", "backbone.backbone.layer3.0.conv1.weight", "backbone.backbone.layer3.0.bn1.weight", "backbone.backbone.layer3.0.bn1.bias", "backbone.backbone.layer3.0.bn1.running_mean", "backbone.backbone.layer3.0.bn1.running_var", "backbone.backbone.layer3.0.conv2.weight", "backbone.backbone.layer3.0.bn2.weight", "backbone.backbone.layer3.0.bn2.bias", "backbone.backbone.layer3.0.bn2.running_mean", "backbone.backbone.layer3.0.bn2.running_var", "backbone.backbone.layer3.0.conv3.weight", "backbone.backbone.layer3.0.bn3.weight", "backbone.backbone.layer3.0.bn3.bias", "backbone.backbone.layer3.0.bn3.running_mean", "backbone.backbone.layer3.0.bn3.running_var", "backbone.backbone.layer3.0.downsample.0.weight", "backbone.backbone.layer3.0.downsample.1.weight", "backbone.backbone.layer3.0.downsample.1.bias", "backbone.backbone.layer3.0.downsample.1.running_mean", "backbone.backbone.layer3.0.downsample.1.running_var", "backbone.backbone.layer3.1.conv1.weight", "backbone.backbone.layer3.1.bn1.weight", "backbone.backbone.layer3.1.bn1.bias", "backbone.backbone.layer3.1.bn1.running_mean", "backbone.backbone.layer3.1.bn1.running_var", "backbone.backbone.layer3.1.conv2.weight", "backbone.backbone.layer3.1.bn2.weight", "backbone.backbone.layer3.1.bn2.bias", "backbone.backbone.layer3.1.bn2.running_mean", "backbone.backbone.layer3.1.bn2.running_var", "backbone.backbone.layer3.1.conv3.weight", "backbone.backbone.layer3.1.bn3.weight", "backbone.backbone.layer3.1.bn3.bias", "backbone.backbone.layer3.1.bn3.running_mean", "backbone.backbone.layer3.1.bn3.running_var", "backbone.backbone.layer3.2.conv1.weight", "backbone.backbone.layer3.2.bn1.weight", "backbone.backbone.layer3.2.bn1.bias", "backbone.backbone.layer3.2.bn1.running_mean", "backbone.backbone.layer3.2.bn1.running_var", "backbone.backbone.layer3.2.conv2.weight", "backbone.backbone.layer3.2.bn2.weight", "backbone.backbone.layer3.2.bn2.bias", "backbone.backbone.layer3.2.bn2.running_mean", "backbone.backbone.layer3.2.bn2.running_var", "backbone.backbone.layer3.2.conv3.weight", "backbone.backbone.layer3.2.bn3.weight", "backbone.backbone.layer3.2.bn3.bias", "backbone.backbone.layer3.2.bn3.running_mean", "backbone.backbone.layer3.2.bn3.running_var", "backbone.backbone.layer3.3.conv1.weight", "backbone.backbone.layer3.3.bn1.weight", "backbone.backbone.layer3.3.bn1.bias", "backbone.backbone.layer3.3.bn1.running_mean", "backbone.backbone.layer3.3.bn1.running_var", "backbone.backbone.layer3.3.conv2.weight", "backbone.backbone.layer3.3.bn2.weight", "backbone.backbone.layer3.3.bn2.bias", "backbone.backbone.layer3.3.bn2.running_mean", "backbone.backbone.layer3.3.bn2.running_var", "backbone.backbone.layer3.3.conv3.weight", "backbone.backbone.layer3.3.bn3.weight", "backbone.backbone.layer3.3.bn3.bias", "backbone.backbone.layer3.3.bn3.running_mean", "backbone.backbone.layer3.3.bn3.running_var", "backbone.backbone.layer3.4.conv1.weight", "backbone.backbone.layer3.4.bn1.weight", "backbone.backbone.layer3.4.bn1.bias", "backbone.backbone.layer3.4.bn1.running_mean", "backbone.backbone.layer3.4.bn1.running_var", "backbone.backbone.layer3.4.conv2.weight", "backbone.backbone.layer3.4.bn2.weight", "backbone.backbone.layer3.4.bn2.bias", "backbone.backbone.layer3.4.bn2.running_mean", "backbone.backbone.layer3.4.bn2.running_var", "backbone.backbone.layer3.4.conv3.weight", "backbone.backbone.layer3.4.bn3.weight", "backbone.backbone.layer3.4.bn3.bias", "backbone.backbone.layer3.4.bn3.running_mean", "backbone.backbone.layer3.4.bn3.running_var", "backbone.backbone.layer3.5.conv1.weight", "backbone.backbone.layer3.5.bn1.weight", "backbone.backbone.layer3.5.bn1.bias", "backbone.backbone.layer3.5.bn1.running_mean", "backbone.backbone.layer3.5.bn1.running_var", "backbone.backbone.layer3.5.conv2.weight", "backbone.backbone.layer3.5.bn2.weight", "backbone.backbone.layer3.5.bn2.bias", "backbone.backbone.layer3.5.bn2.running_mean", "backbone.backbone.layer3.5.bn2.running_var", "backbone.backbone.layer3.5.conv3.weight", "backbone.backbone.layer3.5.bn3.weight", "backbone.backbone.layer3.5.bn3.bias", "backbone.backbone.layer3.5.bn3.running_mean", "backbone.backbone.layer3.5.bn3.running_var", "backbone.backbone.layer4.0.conv1.weight", "backbone.backbone.layer4.0.bn1.weight", "backbone.backbone.layer4.0.bn1.bias", "backbone.backbone.layer4.0.bn1.running_mean", "backbone.backbone.layer4.0.bn1.running_var", "backbone.backbone.layer4.0.conv2.weight", "backbone.backbone.layer4.0.bn2.weight", "backbone.backbone.layer4.0.bn2.bias", "backbone.backbone.layer4.0.bn2.running_mean", "backbone.backbone.layer4.0.bn2.running_var", "backbone.backbone.layer4.0.conv3.weight", "backbone.backbone.layer4.0.bn3.weight", "backbone.backbone.layer4.0.bn3.bias", "backbone.backbone.layer4.0.bn3.running_mean", "backbone.backbone.layer4.0.bn3.running_var", "backbone.backbone.layer4.0.downsample.0.weight", "backbone.backbone.layer4.0.downsample.1.weight", "backbone.backbone.layer4.0.downsample.1.bias", "backbone.backbone.layer4.0.downsample.1.running_mean", "backbone.backbone.layer4.0.downsample.1.running_var", "backbone.backbone.layer4.1.conv1.weight", "backbone.backbone.layer4.1.bn1.weight", "backbone.backbone.layer4.1.bn1.bias", "backbone.backbone.layer4.1.bn1.running_mean", "backbone.backbone.layer4.1.bn1.running_var", "backbone.backbone.layer4.1.conv2.weight", "backbone.backbone.layer4.1.bn2.weight", "backbone.backbone.layer4.1.bn2.bias", "backbone.backbone.layer4.1.bn2.running_mean", "backbone.backbone.layer4.1.bn2.running_var", "backbone.backbone.layer4.1.conv3.weight", "backbone.backbone.layer4.1.bn3.weight", "backbone.backbone.layer4.1.bn3.bias", "backbone.backbone.layer4.1.bn3.running_mean", "backbone.backbone.layer4.1.bn3.running_var", "backbone.backbone.layer4.2.conv1.weight", "backbone.backbone.layer4.2.bn1.weight", "backbone.backbone.layer4.2.bn1.bias", "backbone.backbone.layer4.2.bn1.running_mean", "backbone.backbone.layer4.2.bn1.running_var", "backbone.backbone.layer4.2.conv2.weight", "backbone.backbone.layer4.2.bn2.weight", "backbone.backbone.layer4.2.bn2.bias", "backbone.backbone.layer4.2.bn2.running_mean", "backbone.backbone.layer4.2.bn2.running_var", "backbone.backbone.layer4.2.conv3.weight", "backbone.backbone.layer4.2.bn3.weight", "backbone.backbone.layer4.2.bn3.bias", "backbone.backbone.layer4.2.bn3.running_mean", "backbone.backbone.layer4.2.bn3.running_var", "backbone.backbone.fc.weight", "backbone.backbone.fc.bias", "input_proj.weight", "input_proj.bias", "encoder.layers.0.norm1.weight", "encoder.layers.0.norm1.bias", "encoder.layers.0.norm2.weight", "encoder.layers.0.norm2.bias", "encoder.layers.0.self_attn.in_proj_weight", "encoder.layers.0.self_attn.in_proj_bias", "encoder.layers.0.self_attn.out_proj.weight", "encoder.layers.0.self_attn.out_proj.bias", "encoder.layers.0.mlp.linear1.weight", "encoder.layers.0.mlp.linear1.bias", "encoder.layers.0.mlp.linear2.weight", "encoder.layers.0.mlp.linear2.bias", "encoder.layers.1.norm1.weight", "encoder.layers.1.norm1.bias", "encoder.layers.1.norm2.weight", "encoder.layers.1.norm2.bias", "encoder.layers.1.self_attn.in_proj_weight", "encoder.layers.1.self_attn.in_proj_bias", "encoder.layers.1.self_attn.out_proj.weight", "encoder.layers.1.self_attn.out_proj.bias", "encoder.layers.1.mlp.linear1.weight", "encoder.layers.1.mlp.linear1.bias", "encoder.layers.1.mlp.linear2.weight", "encoder.layers.1.mlp.linear2.bias", "encoder.layers.2.norm1.weight", "encoder.layers.2.norm1.bias", "encoder.layers.2.norm2.weight", "encoder.layers.2.norm2.bias", "encoder.layers.2.self_attn.in_proj_weight", "encoder.layers.2.self_attn.in_proj_bias", "encoder.layers.2.self_attn.out_proj.weight", "encoder.layers.2.self_attn.out_proj.bias", "encoder.layers.2.mlp.linear1.weight", "encoder.layers.2.mlp.linear1.bias", "encoder.layers.2.mlp.linear2.weight", "encoder.layers.2.mlp.linear2.bias", "encoder.norm.weight", "encoder.norm.bias", "ope.iterative_adaptation.layers.0.norm1.weight", "ope.iterative_adaptation.layers.0.norm1.bias", "ope.iterative_adaptation.layers.0.norm2.weight", "ope.iterative_adaptation.layers.0.norm2.bias", "ope.iterative_adaptation.layers.0.norm3.weight", "ope.iterative_adaptation.layers.0.norm3.bias", "ope.iterative_adaptation.layers.0.self_attn.in_proj_weight", "ope.iterative_adaptation.layers.0.self_attn.in_proj_bias", "ope.iterative_adaptation.layers.0.self_attn.out_proj.weight", "ope.iterative_adaptation.layers.0.self_attn.out_proj.bias", "ope.iterative_adaptation.layers.0.enc_dec_attn.in_proj_weight", "ope.iterative_adaptation.layers.0.enc_dec_attn.in_proj_bias", "ope.iterative_adaptation.layers.0.enc_dec_attn.out_proj.weight", "ope.iterative_adaptation.layers.0.enc_dec_attn.out_proj.bias", "ope.iterative_adaptation.layers.0.mlp.linear1.weight", "ope.iterative_adaptation.layers.0.mlp.linear1.bias", "ope.iterative_adaptation.layers.0.mlp.linear2.weight", "ope.iterative_adaptation.layers.0.mlp.linear2.bias", "ope.iterative_adaptation.layers.1.norm1.weight", "ope.iterative_adaptation.layers.1.norm1.bias", "ope.iterative_adaptation.layers.1.norm2.weight", "ope.iterative_adaptation.layers.1.norm2.bias", "ope.iterative_adaptation.layers.1.norm3.weight", "ope.iterative_adaptation.layers.1.norm3.bias", "ope.iterative_adaptation.layers.1.self_attn.in_proj_weight", "ope.iterative_adaptation.layers.1.self_attn.in_proj_bias", "ope.iterative_adaptation.layers.1.self_attn.out_proj.weight", "ope.iterative_adaptation.layers.1.self_attn.out_proj.bias", "ope.iterative_adaptation.layers.1.enc_dec_attn.in_proj_weight", "ope.iterative_adaptation.layers.1.enc_dec_attn.in_proj_bias", "ope.iterative_adaptation.layers.1.enc_dec_attn.out_proj.weight", "ope.iterative_adaptation.layers.1.enc_dec_attn.out_proj.bias", "ope.iterative_adaptation.layers.1.mlp.linear1.weight", "ope.iterative_adaptation.layers.1.mlp.linear1.bias", "ope.iterative_adaptation.layers.1.mlp.linear2.weight", "ope.iterative_adaptation.layers.1.mlp.linear2.bias", "ope.iterative_adaptation.layers.2.norm1.weight", "ope.iterative_adaptation.layers.2.norm1.bias", "ope.iterative_adaptation.layers.2.norm2.weight", "ope.iterative_adaptation.layers.2.norm2.bias", "ope.iterative_adaptation.layers.2.norm3.weight", "ope.iterative_adaptation.layers.2.norm3.bias", "ope.iterative_adaptation.layers.2.self_attn.in_proj_weight", "ope.iterative_adaptation.layers.2.self_attn.in_proj_bias", "ope.iterative_adaptation.layers.2.self_attn.out_proj.weight", "ope.iterative_adaptation.layers.2.self_attn.out_proj.bias", "ope.iterative_adaptation.layers.2.enc_dec_attn.in_proj_weight", "ope.iterative_adaptation.layers.2.enc_dec_attn.in_proj_bias", "ope.iterative_adaptation.layers.2.enc_dec_attn.out_proj.weight", "ope.iterative_adaptation.layers.2.enc_dec_attn.out_proj.bias", "ope.iterative_adaptation.layers.2.mlp.linear1.weight", "ope.iterative_adaptation.layers.2.mlp.linear1.bias", "ope.iterative_adaptation.layers.2.mlp.linear2.weight", "ope.iterative_adaptation.layers.2.mlp.linear2.bias", "ope.iterative_adaptation.norm.weight", "ope.iterative_adaptation.norm.bias", "ope.shape_or_objectness.0.weight", "ope.shape_or_objectness.0.bias", "ope.shape_or_objectness.2.weight", "ope.shape_or_objectness.2.bias", "ope.shape_or_objectness.4.weight", "ope.shape_or_objectness.4.bias", "regression_head.regressor.0.layer.0.weight", "regression_head.regressor.0.layer.0.bias", "regression_head.regressor.1.layer.0.weight", "regression_head.regressor.1.layer.0.bias", "regression_head.regressor.2.layer.0.weight", "regression_head.regressor.2.layer.0.bias", "regression_head.regressor.3.weight", "regression_head.regressor.3.bias", "aux_heads.0.regressor.0.layer.0.weight", "aux_heads.0.regressor.0.layer.0.bias", "aux_heads.0.regressor.1.layer.0.weight", "aux_heads.0.regressor.1.layer.0.bias", "aux_heads.0.regressor.2.layer.0.weight", "aux_heads.0.regressor.2.layer.0.bias", "aux_heads.0.regressor.3.weight", "aux_heads.0.regressor.3.bias", "aux_heads.1.regressor.0.layer.0.weight", "aux_heads.1.regressor.0.layer.0.bias", "aux_heads.1.regressor.1.layer.0.weight", "aux_heads.1.regressor.1.layer.0.bias", "aux_heads.1.regressor.2.layer.0.weight", "aux_heads.1.regressor.2.layer.0.bias", "aux_heads.1.regressor.3.weight", "aux_heads.1.regressor.3.bias".
Unexpected key(s) in state_dict: "module.backbone.backbone.conv1.weight", "module.backbone.backbone.bn1.weight", "module.backbone.backbone.bn1.bias", "module.backbone.backbone.bn1.running_mean", "module.backbone.backbone.bn1.running_var", "module.backbone.backbone.layer1.0.conv1.weight", "module.backbone.backbone.layer1.0.bn1.weight", "module.backbone.backbone.layer1.0.bn1.bias", "module.backbone.backbone.layer1.0.bn1.running_mean", "module.backbone.backbone.layer1.0.bn1.running_var", "module.backbone.backbone.layer1.0.conv2.weight", "module.backbone.backbone.layer1.0.bn2.weight", "module.backbone.backbone.layer1.0.bn2.bias", "module.backbone.backbone.layer1.0.bn2.running_mean", "module.backbone.backbone.layer1.0.bn2.running_var", "module.backbone.backbone.layer1.0.conv3.weight", "module.backbone.backbone.layer1.0.bn3.weight", "module.backbone.backbone.layer1.0.bn3.bias", "module.backbone.backbone.layer1.0.bn3.running_mean", "module.backbone.backbone.layer1.0.bn3.running_var", "module.backbone.backbone.layer1.0.downsample.0.weight", "module.backbone.backbone.layer1.0.downsample.1.weight", "module.backbone.backbone.layer1.0.downsample.1.bias", "module.backbone.backbone.layer1.0.downsample.1.running_mean", "module.backbone.backbone.layer1.0.downsample.1.running_var", "module.backbone.backbone.layer1.1.conv1.weight", "module.backbone.backbone.layer1.1.bn1.weight", "module.backbone.backbone.layer1.1.bn1.bias", "module.backbone.backbone.layer1.1.bn1.running_mean", "module.backbone.backbone.layer1.1.bn1.running_var", "module.backbone.backbone.layer1.1.conv2.weight", "module.backbone.backbone.layer1.1.bn2.weight", "module.backbone.backbone.layer1.1.bn2.bias", "module.backbone.backbone.layer1.1.bn2.running_mean", "module.backbone.backbone.layer1.1.bn2.running_var", "module.backbone.backbone.layer1.1.conv3.weight", "module.backbone.backbone.layer1.1.bn3.weight", "module.backbone.backbone.layer1.1.bn3.bias", "module.backbone.backbone.layer1.1.bn3.running_mean", "module.backbone.backbone.layer1.1.bn3.running_var", "module.backbone.backbone.layer1.2.conv1.weight", "module.backbone.backbone.layer1.2.bn1.weight", "module.backbone.backbone.layer1.2.bn1.bias", "module.backbone.backbone.layer1.2.bn1.running_mean", "module.backbone.backbone.layer1.2.bn1.running_var", "module.backbone.backbone.layer1.2.conv2.weight", "module.backbone.backbone.layer1.2.bn2.weight", "module.backbone.backbone.layer1.2.bn2.bias", "module.backbone.backbone.layer1.2.bn2.running_mean", "module.backbone.backbone.layer1.2.bn2.running_var", "module.backbone.backbone.layer1.2.conv3.weight", "module.backbone.backbone.layer1.2.bn3.weight", "module.backbone.backbone.layer1.2.bn3.bias", "module.backbone.backbone.layer1.2.bn3.running_mean", "module.backbone.backbone.layer1.2.bn3.running_var", "module.backbone.backbone.layer2.0.conv1.weight", "module.backbone.backbone.layer2.0.bn1.weight", "module.backbone.backbone.layer2.0.bn1.bias", "module.backbone.backbone.layer2.0.bn1.running_mean", "module.backbone.backbone.layer2.0.bn1.running_var", "module.backbone.backbone.layer2.0.conv2.weight", "module.backbone.backbone.layer2.0.bn2.weight", "module.backbone.backbone.layer2.0.bn2.bias", "module.backbone.backbone.layer2.0.bn2.running_mean", "module.backbone.backbone.layer2.0.bn2.running_var", "module.backbone.backbone.layer2.0.conv3.weight", "module.backbone.backbone.layer2.0.bn3.weight", "module.backbone.backbone.layer2.0.bn3.bias", "module.backbone.backbone.layer2.0.bn3.running_mean", "module.backbone.backbone.layer2.0.bn3.running_var", "module.backbone.backbone.layer2.0.downsample.0.weight", "module.backbone.backbone.layer2.0.downsample.1.weight", "module.backbone.backbone.layer2.0.downsample.1.bias", "module.backbone.backbone.layer2.0.downsample.1.running_mean", "module.backbone.backbone.layer2.0.downsample.1.running_var", "module.backbone.backbone.layer2.1.conv1.weight", "module.backbone.backbone.layer2.1.bn1.weight", "module.backbone.backbone.layer2.1.bn1.bias", "module.backbone.backbone.layer2.1.bn1.running_mean", "module.backbone.backbone.layer2.1.bn1.running_var", "module.backbone.backbone.layer2.1.conv2.weight", "module.backbone.backbone.layer2.1.bn2.weight", "module.backbone.backbone.layer2.1.bn2.bias", "module.backbone.backbone.layer2.1.bn2.running_mean", "module.backbone.backbone.layer2.1.bn2.running_var", "module.backbone.backbone.layer2.1.conv3.weight", "module.backbone.backbone.layer2.1.bn3.weight", "module.backbone.backbone.layer2.1.bn3.bias", "module.backbone.backbone.layer2.1.bn3.running_mean", "module.backbone.backbone.layer2.1.bn3.running_var", "module.backbone.backbone.layer2.2.conv1.weight", "module.backbone.backbone.layer2.2.bn1.weight", "module.backbone.backbone.layer2.2.bn1.bias", "module.backbone.backbone.layer2.2.bn1.running_mean", "module.backbone.backbone.layer2.2.bn1.running_var", "module.backbone.backbone.layer2.2.conv2.weight", "module.backbone.backbone.layer2.2.bn2.weight", "module.backbone.backbone.layer2.2.bn2.bias", "module.backbone.backbone.layer2.2.bn2.running_mean", "module.backbone.backbone.layer2.2.bn2.running_var", "module.backbone.backbone.layer2.2.conv3.weight", "module.backbone.backbone.layer2.2.bn3.weight", "module.backbone.backbone.layer2.2.bn3.bias", "module.backbone.backbone.layer2.2.bn3.running_mean", "module.backbone.backbone.layer2.2.bn3.running_var", "module.backbone.backbone.layer2.3.conv1.weight", "module.backbone.backbone.layer2.3.bn1.weight", "module.backbone.backbone.layer2.3.bn1.bias", "module.backbone.backbone.layer2.3.bn1.running_mean", "module.backbone.backbone.layer2.3.bn1.running_var", "module.backbone.backbone.layer2.3.conv2.weight", "module.backbone.backbone.layer2.3.bn2.weight", "module.backbone.backbone.layer2.3.bn2.bias", "module.backbone.backbone.layer2.3.bn2.running_mean", "module.backbone.backbone.layer2.3.bn2.running_var", "module.backbone.backbone.layer2.3.conv3.weight", "module.backbone.backbone.layer2.3.bn3.weight", "module.backbone.backbone.layer2.3.bn3.bias", "module.backbone.backbone.layer2.3.bn3.running_mean", "module.backbone.backbone.layer2.3.bn3.running_var", "module.backbone.backbone.layer3.0.conv1.weight", "module.backbone.backbone.layer3.0.bn1.weight", "module.backbone.backbone.layer3.0.bn1.bias", "module.backbone.backbone.layer3.0.bn1.running_mean", "module.backbone.backbone.layer3.0.bn1.running_var", "module.backbone.backbone.layer3.0.conv2.weight", "module.backbone.backbone.layer3.0.bn2.weight", "module.backbone.backbone.layer3.0.bn2.bias", "module.backbone.backbone.layer3.0.bn2.running_mean", "module.backbone.backbone.layer3.0.bn2.running_var", "module.backbone.backbone.layer3.0.conv3.weight", "module.backbone.backbone.layer3.0.bn3.weight", "module.backbone.backbone.layer3.0.bn3.bias", "module.backbone.backbone.layer3.0.bn3.running_mean", "module.backbone.backbone.layer3.0.bn3.running_var", "module.backbone.backbone.layer3.0.downsample.0.weight", "module.backbone.backbone.layer3.0.downsample.1.weight", "module.backbone.backbone.layer3.0.downsample.1.bias", "module.backbone.backbone.layer3.0.downsample.1.running_mean", "module.backbone.backbone.layer3.0.downsample.1.running_var", "module.backbone.backbone.layer3.1.conv1.weight", "module.backbone.backbone.layer3.1.bn1.weight", "module.backbone.backbone.layer3.1.bn1.bias", "module.backbone.backbone.layer3.1.bn1.running_mean", "module.backbone.backbone.layer3.1.bn1.running_var", "module.backbone.backbone.layer3.1.conv2.weight", "module.backbone.backbone.layer3.1.bn2.weight", "module.backbone.backbone.layer3.1.bn2.bias", "module.backbone.backbone.layer3.1.bn2.running_mean", "module.backbone.backbone.layer3.1.bn2.running_var", "module.backbone.backbone.layer3.1.conv3.weight", "module.backbone.backbone.layer3.1.bn3.weight", "module.backbone.backbone.layer3.1.bn3.bias", "module.backbone.backbone.layer3.1.bn3.running_mean", "module.backbone.backbone.layer3.1.bn3.running_var", "module.backbone.backbone.layer3.2.conv1.weight", "module.backbone.backbone.layer3.2.bn1.weight", "module.backbone.backbone.layer3.2.bn1.bias", "module.backbone.backbone.layer3.2.bn1.running_mean", "module.backbone.backbone.layer3.2.bn1.running_var", "module.backbone.backbone.layer3.2.conv2.weight", "module.backbone.backbone.layer3.2.bn2.weight", "module.backbone.backbone.layer3.2.bn2.bias", "module.backbone.backbone.layer3.2.bn2.running_mean", "module.backbone.backbone.layer3.2.bn2.running_var", "module.backbone.backbone.layer3.2.conv3.weight", "module.backbone.backbone.layer3.2.bn3.weight", "module.backbone.backbone.layer3.2.bn3.bias", "module.backbone.backbone.layer3.2.bn3.running_mean", "module.backbone.backbone.layer3.2.bn3.running_var", "module.backbone.backbone.layer3.3.conv1.weight", "module.backbone.backbone.layer3.3.bn1.weight", "module.backbone.backbone.layer3.3.bn1.bias", "module.backbone.backbone.layer3.3.bn1.running_mean", "module.backbone.backbone.layer3.3.bn1.running_var", "module.backbone.backbone.layer3.3.conv2.weight", "module.backbone.backbone.layer3.3.bn2.weight", "module.backbone.backbone.layer3.3.bn2.bias", "module.backbone.backbone.layer3.3.bn2.running_mean", "module.backbone.backbone.layer3.3.bn2.running_var", "module.backbone.backbone.layer3.3.conv3.weight", "module.backbone.backbone.layer3.3.bn3.weight", "module.backbone.backbone.layer3.3.bn3.bias", "module.backbone.backbone.layer3.3.bn3.running_mean", "module.backbone.backbone.layer3.3.bn3.running_var", "module.backbone.backbone.layer3.4.conv1.weight", "module.backbone.backbone.layer3.4.bn1.weight", "module.backbone.backbone.layer3.4.bn1.bias", "module.backbone.backbone.layer3.4.bn1.running_mean", "module.backbone.backbone.layer3.4.bn1.running_var", "module.backbone.backbone.layer3.4.conv2.weight", "module.backbone.backbone.layer3.4.bn2.weight", "module.backbone.backbone.layer3.4.bn2.bias", "module.backbone.backbone.layer3.4.bn2.running_mean", "module.backbone.backbone.layer3.4.bn2.running_var", "module.backbone.backbone.layer3.4.conv3.weight", "module.backbone.backbone.layer3.4.bn3.weight", "module.backbone.backbone.layer3.4.bn3.bias", "module.backbone.backbone.layer3.4.bn3.running_mean", "module.backbone.backbone.layer3.4.bn3.running_var", "module.backbone.backbone.layer3.5.conv1.weight", "module.backbone.backbone.layer3.5.bn1.weight", "module.backbone.backbone.layer3.5.bn1.bias", "module.backbone.backbone.layer3.5.bn1.running_mean", "module.backbone.backbone.layer3.5.bn1.running_var", "module.backbone.backbone.layer3.5.conv2.weight", "module.backbone.backbone.layer3.5.bn2.weight", "module.backbone.backbone.layer3.5.bn2.bias", "module.backbone.backbone.layer3.5.bn2.running_mean", "module.backbone.backbone.layer3.5.bn2.running_var", "module.backbone.backbone.layer3.5.conv3.weight", "module.backbone.backbone.layer3.5.bn3.weight", "module.backbone.backbone.layer3.5.bn3.bias", "module.backbone.backbone.layer3.5.bn3.running_mean", "module.backbone.backbone.layer3.5.bn3.running_var", "module.backbone.backbone.layer4.0.conv1.weight", "module.backbone.backbone.layer4.0.bn1.weight", "module.backbone.backbone.layer4.0.bn1.bias", "module.backbone.backbone.layer4.0.bn1.running_mean", "module.backbone.backbone.layer4.0.bn1.running_var", "module.backbone.backbone.layer4.0.conv2.weight", "module.backbone.backbone.layer4.0.bn2.weight", "module.backbone.backbone.layer4.0.bn2.bias", "module.backbone.backbone.layer4.0.bn2.running_mean", "module.backbone.backbone.layer4.0.bn2.running_var", "module.backbone.backbone.layer4.0.conv3.weight", "module.backbone.backbone.layer4.0.bn3.weight", "module.backbone.backbone.layer4.0.bn3.bias", "module.backbone.backbone.layer4.0.bn3.running_mean", "module.backbone.backbone.layer4.0.bn3.running_var", "module.backbone.backbone.layer4.0.downsample.0.weight", "module.backbone.backbone.layer4.0.downsample.1.weight", "module.backbone.backbone.layer4.0.downsample.1.bias", "module.backbone.backbone.layer4.0.downsample.1.running_mean", "module.backbone.backbone.layer4.0.downsample.1.running_var", "module.backbone.backbone.layer4.1.conv1.weight", "module.backbone.backbone.layer4.1.bn1.weight", "module.backbone.backbone.layer4.1.bn1.bias", "module.backbone.backbone.layer4.1.bn1.running_mean", "module.backbone.backbone.layer4.1.bn1.running_var", "module.backbone.backbone.layer4.1.conv2.weight", "module.backbone.backbone.layer4.1.bn2.weight", "module.backbone.backbone.layer4.1.bn2.bias", "module.backbone.backbone.layer4.1.bn2.running_mean", "module.backbone.backbone.layer4.1.bn2.running_var", "module.backbone.backbone.layer4.1.conv3.weight", "module.backbone.backbone.layer4.1.bn3.weight", "module.backbone.backbone.layer4.1.bn3.bias", "module.backbone.backbone.layer4.1.bn3.running_mean", "module.backbone.backbone.layer4.1.bn3.running_var", "module.backbone.backbone.layer4.2.conv1.weight", "module.backbone.backbone.layer4.2.bn1.weight", "module.backbone.backbone.layer4.2.bn1.bias", "module.backbone.backbone.layer4.2.bn1.running_mean", "module.backbone.backbone.layer4.2.bn1.running_var", "module.backbone.backbone.layer4.2.conv2.weight", "module.backbone.backbone.layer4.2.bn2.weight", "module.backbone.backbone.layer4.2.bn2.bias", "module.backbone.backbone.layer4.2.bn2.running_mean", "module.backbone.backbone.layer4.2.bn2.running_var", "module.backbone.backbone.layer4.2.conv3.weight", "module.backbone.backbone.layer4.2.bn3.weight", "module.backbone.backbone.layer4.2.bn3.bias", "module.backbone.backbone.layer4.2.bn3.running_mean", "module.backbone.backbone.layer4.2.bn3.running_var", "module.backbone.backbone.fc.weight", "module.backbone.backbone.fc.bias", "module.input_proj.weight", "module.input_proj.bias", "module.encoder.layers.0.norm1.weight", "module.encoder.layers.0.norm1.bias", "module.encoder.layers.0.norm2.weight", "module.encoder.layers.0.norm2.bias", "module.encoder.layers.0.self_attn.in_proj_weight", "module.encoder.layers.0.self_attn.in_proj_bias", "module.encoder.layers.0.self_attn.out_proj.weight", "module.encoder.layers.0.self_attn.out_proj.bias", "module.encoder.layers.0.mlp.linear1.weight", "module.encoder.layers.0.mlp.linear1.bias", "module.encoder.layers.0.mlp.linear2.weight", "module.encoder.layers.0.mlp.linear2.bias", "module.encoder.layers.1.norm1.weight", "module.encoder.layers.1.norm1.bias", "module.encoder.layers.1.norm2.weight", "module.encoder.layers.1.norm2.bias", "module.encoder.layers.1.self_attn.in_proj_weight", "module.encoder.layers.1.self_attn.in_proj_bias", "module.encoder.layers.1.self_attn.out_proj.weight", "module.encoder.layers.1.self_attn.out_proj.bias", "module.encoder.layers.1.mlp.linear1.weight", "module.encoder.layers.1.mlp.linear1.bias", "module.encoder.layers.1.mlp.linear2.weight", "module.encoder.layers.1.mlp.linear2.bias", "module.encoder.layers.2.norm1.weight", "module.encoder.layers.2.norm1.bias", "module.encoder.layers.2.norm2.weight", "module.encoder.layers.2.norm2.bias", "module.encoder.layers.2.self_attn.in_proj_weight", "module.encoder.layers.2.self_attn.in_proj_bias", "module.encoder.layers.2.self_attn.out_proj.weight", "module.encoder.layers.2.self_attn.out_proj.bias", "module.encoder.layers.2.mlp.linear1.weight", "module.encoder.layers.2.mlp.linear1.bias", "module.encoder.layers.2.mlp.linear2.weight", "module.encoder.layers.2.mlp.linear2.bias", "module.encoder.norm.weight", "module.encoder.norm.bias", "module.ope.iterative_adaptation.layers.0.norm1.weight", "module.ope.iterative_adaptation.layers.0.norm1.bias", "module.ope.iterative_adaptation.layers.0.norm2.weight", "module.ope.iterative_adaptation.layers.0.norm2.bias", "module.ope.iterative_adaptation.layers.0.norm3.weight", "module.ope.iterative_adaptation.layers.0.norm3.bias", "module.ope.iterative_adaptation.layers.0.self_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.0.self_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.0.self_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.0.self_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.0.enc_dec_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.0.enc_dec_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.0.enc_dec_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.0.enc_dec_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.0.mlp.linear1.weight", "module.ope.iterative_adaptation.layers.0.mlp.linear1.bias", "module.ope.iterative_adaptation.layers.0.mlp.linear2.weight", "module.ope.iterative_adaptation.layers.0.mlp.linear2.bias", "module.ope.iterative_adaptation.layers.1.norm1.weight", "module.ope.iterative_adaptation.layers.1.norm1.bias", "module.ope.iterative_adaptation.layers.1.norm2.weight", "module.ope.iterative_adaptation.layers.1.norm2.bias", "module.ope.iterative_adaptation.layers.1.norm3.weight", "module.ope.iterative_adaptation.layers.1.norm3.bias", "module.ope.iterative_adaptation.layers.1.self_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.1.self_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.1.self_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.1.self_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.1.enc_dec_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.1.enc_dec_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.1.enc_dec_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.1.enc_dec_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.1.mlp.linear1.weight", "module.ope.iterative_adaptation.layers.1.mlp.linear1.bias", "module.ope.iterative_adaptation.layers.1.mlp.linear2.weight", "module.ope.iterative_adaptation.layers.1.mlp.linear2.bias", "module.ope.iterative_adaptation.layers.2.norm1.weight", "module.ope.iterative_adaptation.layers.2.norm1.bias", "module.ope.iterative_adaptation.layers.2.norm2.weight", "module.ope.iterative_adaptation.layers.2.norm2.bias", "module.ope.iterative_adaptation.layers.2.norm3.weight", "module.ope.iterative_adaptation.layers.2.norm3.bias", "module.ope.iterative_adaptation.layers.2.self_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.2.self_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.2.self_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.2.self_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.2.enc_dec_attn.in_proj_weight", "module.ope.iterative_adaptation.layers.2.enc_dec_attn.in_proj_bias", "module.ope.iterative_adaptation.layers.2.enc_dec_attn.out_proj.weight", "module.ope.iterative_adaptation.layers.2.enc_dec_attn.out_proj.bias", "module.ope.iterative_adaptation.layers.2.mlp.linear1.weight", "module.ope.iterative_adaptation.layers.2.mlp.linear1.bias", "module.ope.iterative_adaptation.layers.2.mlp.linear2.weight", "module.ope.iterative_adaptation.layers.2.mlp.linear2.bias", "module.ope.iterative_adaptation.norm.weight", "module.ope.iterative_adaptation.norm.bias", "module.regression_head.regressor.0.layer.0.weight", "module.regression_head.regressor.0.layer.0.bias", "module.regression_head.regressor.1.layer.0.weight", "module.regression_head.regressor.1.layer.0.bias", "module.regression_head.regressor.2.layer.0.weight", "module.regression_head.regressor.2.layer.0.bias", "module.regression_head.regressor.3.weight", "module.regression_head.regressor.3.bias", "module.aux_heads.0.regressor.0.layer.0.weight", "module.aux_heads.0.regressor.0.layer.0.bias", "module.aux_heads.0.regressor.1.layer.0.weight", "module.aux_heads.0.regressor.1.layer.0.bias", "module.aux_heads.0.regressor.2.layer.0.weight", "module.aux_heads.0.regressor.2.layer.0.bias", "module.aux_heads.0.regressor.3.weight", "module.aux_heads.0.regressor.3.bias", "module.aux_heads.1.regressor.0.layer.0.weight", "module.aux_heads.1.regressor.0.layer.0.bias", "module.aux_heads.1.regressor.1.layer.0.weight", "module.aux_heads.1.regressor.1.layer.0.bias", "module.aux_heads.1.regressor.2.layer.0.weight", "module.aux_heads.1.regressor.2.layer.0.bias", "module.aux_heads.1.regressor.3.weight", "module.aux_heads.1.regressor.3.bias", "module.ope.shape_or_objectness.0.weight", "module.ope.shape_or_objectness.0.bias", "module.ope.shape_or_objectness.2.weight", "module.ope.shape_or_objectness.2.bias", "module.ope.shape_or_objectness.4.weight", "module.ope.shape_or_objectness.4.bias".
File "/tmp/loca-main/evaluate.py", line 45, in evaluate
model.load_state_dict(state_dict)
File "/tmp/loca-main/evaluate.py", line 95, in
evaluate(args)

[viggykrish]

I ran evaluate.py with the flags shown in the README (in serial mode). Unfortunately, the results didn't improve and in fact look like garbage. I think setting strict=False is not helping with the results eventhough the code ran.

evaluate.py --model_name loca_few_shot --backbone resnet50 --swav_backbone --reduction 8 --image_size 512 --num_enc_layers 3 --num_ope_iterative_steps 3 --emb_dim 256 --num_heads 8 --kernel_dim 3 --num_objects 3 --pre_norm

[viggykrish]

I am able to reproduce the results in the paper using the evaluate script. I was able to get past the errors in my previous comment by modifying the checkpoint load as follows

model = build_model(args)
model = torch.nn.DataParallel(model,device_ids=[gpu])
model.to(device)

state_dict = torch.load(os.path.join(args.model_path, f'{args.model_name}.pt'))['model']
state_dict = {k if 'module.' in k else 'module.' + k: v for k, v in state_dict1.items()}
model.load_state_dict(state_dict)

evaluate.py --model_name loca_few_shot --backbone resnet50 --swav_backbone --reduction 8 --image_size 512 --num_enc_layers 3 --num_ope_iterative_steps 3 --emb_dim 256 --num_heads 8 --kernel_dim 3 --num_objects 3 --pre_norm

Val set MAE: 10.22 RMSE: 32.52
Test set MAE: 10.71 RMSE: 56.28

[djukicn]

@viggykrish I see that the issue for you is a mismatch between state dicts. Can you tell me which version of PyTorch you use?

[5k5000]

I could confirm that the code and weights provided by the authors could achieve the same results in their paper.

[niki-amini-naieni]

These are the results I got for FSC-147 -

From running the evaluation code with the provided checkpoints:
Val MAE: 10.22, Val RMSE 32.52, Test MAE: 10.71, Test RMSE: 56.28

From training the model using the provided training code:
Val MAE: 10.25, Val RMSE: 34.88, Test MAE: 11.43, Test RMSE: 69.84

Most results (except test RMSE) look reasonable compared to the published ones. The test RMSE from running the training code is quite high but still SOTA for class-agnostic counting.

[djukicn]

Thank you @niki-amini-naieni for rerunning the experiments and confirming that the results reproduce. Regarding the higher RMSE, I noticed that all 4 metrics have a bit of variance across runs. I observed this for all competing methods, too. I would say that this is caused by the not-so-nice distribution of object counts in FSC147. Nonetheless, I performed several runs and the results were consistently SOTA (with some runs obviously better than others).