Question about the output of the coarse registration

[leblond14u]

Hi,
Thanks for your work,

I have two Gaussian clouds that are already aligned and I'm testing the coarse registration method with them.
However the result is quite disappointing and giving me a transform result that is far from being an identity matrix :

[[-0.07166722 -0.5279686   0.00634995  4.515895  ]
 [ 0.52784467 -0.07148125  0.01406345 -1.9595625 ]
 [-0.01308281  0.00818184  0.53262484 -0.06575554]
 [ 0.          0.          0.          1.        ]]

On top of this the scales are also off : ref_adjust_scale = 0.09 and src_adjust_scale = 0.09 .
I produced those gaussian clouds with spherical harmonics of degree 0, is it the cause of my poor results ?

Also, are the colored points in the saved demo_outputs clouds the points used by the superPoint matcher ?

Thanks in advance,
Best regards,

Hugo

[Jiahao620]

Hi Hugo,
Could you please provide your original file?
The colored points in the saved demo_outputs clouds are the sampling points for coarse registration.
Best,
Jiahao

[leblond14u]

Hi,

I can't provide the files directly but they are comparable to scenes from Kitti.
Fairly large scale so, do you think my problems could come from the size of my scenes ?

I seen on the geotransformer repo that they apparently had weights available for Kitti.
Do you know how to adapt your code for kitti like scenes ?
Do you have weights for larger scenes that you could share ?

thanks in advance,
Best,
Hugo

[Jiahao620]

Maybe the problem lies in the overlap area of ​​the two scenes? How much overlap area between the two scenes in your data?

[leblond14u]

The area of overlap is quite large between the two clouds, half of the first and a good 1/4th of the second ...
Is your method already supposed to be working on large-scale scenarios ?
Does the fact I'm not using any spherical harmonics could cause a problem as well ?

Have a good day,
Best,

[Jiahao620]

Maybe that is the problem of the domain gap between your dataset and ScanNet-GSReg. I dont think the reason is not using any spherical harmonics. It would be even better if you could provide me with your data so that I can help you find the reason.

[leblond14u]

Hi,

I tested on an interior scene from TUM-RGBD with 60% of identical frames. I transformed one half of the data in translation (part B) yet I still have bad coarse registration results...
Is there a way to be sure the version of the code I have is properly functioning with a ground truth benchmark, like estimating a know translation for instance ?
I successfully downloaded the ScanNet-GSReg first batch of sequences, is there an associated GT transform I can use to ensure that my code has the proper behavior ?

Thanks in advance,
Best,

Hugo

[Jiahao620]

You can evaluate this model on the sample data (https://1drv.ms/f/c/31078419ab45666e/EiUYfzkosRpPv2zhu38kLbwBIR-GA2IvvCyMYc3VGWdFhA?e=o2MSW2)
The Identity matrix is the GT transform of the ScanNet-GSReg train dataset and we performed data augmentation at each iteration while training.
The ScanNet-GSReg first batch of sequences might be part of the ScanNet-GSReg training dataset.

[leblond14u]

Thanks for your quick answer !
I'm gonna test this, I suspect having an issue with SuperPoint and my GPU which would lead to a bad estimation ...
It could also explain my previous issues with the coarse estimation.
Do you know by any chance if there's a way to use your pipeline on cpu ?

Have a great day,
Best regards,

[Jiahao620]

You should modify the code in L142 of demo.py

model = create_model(cfg).cuda()

and the code in L148-L151 of demo.py

data_dict = to_cuda(data_dict)
output_dict = model(data_dict)
data_dict = release_cuda(data_dict)
output_dict = release_cuda(output_dict)

You can put the model and data on the cpu instead of gpu.
I hope this is helpful to you.

[leblond14u]

Hi,

Thanks for your answer.
It doesn't seem to directly work like this for me.
Down the line things are expected to be on the GPU in geotransformer modules even if the model and data are placed on cpu.
Is there something else I need to modify for this to work ?

Also this is the outputs I'm having on my computer on your test data :

Filtered opacity shape: (5388524,)
Filtered x shape: (5388524,), y shape: (5388524,), z shape: (5388524,)
Filtered points index count: 1060391
f_dc_0 shape: (5388524,), f_dc_1 shape: (5388524,), f_dc_2 shape: (5388524,)
Filtered opacity shape: (5918704,)
Filtered x shape: (5918704,), y shape: (5918704,), z shape: (5918704,)
Filtered points index count: 1108566
f_dc_0 shape: (5918704,), f_dc_1 shape: (5918704,), f_dc_2 shape: (5918704,)

ref_adjust_scale, src_adjust_scale : 
0.12782205014259093 0.14014728087164985
estimated_transform & estimated_transform_scale : 
[[-0.2159141  -0.1426663   0.57406247 -0.16285795]
 [-0.12934779 -0.584933   -0.19401762  0.8761975 ]
 [ 0.5772093  -0.18444514  0.17125925  0.14018455]
 [ 0.          0.          0.          1.        ]]
[[-0.23673359 -0.15642287  0.6294164  -1.0533797 ]
 [-0.14182012 -0.64133507 -0.21272576  2.677458  ]
 [ 0.6328666  -0.20223026  0.1877729   3.326653  ]
 [ 0.          0.          0.          1.        ]]

And those are the results obtained from another computer :

estimated_transform & estimated_transform_scale : 
0.1278220327633109 0.14014743156693782
[[ 0.33312228 -0.13107766  0.56637645 -0.14853545]
 [ 0.10456497  0.65563565  0.09023376  0.01307569]
 [-0.5718652   0.0435271   0.34642413  0.14266431]
 [ 0.          0.          0.          1.        ]]
estimated_transform & estimated_transform_scale : 
[[ 0.36524406 -0.14371699  0.62099004 -0.4264177 ]
 [ 0.11464778  0.71885616  0.09893467 -0.63582695]
 [-0.627008    0.04772425  0.3798285   3.151507  ]
 [ 0.          0.          0.          1.        ]]

It seems there is an issue somewhere ...

Thanks in advance,

Hugo

[Jiahao620]

In addition, you should revise the code in the function point_to_node_partition:

GaussReg/experiments/geotransformer.gaussian_splatting.indoor/model.py

Line 99 in aed5e66

_, ref_node_masks, ref_node_knn_indices, ref_node_knn_masks = point_to_node_partition(

, which includes the operation of transferring data to the GPU:

GaussReg/geotransformer/modules/ops/pointcloud_partition.py

Line 87 in aed5e66

node_masks = torch.zeros(nodes.shape[0], dtype=torch.bool).cuda() # (M,)

.
My apologies for overlooking this. I will update this part as well.

I'm also confused by your results. Has anyone else encountered this issue?