Support for rendering with differentiable camera to_world transform

[sjobeek]

Hello! Thank you for the very interesting research and implementation.

While trying to evaluate the robustness of this method to imperfect camera parameters, I noticed that the to_world camera transforms are not differentiable. I'm very interested in support for this, for the purpose of fine-tuning camera parameters near the end of the SDF optimization.

The Mitsuba3 library is working on an implementation of differentiable to_world camera parameter in the following issue, and I'm rather hopeful that these changes could allow for the fine-tuning of camera parameters:
mitsuba-renderer/mitsuba3#251

Do you expect you expect the custom sdf_direct_reparam integrator from this library to work with the changes Mitsuba3 folks are implementing?