Bug: Rendering becomes corrupted after a certain amount of frames

[elias-Mimetrik]

Checklist

• I have searched for similar issues.
• For Python issues, I have tested with the latest development wheel.
• I have checked the release documentation and the latest documentation (for main branch).

Describe the issue

I am creating an application where a renderer is created and destoyed 1000s of times. A single renderer is created inside a loop and destroyed at the end of the loop.

The problem is that after some 18700 iterations, rendering becomes corrupted and it turns black.

The rendered images move from this:

To (after 18700 iterations or less than 15 minutes):

I managed to replicate this issue on the Offscreenrendering.cpp example. Please run the modified example below:

Steps to reproduce the bug

// Modified offscreenrendering example such that the bug is reproduced.
#include "open3d/Open3D.h"
#include "open3d/visualization/rendering/Camera.h"
#include "open3d/visualization/rendering/filament/FilamentEngine.h"
#include "open3d/visualization/rendering/filament/FilamentRenderer.h"

using namespace open3d;
using namespace open3d::visualization::gui;
using namespace open3d::visualization::rendering;

// Headless rendering requires Open3D to be compiled with OSMesa support.
// Add -DENABLE_HEADLESS_RENDERING=ON when you run CMake.
static const bool kUseHeadless = false;

static const std::string kOutputFilename = "offscreen.png";

int main(int argc, const char *argv[]) {
    const int width = 640;
    const int height = 480;

    auto &app = Application::GetInstance();
    app.Initialize("/usr/local/share/resources/");

    if (kUseHeadless) {
        EngineInstance::EnableHeadless();
    }

    // This example demonstrates rendering to an image without a window.
    // If you want to render to an image from within a window you should use
    // scene->GetScene()->RenderToImage() instead.
    int counter = 0;
    while(true) {
        auto *renderer =
        new FilamentRenderer(EngineInstance::GetInstance(), width, height,
                                EngineInstance::GetResourceManager());
        auto *scene = new Open3DScene(*renderer);

        MaterialRecord mat;
        mat.shader = "defaultLit";
        auto torus = open3d::geometry::TriangleMesh::CreateTorus();
        torus->ComputeVertexNormals();
        torus->PaintUniformColor({1.0f, 1.0f, 0.0f});
        scene->AddGeometry("torus", torus.get(), mat);
        scene->ShowAxes(true);

        scene->GetCamera()->SetProjection(60.0f, float(width) / float(height), 0.1f,
                                        10.0f, Camera::FovType::Vertical);
        scene->GetCamera()->LookAt({0.0f, 0.0f, 0.0f}, {3.0f, 3.0f, 3.0f},
                                {0.0f, 1.0f, 0.0f});

        auto img = app.RenderToImage(*renderer, scene->GetView(), scene->GetScene(), width, height);
        if (counter % 100 == 0) {
            io::WriteImage("./out/" + std::to_string(counter) + ".png", *img);
        }
        counter++;

        // We manually delete these because Filament requires that things get
        // destructed in the right order.
        delete scene;
        delete renderer;
    }

    // Cleanup Filament. Normally this is done by app.Run(), but since we are
    // not using that we need to do it ourselves.
    app.OnTerminate();
}

Error message

No error messages.

Expected behavior

The rendering should not turn to black; instead it turns to black every time on exactly the same iteration.

A standalone example is pasted above. It is a replica of Offscreenrendering example but repeated 1000s of times. I run it on my machine for 15 minutes until the rendered images start to become corrupted.

Open3D, Python and System information

- Operating system: `Ubuntu 24.04`
- Python version: `3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0]`
- Open3D version: `0.19 build from source with and without CUDA; with SHARED_LIBS=ON and OFF`. Also tried `0.18 with and without CUDA precompiled packages.`
- System architecture: `x86`
- Is this a remote workstation?: no
- How did you install Open3D?: 0.19 build from source; also tried precompiled package for 0.18.
- Compiler version (if built from source): `gcc 13.3.0`

Additional information

No response

[elias-Mimetrik]

UPDATE:

The bug exists even when a single renderer instance is used. After rendering 130k frames, the rendered image becomes blank:

Here is the code to reproduce this bug (Offscreenrendering.cpp example with minor edits):

#include "open3d/Open3D.h"
#include "open3d/visualization/rendering/Camera.h"
#include "open3d/visualization/rendering/filament/FilamentEngine.h"
#include "open3d/visualization/rendering/filament/FilamentRenderer.h"

using namespace open3d;
using namespace open3d::visualization::gui;
using namespace open3d::visualization::rendering;

static const bool kUseHeadless = false;

int main(int argc, const char *argv[]) {
    const int width = 1280;
    const int height = 800;

    auto &app = Application::GetInstance();
    app.Initialize("/usr/local/share/resources/");

    if (kUseHeadless) {
        EngineInstance::EnableHeadless();
    }

    auto *renderer =
    new FilamentRenderer(EngineInstance::GetInstance(), width, height,
                            EngineInstance::GetResourceManager());
    auto *scene = new Open3DScene(*renderer);

    MaterialRecord mat;
    mat.shader = "defaultLit";
    auto torus = open3d::geometry::TriangleMesh::CreateTorus();
    torus->ComputeVertexNormals();
    torus->PaintUniformColor({1.0f, 1.0f, 0.0f});
    scene->AddGeometry("torus", torus.get(), mat);
    scene->ShowAxes(true);

    scene->GetCamera()->SetProjection(60.0f, float(width) / float(height), 0.1f,
                                    10.0f, Camera::FovType::Vertical);
    scene->GetCamera()->LookAt({0.0f, 0.0f, 0.0f}, {3.0f, 3.0f, 3.0f},
                            {0.0f, 1.0f, 0.0f});

    int counter = 0;
    while(true) {
        auto img = app.RenderToImage(*renderer, scene->GetView(), scene->GetScene(), width, height);
        if (counter % 100 == 0) {
            io::WriteImage("./out/" + std::to_string(counter) + ".png", *img);
        }
        counter++;
    }

    delete scene;
    delete renderer;
    app.OnTerminate();
}

[patrikhuber]

I can reproduce this issue on Windows with MSVC, with an older as well as newer (0.19) Open3D version.

Very interestingly the rendering becomes blank or messed up in exactly the same frame on my Windows machine as on @elias-Mimetrik's Linux setup, I find that quite remarkable, and maybe that points to something quite deterministic like an overflow issue that always happens at the same time (and cross-platform). As it always seems to fail after the same iteration too. So potentially not some random memory corruption issue. (Though it seems, in addition to the issue here, the renderer also doesn't properly clean up after itself, as I've discovered in #7132).