Feature/na 194

[SimaoBolota-MetaCell]

This PR includes tests for the Gain functionality of a Neuroglancer viewer. The tests are designed to verify the effect of the volume rendering gain parameter on the brightness of the rendered image.

The tests are divided into three parts:

• test_no_gain(webdriver): This test sets up a Neuroglancer viewer with a volume rendering gain of 0, waits for the viewer to load, takes a screenshot, and performs some basic validations on the screenshot. It also calculates the average pixel value of the screenshot for later comparison.

• test_gain(webdriver): This test is similar to test_no_gain, but it sets the volume rendering gain to 10. It also takes a screenshot, performs the same validations, and calculates the average pixel value.

• test_gain_difference(): This test compares the average pixel values calculated in test_no_gain and test_gain. It asserts that the average pixel value of the screenshot taken with a gain of 10 is greater than the average pixel value of the screenshot taken with a gain of 0. This verifies that increasing the volume rendering gain increases the brightness of the rendered image.

These tests help ensure that the volume rendering gain parameter is working as expected and that the viewer is able to capture the effect of this parameter on the rendered image.

To run the test locally: PYTHONPATH="~/.local/bin" python -m tox -e chrome -- python/tests/gain_test.py

[seankmartin]

From what I can tell, s.postion = blah is what is causing the zoom look. I guess s.position isn't always working

[SimaoBolota-MetaCell]

Hey @seankmartin I changed the data source as you requested.
However I need help on two things

1. I tried changing the code to not add a new image layer for each of the tests but was unsuccessful. I kind of left it half way but it still is adding an image layer for each test. Could you have a look?
2. When the gain is increased to 10 it gets brighter, however the cube kind of moves position or it doesn't load all chunks, Not sure exactly what is going on. When I disable and enable again Volume rendering it then appears as it should. (video attached)

Screen.Recording.2024-02-29.at.20.10.55.mov

[SimaoBolota-MetaCell]

@seankmartin I found a different way, that solves issue number 1 of the previous comment.
But I can't get the data to load for some reason.
(it's a WIP to see if I could get some of the problems fixed)

Here's the code for it, which I adapted from the shader.controls.py

# @license
# Copyright 2021 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tests that shader control parameters can be specified from Python."""

import neuroglancer
import numpy as np
import neuroglancer
import numpy as np
from time import sleep
import pytest
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from PIL import Image
import io

def add_render_panel(side="left", row=0, col=0):
    return neuroglancer.LayerSidePanelState(
        side=side,
        col=col,
        row=row,
        tab="rendering",
        tabs=["rendering", "source"],
    )
    
def test_gain(webdriver, **kwargs):
    with webdriver.viewer.txn() as s:
        shape = (50,) * 3 
        dimensions = neuroglancer.CoordinateSpace(names=["x", "y", "z"], units="nm", scales=[400, 400, 400])
        data = np.full(shape=shape, fill_value=255, dtype=np.uint8)
        local_volume = neuroglancer.LocalVolume(data, dimensions)
        s.layers.append(
            name="image",
            layer=neuroglancer.ImageLayer(
                source=local_volume,
                volume_rendering=True,
                tool_bindings={
                     "A": neuroglancer.VolumeRenderingGainTool(),
                },
                panels=[add_render_panel()],
                **kwargs,
                ),
            shader="""
void main() {
    emitRGBA(vec4(1.0, 1.0, 1.0, 0.001));
    }
    """,

        )
        s.layout = "3d"

    gain = webdriver.viewer.state.layers["image"].volumeRenderingGain
    print(gain)
    assert gain == 1

    def expect_gain():
        sleep(3)
        webdriver.sync()
        WebDriverWait(webdriver.driver, 60).until(
            lambda driver: driver.execute_script('return document.readyState') == 'complete'
    )
    #     screenshot = webdriver.viewer.screenshot(size=[10, 10]).screenshot
    #     np.testing.assert_array_equal(
    #         screenshot.image_pixels,
    #         np.tile(np.array(color, dtype=np.uint8), (10, 10, 1)),
    #     )

    with webdriver.viewer.txn() as s:
        s.layers["image"].volumeRenderingGain =0
    sleep(4)
    with webdriver.viewer.txn() as s:
        s.layers["image"].volumeRenderingGain =10
    
    expect_gain()

[seankmartin]

Hey @SimaoBolota-MetaCell - from what I can tell at the moment, it's actually the screenshot() function hanging. That's part of the viewer, not just for testing. I'll check later if there is some bug with taking screenshots with volume rendering mode on in general. Perhaps best to revisit this at that point!

[seankmartin]

For reference, the stripped back version:

import neuroglancer
import numpy as np
import pytest


def test_gain(webdriver):
    with webdriver.viewer.txn() as s:
        shape = (50,) * 3
        dimensions = neuroglancer.CoordinateSpace(
            names=["x", "y", "z"], units="nm", scales=[400, 400, 400]
        )
        s.dimensions = dimensions
        data = np.full(shape=shape, fill_value=255, dtype=np.uint8)
        local_volume = neuroglancer.LocalVolume(data, dimensions)
        s.layers.append(
            name="image",
            layer=neuroglancer.ImageLayer(
                source=local_volume,
                volume_rendering=True,
                volume_rendering_depth_samples=400,
            ),
            visible=True,
            shader="""
void main() {
    emitRGBA(vec4(1.0, 1.0, 1.0, 0.001));
    }
    """,
        )
        s.layout = "3d"
        s.show_axis_lines = False

    gain = webdriver.viewer.state.layers["image"].volumeRenderingGain
    assert gain == 0

    # webdriver.sync()
    screenshot1 = webdriver.viewer.screenshot(size=[300, 300]).screenshot
    with webdriver.viewer.txn() as s:
        s.layers["image"].volumeRenderingGain = 10
    # webdriver.sync()
    screenshot2 = webdriver.viewer.screenshot(size=[300, 300]).screenshot
    assert np.all(screenshot1.image_pixels < screenshot2.image_pixels)

    with open("p1", "wb") as f:
        f.write(screenshot1.image)

    with open("p2", "wb") as f:
        f.write(screenshot2.image)

[seankmartin]

I've been able to produce a hang on volume rendering screenshot outside of tests but nothing reproducible. I do have a code snippet that never seems to hang, which might be useful for a start:

import argparse
import webbrowser
import neuroglancer
import numpy as np
import neuroglancer.cli


def add_image_layer(state):
    shape = (2500, 2500, 250)
    data = np.full(shape=shape, fill_value=255, dtype="uint8")
    dimensions = neuroglancer.CoordinateSpace(
        names=["x", "y", "z"], units="nm", scales=[40, 40, 400]
    )
    local_volume = neuroglancer.LocalVolume(data, dimensions)
    state.layers["image"] = neuroglancer.ImageLayer(
        source=local_volume, volume_rendering_mode="ON", shader=get_shader()
    )


def get_shader():
    return """
#uicontrol invlerp normalized(range=[0,255], clamp=true)
#uicontrol vec3 color color(default="white")
void main() {
    float val = normalized();
    emitRGBA(vec4(color, val));
    }
"""


def launch_nglancer():
    ap = argparse.ArgumentParser()
    neuroglancer.cli.add_server_arguments(ap)
    args = ap.parse_args()
    neuroglancer.cli.handle_server_arguments(args)
    viewer = neuroglancer.Viewer()
    return viewer


if __name__ == "__main__":
    viewer = launch_nglancer()
    with viewer.txn() as s:
        add_image_layer(s)
    webbrowser.open_new(viewer.get_viewer_url())

    print(viewer.state.layers["image"].volumeRenderingGain)

    s1 = viewer.screenshot(size=[1000, 1000])
    print("shot 1 taken")

    with viewer.txn() as s:
        s.layers["image"].volume_rendering_gain = 1.0
    s2 = viewer.screenshot(size=[1000, 1000])
    print(viewer.state.layers["image"].volumeRenderingGain)
    print("shot 2 taken")

    with open("p1.png", "wb") as f:
        f.write(s1.screenshot.image)

    with open("p2.png", "wb") as f:
        f.write(s2.screenshot.image)

[seankmartin]

Adding state.layout = "3d" causes a hang after taking one screenshot. Looks like one screenshot might work fine though, so we might need to adopt: One layer, one screenshot for the moment. Sorry about that

[seankmartin]

Another example to keep things in the one place (fails on the sync call with hang)

import neuroglancer
import numpy as np


def test_max_projection(webdriver):
    def create_volume():
        shape = (10, 40, 50)

        # Create the volume
        volume = np.zeros(shape)

        # Fill each row across the last dimension with 90 random data points and 10 data points that are 1s
        rng = np.random.default_rng()
        for i in range(shape[0]):
            for j in range(shape[1]):
                random_indices = rng.choice(shape[2], size=40, replace=False)
                volume[i, j, random_indices] = rng.random(40)
                volume[i, j, ~np.isin(np.arange(shape[2]), random_indices)] = 1
        return volume

    def get_shader():
        return """
#uicontrol invlerp normalized(range=[0,1])
void main() {
    emitGrayscale(normalized());
}
"""

    with webdriver.viewer.txn() as s:
        s.dimensions = neuroglancer.CoordinateSpace(
            names=["x", "y", "z"], units="nm", scales=[1, 1, 1]
        )
        s.layers["image"] = neuroglancer.ImageLayer(
            source=neuroglancer.LocalVolume(create_volume(), dimensions=s.dimensions),
            volume_rendering_mode="MAX",
            shader=get_shader(),
        )
        #s.layout = "3d"
        s.show_axis_lines = False

    assert webdriver.viewer.state.layers["image"].volume_rendering_mode == "MAX"
    webdriver.sync()
    screenshot = webdriver.viewer.screenshot(size=[100, 100]).screenshot
    np.testing.assert_array_equal(
        screenshot.image_pixels,
        np.tile(np.array([255, 255, 255, 255], dtype=np.uint8), (100, 100, 1)),
    )

[SimaoBolota-MetaCell]

@seankmartin I changed the order of the tests, so now it starts by having a Gain of 10 and then decreases its value to 0.
It seems that the hanging isasue is more evident in this case, and I still see it happening. I already merged the most recent changes in feature/transfer-function-shader-widget into my branch and the hanging issue still happens.

Screen.Recording.2024-05-02.at.12.19.59.mov

[seankmartin]

So I'm pretty sure the hanging in this case is due to not all the chunks loading in @SimaoBolota-MetaCell . Do you think the same? The old hang was due to webdriver.sync(). But this hang is from waiting for all the chunks to load in 8/8 from what I can see

[SimaoBolota-MetaCell]

@seankmartin That's what i thought initially but if you remove that check by commenting lines 116-118 , so it won't wait for the 8/8 to be reached it still hangs, see below

Screen.Recording.2024-05-03.at.16.12.53.mov

Also tried by commenting lines 112-118 to see if the hang issue was the page not being fully loaded, but it still happens. See below

Screen.Recording.2024-05-03.at.16.14.00.mov

However @seankmartin , if I comment out the second shared_webdriver.sync() in line 110, it seems to be working 😄 . See below

Screen.Recording.2024-05-03.at.16.22.06.mov

I had commented out the state.layout 3D line because in 3D only it seems that the no gain data moves for some reason (eventhough the tests still pass because the pixel value is still bigger), while for when all 3D and 2D are displayed it doesn't happen.

Screen.Recording.2024-05-03.at.16.23.18.mov

Could you try both of them out on your end? I am pushing a commit with the line for the webdriver sync already commented out
If it works on your end then please let me know if you prefer to have the 3D shown even with that displacement in the image or with both 3D and 2D

Thanks!

[seankmartin]

Hey @SimaoBolota-MetaCell, thanks for finding a configuration that works! It seems a little odd that the second sync can't be there (nothing to do with the code here, I just mean in general)!

I've tried to cut down some things to check what might be the cause of the second sync producing a hang. You can see that here https://github.com/MetaCell/neuroglancer/tree/minimal-na-194

[SimaoBolota-MetaCell]

Hey @seankmartin I checked your branch https://github.com/MetaCell/neuroglancer/tree/minimal-na-194 and I see your point because it makes it way smaller, however it seems t get stuck for me when running your version

Screen.Recording.2024-05-14.at.18.10.22.mov

I've tried to comment parts of the code to try and pinpoint why it hangs exactly but can't figure it out, it's not the webdriver.sync or the state. parameters

I tried the version I have on my branch and it still works for me

Screen.Recording.2024-05-14.at.18.19.49.mov

should I try and clean that one up? Like removing sleeps and all that so its at least faster (even if not smaller xD ) - looks like this

Screen.Recording.2024-05-14.at.18.21.11.mov

[SimaoBolota-MetaCell]

Hey @seankmartin I found a solution that seems to kind of fix the issue, for me at least i+sometimes it hangs now and sometimes doesn't, compared to always hanging. I just added this:
1654cdc

Could you check on your side to see what happens?

On the other hand, should the working test be merged? The one in this PR ?

[seankmartin]

The problem @SimaoBolota-MetaCell is that this PR doesn't always work on my machine, and since the timeouts are very long, it's not quick to rerun if the test was flakey

Sorry! I know this one has been hanging around a while