The Function of Pretrained Model

[XiangyuWu]

Hi, Michael!

I am implementing your project and have seen great improvements it makes for image enhancing with the pretrained model local_laplacian. And there are also many other models, such as expertC_comparison, experts_cm1, etc, for which I don't clearly understand their functions through their names.

Is there any introduction for the pretrained models? Any reply would be appreciated.

[mgharbi]

The expert models reproduce the retouches of human photographer as described in the paper. Variations like _cm1 refer to “channel multiplier” which were used to explore performance trade-offs. I included them for completeness, but they are not described in the paper. Mika

…

On Jul 10, 2018, at 9:08 AM, XiangyuWu ***@***.***> wrote: Hi, Michael! I am implementing your project and have seen great improvements it makes for image enhancing with the pretrained model local_laplacian. And there are also many other models, such as expertC_comparison, experts_cm1, etc, for which I don't clearly understand their functions through their names. Is there any introduction for these the pretrained model? Any reply would be appreciated. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[butterl]

@mgharbi

Only hdr model is separately trained with RAW or YUV ?

From the quote, we are not sure if the 16 bit linear input is directly DNG stream from camera or just YUV422 (16 bit YUV ) ?
seems YUV is mostly used in Android camera output

Our Android demo approximates this by undoing the RGB->YUV conversion and
white balance, and tone mapping performed by the Qualcomm SOC.

[mgharbi]

The trained models provided here (except the hdrp model) are trained on RGB images. The hdrp model was trained on a custom YUV format following the HDR+ dataset: https://hdrplusdata.org. This pretrained model is unfortunately no longer supported.

The quote refers to the Android demo with showcase in our supplemental video (not released). For the live viewfinder in this app, the models were trained on the YUV data provided by the Android's Camera2 API.

[XiangyuWu]

@mgharbi Many thanks for your explanation.

[butterl]

@mgharbi thanks for reaching out,
Do we need any preprocess with the DNG file ？ I tried DNG in HDR+ subset of bursts (153 bursts, 37 GiB) with rawpy（return none with imread）

if os.path.splitext(input_path)[1] == ".dng":
    with rawpy.imread(input_path) as raw:
    im_input = raw.postprocess() # RGB here 

('black_level_per_channel:', [0, 0, 0, 0])
('color_matrix:', array([[ 1.1273876 , -0.43464163,  0.307254  ,  0.        ],
       [-0.13817634,  1.0466146 ,  0.09156165,  0.        ],
       [-0.01016605, -0.32962185,  1.339788  ,  0.        ]],
      dtype=float32))
('rgb_xyz_matrix:', array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]], dtype=float32))
('tone_curve:', array([    0,     1,     2, ..., 65533, 65534, 65535], dtype=uint16))

but still got a strange color output（more green than normal）

Will train a new HDR model with the dataset once the I tried get normal color

By the way, could you share your inference time on Android (Halide with opencl to run on gpu)?
I see >4s inference time with the pretrained local laplace model，and 2s with RGB to [0,1] float and 3s from model outputs to save to png with my test

[peppaseven]

To butterl, did you find right training method? and how about your performance with android opencl? whether drop this or have got expected result?

[butterl]

@peppaseven the old problem is because the input not match the pretrained model. And the PSNR is 21.4db at most （that‘s the biggest problem, you could see my result here）. And I didn't try opencl on phone yet ,and if it could not rich a 28db+ psnr, that’s not time to port to phone

[peppaseven]

I also want to know whether someone can duplicate same result as mgharbi stated, I am a freshman for deep learning field,just try to find a model to enhance low resolution video and pictures. if can't achieve 28 or higher PSNR, I have to drop training plan...