hole fill, and then align

[timprepscius]

Hello.

In c++.
I have a a rs2::frameset, it contains at least a color and a depth.

I need to hole fill it.
and then align the color to depth.

I want the aligned color to have the entire image, not just the parts where there are depth values.

How do you do this with your API?

[MartyG-RealSense]

Hi @timprepscius #2157 (comment) has an example of a C++ script that demonstrates post-processing filters (including hole filling) and depth-to-color alignment.

If you change the line rs2_stream align_to = RS2_STREAM_COLOR; to rs2_stream align_to = RS2_STREAM_DEPTH; then color-to-depth alignment should be performed instead of depth-to-color. This will help to ensure that the aligned image stretches to fill the screen space and does not have a black border around its edge where some of the detail is cut off.

[timprepscius]

Can you please check that example to make sure it

1. FILTERS FIRST!! - not second
2. then aligns.

It seems to
ALIGN and then FILTER.

Could you please put just the lines of code, given a frame set and an align and a hole filters, that FILTERS FIRST, and aligns second.
Variables can be named "frameset" "aligner" and "hole_filler"

When you are reading this, you may think to yourself, "what is with this guy and the capital letters, just reverse the lines of code." But, as I understand your library, this is not possible.

[MartyG-RealSense]

Placing post-processing filters first and alignment second is a recommendation by Intel but not compulsory, and some have found that their particular project works better with alignment first and filters second.

You are correct that the linked-to example code applies alignment first and filters second.

I recommend trying the example code to see how well it performs for you.

[timprepscius]

in:

rs2::processing_block frame_processor(
[&](rs2::frameset data, // Input frameset (from the pipeline)
rs2::frame_source& source)

where does rs2::frame_source come from?

[timprepscius]

If anyone is reading this in the future, here is some code which does the depth filling first, so you can have a mostly ok aligned color image, because, the rs2_project_color_pixel_to_depth_pixel does not work. It really seems like it should, but it doesn't. I still haven't figured out why. Their algorithm is basically, find the ray that pixel is on, and then traverse that ray finding the best fit. It doesn't work. Don't bother, you have to align the whole image.

Note, if you are me and you have been reading everything you can to try to get a valid depth for a color point, it seems the D415 has a mode where you can get a color image from the "left infrared" sensor and it will be aligned without any magic, so maybe go buy one of those cameras, instead of the D43x - D415 has a rolling shutter, so, maybe this will cause other problems.

In the mean time, also, if you are me, and you are flummoxed by this libraries need to make everything hard to do, specifically revolving around creating framesets or modifying them, I still have not figured out how to create a frameset. I think, somehow you can get the rs2_frame_source from an existing frameset and then use this, but it's quite a few layers inside of this onion, so - if you figure out how, please post to this message.

In the meantime, here is some code:

	rs2::processing_block frame_processor(
		[&](rs2::frameset data, rs2::frame_source& source)
		{
			std::vector<rs2::frame> frames;
			for (auto frame: data)
			{
				if (frame.get_profile().stream_type() != RS2_STREAM_DEPTH)
				{
					frames.push_back(frame);
				}
				else
				{
					rs2::hole_filling_filter hole_filling_filter;
					auto filled = hole_filling_filter.process(frame);
					
					frames.push_back(filled);
				}
			}
			
			rs2::frameset combined = source.allocate_composite_frame(frames);
			
			source.frame_ready(combined);
		}
	);

	rs2::frame_queue filtered_data;
	frame_processor >> filtered_data; // can you believe this line of code here, i mean, come on.

	frame_processor.invoke(frameSet_);
	rs2::frameset frameSet = filtered_data.wait_for_frame();

[MartyG-RealSense]

Thanks so much @timprepscius for sharing your code with the RealSense community!

The 'RGB color from left infrared sensor' mode is supported on the D405, D415 and D45x models (D455, D455f, D456, D457) but is not supported on the D43x models (D435, D435i, D435if) because the IR sensors on these models are monochrome. More information about this RGB mode can be found at the link below.

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#use-the-left-color-camera

#5847 has C++ information about creating a custom frameset.