Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to enable i.MX vpuenc_h264 ? #296

Open
l4es opened this issue Sep 18, 2021 · 30 comments
Open

How to enable i.MX vpuenc_h264 ? #296

l4es opened this issue Sep 18, 2021 · 30 comments

Comments

@l4es
Copy link

l4es commented Sep 18, 2021

Dear All,

I'm integrating gstreamer-imx for the iMX8M Plus in Buildroot borrowing package implemented here : https://github.com/boundarydevices/buildroot-external-boundary/tree/master/package/gst1-imx-v2

When verifying the available hardware encoder/decoder plugins, I cannot find vpuenc_h264 :

root@imx-camera:~# dmesg | grep hantro
[ 2.063741] hantrodec 0 : module inserted. Major = 236
[ 2.064220] hantrodec 1 : module inserted. Major = 236
[ 2.065434] hantroenc: HW at base <0000000038320000> with ID <0x80006200>

root@imx-camera:~# gst-inspect-1.0 | grep imx
libav: avdec_simbiosis_imx: libav Simbiosis Interactive IMX Video decoder
imxvpu: imxvpudec_sspark: i.MX VPU Sorenson Spark video decoder
imxvpu: imxvpudec_divx5: i.MX VPU DivX 5 & 6 video decoder
imxvpu: imxvpudec_divx4: i.MX VPU DivX 4 video decoder
imxvpu: imxvpudec_divx3: i.MX VPU DivX 3 video decoder
imxvpu: imxvpudec_cavs: i.MX VPU AVS (Audio and Video Coding Standard) video decoder
imxvpu: imxvpudec_vp6: i.MX VPU VP6 video decoder
imxvpu: imxvpudec_vc1: i.MX VPU VC-1 advanced profile video decoder
imxvpu: imxvpudec_wmv3: i.MX VPU WMV3 / Window Media Video 9 / VC-1 simple profile video decoder
imxvpu: imxvpudec_h263: i.MX VPU h.263 video decoder
imxvpu: imxvpudec_mpeg4: i.MX VPU MPEG-4 video decoder
imxvpu: imxvpudec_mpeg2: i.MX VPU MPEG-1 & 2 video decoder
imxvpu: imxvpudec_webp: i.MX VPU WebP video decoder
imxvpu: imxvpudec_jpeg: i.MX VPU JPEG video decoder
imxvpu: imxvpudec_vp9: i.MX VPU VP9 video decoder
imxvpu: imxvpudec_vp8: i.MX VPU VP8 video decoder
imxvpu: imxvpudec_h265: i.MX VPU h.265 / HEVC video decoder
imxvpu: imxvpudec_h264: i.MX VPU h.264 / AVC video decoder
imxv4l2video: imxv4l2videosink: NXP i.MX V4L2 video sink
imxv4l2video: imxv4l2videosrc: NXP i.MX V4L2 video source
imxaudio: imxuniaudiodec: NXP i.MX uniaudio decoder
imx2d: imxpxpvideotransform: i.MX PxP video transform
imx2d: imxpxpvideosink: i.MX PxP video sink
imx2d: imxipuvideotransform: i.MX IPU video transform
imx2d: imxg2dvideotransform: i.MX G2D video transform
imx2d: imxg2dvideosink: i.MX G2D video sink
imx2d: imxg2dcompositor: i.MX G2D video transform

Could someone help to point out what possible missing dependency or configuration that prevents the vpuenc_h264, please ?

Thank very much in advance and best regards,
Khang

@dv1
Copy link
Collaborator

dv1 commented Oct 13, 2021

This currently is not possible, since the i.MX8m plus has a different encoder (VC8000E) that is not supported by libimxvpuapi. I'm working on this.

@l4es
Copy link
Author

l4es commented Jan 21, 2022

Dear @dv1,

This currently is not possible, since the i.MX8m plus has a different encoder (VC8000E) that is not supported by libimxvpuapi. I'm working on this.

Could you share any progress on this, please ?
Thanks and Best Regards,
Pascal

@dv1
Copy link
Collaborator

dv1 commented Apr 14, 2022

Progress is slow, because the VC8000E encoder is not trivial to use, and because many other tasks keep me busy. I am trying to get it in for Yocto Kirkstone, but I cannot guarantee it.

@d-lareg
Copy link

d-lareg commented Feb 3, 2023

Hi @dv1,

since I am in the same situation as @l4es (developing with buildroot), I would be happy to get an update. Is the encoding already implemented for GStreamer in Yocto. I am using the linux kernel 5.10.9, do you see any problems if I integrate the Yocto package in my Buildroot pipeline, or is the kernel to outdated?

@l4es
Copy link
Author

l4es commented Feb 8, 2023

Hi @d-lareg,

Clarification of VPU plugins :
Prior BSP 5.10.35 : vpuenc_h264 & vpudec_h264
Since BSP 5.10.35 : v4l2h264enc & v4l2h264dec

More detail here : https://community.nxp.com/t5/i-MX-Processors/iMX8MPlus-How-to-mix-two-video-sources-into-one-with-Gstreamer/m-p/1532117

Personally, kernel 5.10.9 seems a little bit outdated even-though it has been stayed quite long time with Buildroot.

@dv1
Copy link
Collaborator

dv1 commented Mar 4, 2023

@d-lareg @l4es I am currently facing the problem that parts of the encoder actually collide with libimvxpuapi's current design. I am trying to implement basic encoding support, but it would only expose a subset of the VC8000E's full functionality. I estimate this will take me 1-2 months to finish at this point.

@dv1
Copy link
Collaborator

dv1 commented Mar 30, 2023

@d-lareg @l4es I finally have a working version in git master. It turned out that a combination of hardware problems and driver bugs were causing a ton of confusion, but I can now encode with the VC8000E. h.264 only for now, but everything is prepared for h.265 encoding.

@l4es
Copy link
Author

l4es commented Apr 2, 2023

Hi @dv1,

Great news! i am eager to test it, if possible! Could you advice which commit that I could use for the H.264 encoding, please? Or should I wait for some time for having H265 encoding available as well ?
Many thanks.

@l4es
Copy link
Author

l4es commented Apr 4, 2023

Hi @dv1,
I found that the implementation was done in https://github.com/Freescale/libimxvpuapi.

Thank you

@l4es
Copy link
Author

l4es commented Apr 5, 2023

Hi @dv1,

Just FYI from Gary Bisson of Boundary Devices :

Actually this isn't needed any longer as NXP now provides a userspace daemon that makes the VPU appear as a standard V4L2 device:
https://github.com/buildroot/buildroot/tree/master/package/freescale-imx/imx-vpu-hantro-daemon
With this, you can use the standard v4l2enc plugin of GStreamer without the need for any additional plugin.

But I think we still need your implementation in older BSPs.

@dv1
Copy link
Collaborator

dv1 commented Apr 5, 2023

The last time I tried the daemon, it was prone to multiple crashes, and did need the NXP fork of GStreamer, because some patches on top of v4l2enc were needed. Perhaps this improved, I'll check, but yeah, this works with older BSPs, and also works with upstream GStreamer.

@l4es
Copy link
Author

l4es commented Apr 5, 2023

Dear @dv1,
For using the H264 encoding with the VC8000E of iMX8M Plus, could you advice the commits of gstreamer-imx and libimxdmabuffer that are associated with Freescale/libimxvpuapi@b79a50a, please ?

@dv1
Copy link
Collaborator

dv1 commented Apr 6, 2023

@l4es I recommend using b69bef1ac976d71d8e9f527d61446d2a7e8c77eb for libimxdmabuffer and a2c0128 for gstreamer-imx .

@l4es
Copy link
Author

l4es commented Apr 8, 2023

Hi @dv1,
Thanks very much. Sorry to bother you again, could you share some quick gstreamer command for testing the encoder on iMX8M Plus please?

@dv1
Copy link
Collaborator

dv1 commented Jun 28, 2023

@l4es Sorry for the late reply. A simple example would be:

gst-launch-1.0 videotestsrc num-buffers=300 ! imxvpuenc_h264 ! h264parse ! matroskamux ! filesink location=test.mkv

Another one would be:

gst-launch-1.0 v4l2src ! queue ! imxvpuenc_h264 ! h264parse ! matroskamux ! filesink location=test.mkv

@d-lareg
Copy link

d-lareg commented Jul 6, 2023

Hi @dv1

is OpenMax required to use the encoder? I tested the following pipe....
gst-launch-1.0 videotestsrc num-buffers=300 ! imxvpuenc_h264 ! fakesink
and get the error...
OMX H264 ! HantroHwEncOmx_encoder_create_h264 H264EncInit failed! (-1) openmax_il/source/encoder/encoder_h264.c:773

My buildroot configuration is:
imx-codec = 4.3.5
imx-vpuwrap = 4.3.5
imxdmabuffer = 1.1.3
imx-vpu-hantro = 1.27.0
imx-vpu-hantro-vc = 1.9.0
imxvpuapi = b79a50a (newest)
gstreamer-imx = 4368bd0 (2 weeks old)

The log for GST_DEBUG=4 is attached.

gst_h264.log

@dv1
Copy link
Collaborator

dv1 commented Jul 6, 2023

@d-lareg This is an imx8m mini? Can you re-run this with GST_DEBUG set to 2,*imx*:9 ?

@d-lareg
Copy link

d-lareg commented Jul 7, 2023

@dv1 no its a self developed board equipped with an imx8m+ soc.
gst_h264_detailed.log

@dv1
Copy link
Collaborator

dv1 commented Jul 8, 2023

@d-lareg There are three variants of imx8m: imx8m quad, imx8m mini, imx8m plus. Which one is yours?

@d-lareg
Copy link

d-lareg commented Jul 8, 2023

imx8m+ = imx8m plus

@dv1
Copy link
Collaborator

dv1 commented Jul 9, 2023

Oh, right, I missed the "+" :) The logs indicate a misconfigured libimxvpuapi then:

imxvpuapi2_imx8m_hantro_h1_encoder.c:812:imx_vpu_api_enc_open:�[00m could not create h.264 encoder

The imx8m plus does not have an H1 encoder, it has the VC8000E encoder. Use the latest git master version of libimxvpuapi, and when using its --imx-platform switch, be sure to pass imx8mp to it, not imx8mm. If you are using Yocto, just modify libimxvpuapi's recipe accordingly. I will update the recipes soon when I make new releases, but until then, you have to do that.

@d-lareg
Copy link

d-lareg commented Jul 11, 2023

Argh, my bad! You are right! But now I am facing a build error when building the image with buildroot.

In the mk file of imx-vpu-hantro-vc the header will copied to the main include path:

define IMX_VPU_HANTRO_VC_INSTALL_STAGING_CMDS
	mkdir -p $(STAGING_DIR)/usr/include/
	cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/*.h $(STAGING_DIR)/usr/include/
	$(INSTALL) -D -m 0755 $(@D)/usr/lib/libhantro_vc8000e.so $(STAGING_DIR)/usr/lib/libhantro_vc8000e.so
	$(INSTALL) -D -m 0755 $(@D)/usr/lib/libhantro_vc8000e.so.1 $(STAGING_DIR)/usr/lib/libhantro_vc8000e.so.1
endef

But libimxvpuapi is searching for a subfolder called hantro_VC8000E_enc that does not exists.

elif self.soc_type == 'MX8MP':
	# i.MX8m plus has the Hantro VC8000E encoder
	conf.define('IMXVPUAPI2_VPU_HAS_VC8000E_ENCODER', 1)
	conf.check_cc(uselib_store = 'HANTRO', uselib = 'HANTRO', define_name = '', mandatory = 1, lib = ['hantro_vc8000e', 'm'])
	conf.check_cc(uselib_store = 'HANTRO_ENC', uselib = 'HANTRO', define_name = '', mandatory = 1, includes = [os.path.join(sysroot_path, 'usr/include')], header_name = 'hantro_VC8000E_enc/hevcencapi.h')

@dv1
Copy link
Collaborator

dv1 commented Jul 11, 2023

@d-lareg Please issue a ticket for fixing that mk file. Copying the headers into /usr/include/ like that is wrong. The hantro_VC8000E_enc subdirectory exists for a reason. Some of the headers have very generic names like base_type.h.

@dv1
Copy link
Collaborator

dv1 commented Jul 11, 2023

@d-lareg Also, can you try to patch that mk file yourself? Just change:

cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/*.h $(STAGING_DIR)/usr/include/

to:

cp -dpfr $(@D)/usr/include/hantro_VC8000E_enc/ $(STAGING_DIR)/usr/include/

If everything works with this patch, then I can make a 2.3.0 release, and buildroot can be notified about this problem in that mk file.

@d-lareg
Copy link

d-lareg commented Jul 11, 2023

@dv1 I fixed the copy issue but still having problems with the encoder. I am now getting the following error:

imxvpuapi imxvpuapi2_imx8m_hantro_vc8000_encoder.c:860:imx_vpu_api_enc_open: could not initialize encoder: EWL error (-5)

What does EWL mean?

gst.log

@dv1
Copy link
Collaborator

dv1 commented Jul 11, 2023

@d-lareg EWL errors can happen when certain device nodes are not available.

Check for the presence of /dev/dma_heap/linux,cma-uncached (or /dev/dma_heap/linux,cma) and /dev/mxc_hantro_vc8000e on your machine.

If one of these are missing, then this might be why you get that error.

Another idea is to re-run your program with strace, like: strace -f -e trace=openat ./my-program You might have to install strace on your machine.

@dv1
Copy link
Collaborator

dv1 commented Jul 30, 2023

@d-lareg Did you have any success in fixing this? Was it a problem with the kernel config or the devicetree?

@Wooker
Copy link

Wooker commented Feb 28, 2024

Hello! I'm also struggling with this issue while building gstreamer-imx and its dependencies with buildroot for imx8mp-evk.

The problem arises due to the use of kernel version 5.4.70, where /dev/dma_heap/ is not available.

According to the libimxvpuapi documentation, ION or dma heap CMA should be enabled in the kernel for proper functionality of vpu elements. Despite configuring libimxdmabuffer to use only ION and enabling ION CMA in the kernel, imxvpuenc_h264 still attempts to access /dev/dma_heap/linux,cma-uncached. This behavior was observed through strace, as you suggested previously:

[pid   288] openat(AT_FDCWD, "/dev/mxc_hantro_vc8000e", O_RDWR) = 20
[pid   288] openat(AT_FDCWD, "/dev/dma_heap/linux,cma-uncached", O_RDWR) = -1 ENOENT (No such file or directory)
VCEncInit: EWL Initialization failed

Versions of libraries:

  • libimxdmabuffer 1.1.3
  • libimxvpuapi 2.3.0
  • gstreamer-imx 2.2.0

Any insights or suggestions on resolving this issue would be greatly appreciated!

Please also see the full strace log: trace.log

@Wooker
Copy link

Wooker commented Mar 13, 2024

Hello again! As it turned out, the issue described above appeared due to inconsistent builds of gstreamer and some of its dependencies. While libimxvpuapi was rebuilt, libraries that depend on it were not. Hence, usage of dma_heap was not replaced by usage of ion.

So, simple buildroot rebuild solved the issue.

@ebanez
Copy link

ebanez commented Aug 7, 2024

Hi @dv1!

I'm having a very similar issue with the imx8mm. Using Yocto Kirkstone 5.15.60.

My message is slightly different from the original post:

OMX H264 ! HantroHwEncOmx_encoder_create_h264 H264EncInit failed! (-5) openmax_il/source/encoder/encoder_h264.c:773

where the error code is -5 instead of -3.

The device mxc_hantro_h1 and mxc_hantro is present and cma is enabled

Installed dependency versions:

  • libimxdmabuffer 1.1.2
  • gstreamer-imx 2.1.0
  • libimxvpuapi 2.2.1

Does anybody know what the difference in error code could mean and any help in general? The sample gst test won't even work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants