Unable to uncompress the compressed core file if the file size is so big

[sehdshskj]

Hello, when i use the tool minicoredumper on my pc, i have encountered a problem. If the core file size is so big(14 GB), it is unable to uncompress the compressed file. I think the reason is that the file size is more than the maximum size a single tar file supports. Are there some alternate solutions to be sought for this?

[jogness]

Thank you for reporting this.

I can easily confirm the problem. After looking deeper I notice that the used tar format (ustar) does not support files above 8GB. In order to support larger core files a more modern tar format will be needed. As a first step I would like to release a version of minicoredumper that detects the problem and either does a partial capture or none at all.

[jogness]

One possible alternative is to disable "in_tar" in the recept. However, by >8GB core files, this is probably horribly inefficient. To make things worse, the dump_zero() function in the minicoredumper is also horribly inefficient.
I did some testing of my own, and with "in_tar" set to false and a modified dump_zero() to use 1KB buffers, my test program with 16GB allocated takes about 1.5 minutes to core dump. (With tar it would only takes a few seconds, but the minicoredumper tar will be broken.)
I suppose for the next step (a 2.0.3 release) I will add detection if the core file is >8GB and in that case fallback to direct dumping to gzip. Horribly slow and the compressed core file is much larger than with the tar, but at least a core file would exist.

[jogness]

In the master branch I have pushed a workaround that will be released as 2.0.3 soon. See the commit message for details. It refers to the mailing list thread where various alternatives, workarounds and application implementations were discussed.

I am leaving this issue open because the proper solution is to implement a more modern tar format to support large files.

[sehdshskj]

Hello,
Thank you very much for your reply! I have great admiration for your rigor and pragmatism, and it is a great honor to continue to discuss this issue with you in depth.
I have read the commit message on github and the content of the mailing list thread. Same as I analyzed the other day, the best alternative solution for this is to use modern tar format which support the sparse file with unlimited size. But it takes plenty of time to fix it,
I have another easier but less elegant workaround here. If the core file size is more than 8GB, we can truncate it and split it into multiple segments, and a single tar header with a single segment which size is smaller than 8GB. So after core data dumped, there are multiple file in core.tar.gz. After uncompressing core.tar.gz, we use cat command to merge the files. In this way we can avoid size limitations on a single files. I have used this way to fix on my own computer without much coding and impact on performance I know this is just a temporary replacement. All in all, I'm very much looking forward to the new release.

[dharmendra-nutanix]

Hello John,
Thanks for this awesome tool! Has the work to move to a modern tar format started? If not, we were planning to add pax support and create a pull request for it.

[jogness]

Unfortunately I will not get to updating the tar format anytime soon. Patches to support additional formats to fix this issue are very welcome!

[dharmendra-nutanix]

Hey @jogness, Prakhar from Nutanix has created a pull request at #10 for adding the PAX support. Can you please take a look? Thanks

[jogness]

Yes, I have seen it. I am taking a look at it. Thanks for the ping.

[ShaharHD]

Any updates on getting #10 merged?

Got the same issue with a small tester applicaiton on an Aarch64 linux (Yocto)