Can't Mirror Ubuntu 22.04 LTS repos

[rolffujino]

From 2021/06/14 Ubuntu switched the default compression for dpkg from xz to zstd (CHANGELOG).
I believe this causing my attempts to mirror the Ubuntu 22.04 LTS to fail:

Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/accountsservice_22.07.5-2ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/acct_6.6.4-4build2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/acl_2.3.1-1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/acpi-support_0.144_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/acpid_2.0.33-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/adcli_0.9.1-1ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/adsys_0.8.4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/advancecomp_2.1-2.1ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/adwaita-icon-theme_41.0-1ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aide_0.17.4-1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aide-common_0.17.4-1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aisleriot_3.22.22-1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/alembic_1.7.6-1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/alsa-topology-conf_1.2.5.1-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/alsa-ucm-conf_1.2.6.3-1ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/alsa-utils_1.2.6-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/amavisd-new_2.12.2-1ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/amd64-microcode_3.20191218.1ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/anacron_2.3-31ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/anope_2.0.9-1build2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-api_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-common_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-doc_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-evaluator_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-expirer_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-listener_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aodh-notifier_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2_2.4.52-1ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-bin_2.4.52-1ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-data_2.4.52-1ubuntu4_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-dev_2.4.52-1ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-doc_2.4.52-1ubuntu4_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-ssl-dev_2.4.52-1ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apache2-utils_2.4.52-1ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apg_2.2.3.dfsg.1-5build2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apparmor_3.0.4-2ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apparmor-profiles_3.0.4-2ubuntu2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apparmor-utils_3.0.4-2ubuntu2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apport_2.20.11-0ubuntu82_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apport-gtk_2.20.11-0ubuntu82_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apport-retrace_2.20.11-0ubuntu82_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/appstream_0.15.2-2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt_2.4.5_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-config-icons_0.15.2-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-config-icons-hidpi_0.15.2-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-config-icons-large_0.15.2-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-config-icons-large-hidpi_0.15.2-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-doc_2.4.5_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apt-utils_2.4.5_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aptdaemon_1.1.1+bzr982-0ubuntu39_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aptdaemon-data_1.1.1+bzr982-0ubuntu39_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apturl_0.5.2ubuntu22_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/apturl-common_0.5.2ubuntu22_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aspell_0.60.8-4build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/aspell-doc_0.60.8-4build1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/at-spi2-core_2.44.0-3_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/at-spi2-doc_2.44.0-3_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/attr_2.5.1-1build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/auditd_3.0.7-1build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/authbind_2.1.3build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autoconf_2.71-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autoconf-doc_2.71-2_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autodep8_0.25_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autofs_5.1.8-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/automake_1.16.5-1.3_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autopkgtest_5.20_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autopoint_0.21-4ubuntu4_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/autotools-dev_20220109.1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/avahi-autoipd_0.8-5ubuntu5_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/avahi-daemon_0.8-5ubuntu5_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/avahi-utils_0.8-5ubuntu5_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/b43-fwcutter_019-7build2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/backuppc_4.4.0-5ubuntu2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/backuppc-rsync_3.1.3.0-3build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/baobab_41.0-2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/barbican-api_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/barbican-common_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/barbican-doc_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/barbican-keystone-listener_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/barbican-worker_14.0.0-0ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/base-files_12ubuntu4_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/base-passwd_3.5.52build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bash_5.1-6ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bash-completion_2.11-5ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bash-doc_5.1-6ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bc_1.07.1-3build1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bcache-tools_1.0.8-4ubuntu3_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bcrelay_1.4.0-12build2_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9_9.18.1-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9-dev_9.18.1-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9-dnsutils_9.18.1-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9-doc_9.18.1-1ubuntu1_all.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9-host_9.18.1-1ubuntu1_amd64.deb'!
Could not find a suitable control.tar file within '/home/repo/ubuntu/jammy/14-11-2023_main/packages/bind9-libs_9.18.1-1ubuntu1_amd64.deb'!
Warning: database 'jammy|main|amd64' was modified but no index file was exported.
Changes will only be visible after the next 'export'!
There have been errors!

[rolffujino]

Just noticed this is a graphical fronted for reprepo, therefore this issue relates to this bug ionos-cloud/reprepro#20

[lbr38]

Indeed Repomanager is using reprepro to build deb repositories metadata.

It's seems the zstd support is still experimental.. I don't know how but I will see if I can build reprepro newest version from source and test it.

Sometimes I also think about making my own metadata builder tool to get rid of this...

All I can tell you is that it's going to take time.

[rolffujino]

Supposedly support is added in bullseyebackports version.

Updates between 5.3.0 and 5.3.1:

• fix manpage to add the behaviour if reprepro is linked against liblzma
• mark 'dumpcontents' command as deprecated
• Add Zstd support

Will give it a shot tomorrow.
https://packages.debian.org/bullseye-backports/reprepro

Also might try building repomanager on an Ubuntu image and see how it goes.

[lbr38]

reprepro version is already 5.3.1-1+deb12u1 (changed the base image to debian 12 in the latest release) and it's still broken :-(

[lbr38]

I fixed this issue but I'm now facing another problem with reprepro randomly hanging on some packages import, still related to zstd I think..

I continue my investigation

[rolffujino]

@lbr38 how did you fix the issue, I haven't had an opportunity to work on the issue myself yet and I am curious how you did it.

[rolffujino]

@lbr38 from my testing of your latest stable docker image, the fault doesn't seem to lie entirely with reprepro.
When I upload this Ubuntu package http://archive.ubuntu.com/ubuntu/pool/main/n/netkit-telnet/telnet_0.17-44build1_amd64.deb via the GUI and rebuild the repo, I receive the expected control.tar error.
However if I enter the container and run the command manually, there are no errors at all:

root@3b8c84aa85b3:/home/repo# reprepro --version
reprepro: This is reprepro version 5.3.1
root@3b8c84aa85b3:/home/repo# reprepro -b /home/repo/q/jammy/20-11-2023_main/ includedeb jammy ./telnet_0.17-44build1_amd64.deb 
Exporting indices...

I have confirmed repomanager is not running an older version of the software:

(I fiddled with your code to confirm the version)

I am guessing it might be environment related:

vs:

root@3b8c84aa85b3:/home/repo# env
HOSTNAME=3b8c84aa85b3
MAX_UPLOAD_SIZE=128M
PWD=/home/repo
HOME=/root
FQDN=repomanager.example.com
TERM=xterm
SHLVL=1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
OLDPWD=/var/lib/repomanager

I hope this points you in the right direction.

[lbr38]

Yes it was indeed a missing env var problem. I fixed the PATH var yesterday (but not yet released): https://github.com/lbr38/repomanager/pull/136/files#diff-a07f7ac307fad4d2cda6bc25e9736a9db54d4daed82fed2090daf0a6dadcdd0f

But now as I said I'm facing another problem with reprepro hanging while importing packages (when mirroring ubuntu > jammy > main)

[lbr38]

I released the fix (in v3.7.7) for the Could not find a suitable control.tar file within error, but still have no idea what to do with the reprepro hanging problem.

I looked deep in the source code of reprepro and identified what line might cause the problem but I have no idea what to do next. Also tried to add more logging in the source code, tried to build the latest version of reprepro from source but still facing the same problem. I will see if I can find a way to open an issue to a Debian dev to ask for help.

Maybe you can test on your side and tell me if reprepro is hanging when mirroring ubuntu > jammy > main?

[rolffujino]

@lbr38 I have replicated the issue on my desktop running your latest image, will try fiddling with some pieces:


I left that running for a large number of hours.
Also this is what the processes look like in the container:

It is just sitting around doing nothing, except one process whose CPU utilization has grown slowly throughout the day:

vs

[rolffujino]

@lbr38 why did you decide to use the includedeb method for mirroring large repos like Ubuntu vs the updates method (I am genuinely curious). It seems that using updates, is the intended methodology from reprepro for mirroring scenarios, while includedeb is for local repos, though it does seem a lot slower.

[lbr38]

@rolffujino I'm not using update method because I'm not mirroring with reprepro but with my own internal mirroring system (basically curl) written in PHP (see controllers/Repo/Mirror/*).
Then I use reprepro to build the repository metadata once all the packages have been retrieved, so the
includedeb method is the only one that works in this case :-)

[lbr38]

I found someone having the same issue with the ubuntu repository: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056380

I guess there is nothing else I can do for now, I will keep and eye on the Debian bug report and see if there is any progress in the next days.

[rolffujino]

@lbr38 FYI I did test the update method and it also hangs (even after 8 hours no progress).

[lbr38]

@lbr38 FYI I did test the update method and it also hangs (even after 8 hours no progress).

Alright thank you for testing :-)

[lbr38]

Hello @rolffujino

Can you please confirm that the latest version (3.7.10) solves this issue?
I tried on my side to mirror ubuntu > jammy > main and it's working well.

Seeing no progress on the reprepro bug issued on bugs.debian.org, I just replaced reprepro with another metadata building tool called apt-ftparchive.

[cfiehe]

Hi @lbr38 @rolffujino,
we are facing the same problem with reprepro hanging on packages compressed with zstd. We are running Ubuntu 22.04 and have tested multiple versions (>= 5.3.1 with the Broken pipe fix) , none of them works properly. I have opened an Ubuntu bug (https://bugs.launchpad.net/ubuntu/+source/reprepro/+bug/2047775). I also did some code analysis, but I was not able to narrow down the issue.

This commit fixes the "Broken pipe" issue: https://salsa.debian.org/debian/reprepro/-/commit/6a8ac062c49093ad33689754558e626851070e61. I am not sure, if that fix causes that reprepro hangs from time to time on zstd-compressed packages or if those hangs must be regarded as a separate (independent) issue.

[lbr38]

Hello @cfiehe

Thank you for your message but this issue is now closed as reprepro is no longer part of Repomanager. Having seen no progress on the bug fix on bugs.debian.org, I had to make the decision to abandon reprepro in order not to block this issue for too long.

I hope you will get an answer to your problem from the Debian/Ubuntu team.

[cfiehe]

That is an understandable solution. A colleague and I did some testing and we were able to narrow down the issue, but we are unsure if our changes really solve the issue. We are both no C++ experts: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056380#20