How to determine which errors are fatal from "Some items failed to download. Downloaded data may be incomplete"?

[barrettc]

Hi there,
I'm running into a similar issue as this one with Ubuntu 20.04. I'm trying to run apt-offline in the context of a script and the apt-offline get operation reports a non-zero exit code. When I take the generated bundle to the disconnected machine, I see that I'm able to install my packages correctly so the errors being reported are not true problems. I'm trying to figure out how I can determine which of the missing translation errors are actual problems and which can be ignored so I can handle this in my script. I'll show the verbose output from what I'm seeing - thanks for any help.

apt-offline set:

apt-offline set --verbose /home/bc/.larry/nuc1.sig --update --install-packages net-tools nfs-common
VERBOSE: Namespace(apt_backend='apt-get', func=<function setter at 0x7f1cdbbdf4c0>, generate_changelog=False, set='/home/bc/.larry/nuc1.sig', set_install_packages=['net-tools', 'nfs-common'], set_install_release=None, set_install_src_packages=None, set_simulate=False, set_update=True, set_upgrade=False, src_build_dep=False, upgrade_type='upgrade', verbose=True)
VERBOSE: APT Update Method is of type: apt-get
Gathering details needed for 'update' operation
VERBOSE: Command is: ['/usr/bin/apt-get', '-q', '--print-uris', 'update']
VERBOSE: Calling __FixAptSigs to fix the apt sig problemVERBOSE: APT Install Method is of type: apt-get
Gathering installation details for package ['net-tools', 'nfs-common']
VERBOSE: Command is: ['/usr/bin/apt-get', '-qq', '--print-uris', 'install', 'net-tools', 'nfs-common']

Attached the apt-offline get operation output as a txt file.
apt-offline-get.txt

[barrettc]

After some more digging, I realized that the version of apt-offline installed with apt and Ubuntu 20.04 is 1.8.2. I used dpkg to manually install a .deb version of 1.8.5 and my problem went away. So I suppose this is a workaround for now as I'm stuck on Ubuntu 20.04 for this project.

[rickysarraf]

The translation files not being available is not a fatal problem; which is why we don't bail out on those errors. But given they are errors still, we shouldn't suppress them, which is why we report that those payloads failed.

Similarly, consider of the 100 .debs being downloaded, one failed. We can't abort on it just for that failure. Which is why we report the user of the failure and still complete the overall operation.

As to what all failed, it can be ascertained by running the same operations in --verbose mode.

[barrettc]

I understand what you're saying but when one of the non-fatal files are encountered, the process ends with a non-zero exit code making it difficult to create automation with. Is it possible to exit with some sort of special code like "2" to indicate that there were problems downloading some files but the error should be considered non-fatal?

[rickysarraf]

Currently, it exits with error code 100 here:

apt-offline/apt_offline_core/AptOfflineCoreLib.py

Lines 1499 to 1507 in 05e3ed7

	if len(errlist) > 0:
	log.err("Some items failed to download. Downloaded data may be incomplete\n")
	log.err("Please run in verbose mode to see details about failed items\n")
	log.msg("\n\n")
	log.verbose("The following files failed to be downloaded.\n")
	log.verbose("Not all errors are fatal. For eg. Translation files are not present on all mirrors.\n")
	for error in errlist:
	log.err("%s failed.\n" % (error))
	sys.exit(100)

But errlist is a list to which all failures are appended. Differentiating error types on the basis of source is not being handled.

You could use the same error code 100 but please beware that even the .deb failures fall under the common errors.

apt-offline/apt_offline_core/AptOfflineCoreLib.py

Lines 1300 to 1345 in 05e3ed7

	if url.endswith(".deb"):
	try:
	PackageName = pkgFile.split("_")[0]
	except IndexError:
	log.err("Not getting a package name here is problematic. Better bail out.\n")
	sys.exit(1)
	#INFO: For Package version, we don't want to fail
	try:
	PackageVersion = pkgFile.split("_")[1]
	except IndexError:
	PackageVersion = "NA"
	log.verbose("Weird!! Package version not present. Is it really a deb file?\n")
	#INFO: find_first_match() returns False or a file name with absolute path
	full_file_path = func(Str_CacheDir, pkgFile)
	#INFO: If we find the file in the local Str_CacheDir, we'll execute this block.
	if full_file_path is not False:
	if FetcherInstance.verifyPayloadIntegrity(full_file_path, checksum):
	log.success("%s found in cache%s\n" % (PackageName, LINE_OVERWRITE_FULL))
	#INFO: When we copy the payload from the local cache, we need to update the progressbar
	# Hence we are doing it explicitly for local cache found files
	FetcherInstance.addItem(download_size)
	FetcherInstance.writeData(full_file_path)
	FetcherInstance.processBugReports(PackageName)
	FetcherInstance.updateValue(download_size)
	FetcherInstance.completed()
	if PackageName in list(PackageInstalledVersion.keys()):
	FetcherInstance.buildChangelog(full_file_path, PackageInstalledVersion[PackageName])
	else:
	log.verbose("%s checksum mismatch. Skipping file %s\n" % (pkgFile, LINE_OVERWRITE_FULL) )
	log.msg("Downloading %s - %s %s\n" % (PackageName, log.calcSize(download_size/1024), LINE_OVERWRITE_FULL) )
	if FetcherInstance.download_from_web(url, pkgFile, Str_DownloadDir):
	log.success("%s done %s\n" % (PackageName, LINE_OVERWRITE_FULL) )
	FetcherInstance.writeData(pkgFile)
	FetcherInstance.writeToCache(pkgFile)
	FetcherInstance.processBugReports(PackageName)
	FetcherInstance.updateValue(download_size)
	if PackageName in list(PackageInstalledVersion.keys()):
	FetcherInstance.buildChangelog(pkgFile, PackageInstalledVersion[PackageName])
	else:
	log.err("Failed to download %s\n" % (PackageName))
	errlist.append(PackageName)
	else:

[rickysarraf]

Is it possible to exit with some sort of special code like "2" to indicate that there were problems downloading some files but the error should be considered non-fatal?

I haven't fully validated but looking at the code, it seems that is what it does.