The job definition does not exist anymore and zuul config reports error
because of this, so let's get rid of this from the check queue (gate
queue does not even have it).

Change-Id: I0b211b7c8fcfebeaf9f1bc9bbee0c6527faa53d9

py39 jobs (on ubuntu-focal) started to fail due to recent virtualenv
release (20.26.4 that bundles setuptools 74.1.2) on Yoga, because
we have 'packaging==21.3' in this branch that is not compatible
with the new setuptools [1].

setuptools is bundled in virtualenv, so it has to be capped via the
virtualenv package. tox also needed to be capped (<4) as gate uses
tox 3.28.0, but with capping virtualenv we pull in latest tox as well,
which would cause other errors.

[1] pypa/setuptools#4483

Change-Id: I668557933c51123ce84275a0c718b6e79b3df174

This adds a safety check and detection mechanism to the
tools/test_format_inspector.py utility for verifying those features
outside of glance.

Change-Id: I447e7e51315472f8fa6013d4c4852f54c1e0c43d
(cherry picked from commit b27040b87e43ff4acfb6870ef0d13d54e5ca5caa)
(cherry picked from commit b394ef00c3426771092d099cf1d96077bfa4b919)
(cherry picked from commit aa12d39b453068d9ee8367591eb60d4a15cddece)
(cherry picked from commit a1c94b4)
(cherry picked from commit a5ec69b)

…ned/yoga

We have to address two issues simultaneously:

1. handle multiple cirros images

   The change[1] configures multiple images for the devstack-tempest
   job and all the jobs deriving from it. This causes problem with
   the current post-check-metadata-injection.yaml playbook since
   we only expect one cirros image and perform steps accordingly.
   This change modifies the playbook to handle multiple cirros
   images.
   Failure in the glance-multistore-cinder-import job can be
   seen here[2].

   [1] https://review.opendev.org/c/openstack/tempest/+/831018
   [2] https://zuul.opendev.org/t/openstack/build/68a4d3ec6ce04c87b21d73a333f5b5cd/log/job-output.txt#23161

   This was change I70b4a970d7e1.

2. change the the job config to address resource constraints
   * partial backport of I073216d1bbdd to change swap size
     (we don't include the change from that patch that was needed
     to address changes in Bobcat-era devstack)
   * backporting Ieb96eb6ceb6f to change tempest concurrency

Remove openstack-tox-functional-py36-fips as there is no
job definition for it.

Change-Id: I6cd5b8bbf39f0cfae466f0800faea5a74e960e69
(cherry picked from commit 065e2d7)
(cherry picked from commit 28b05a7)

VMDK:
When parsing a VMDK file to calculate its size, the format_inspector
determines the location of the Descriptor section by reading two
uint64 from the headers of the file and uses them to create the
descriptor CaptureRegion.

It would be possible to craft a VMDK file that commands the
format_inspector to create a very big CaptureRegion, thus exhausting
resources on the glance-api process.

This patch binds the beginning of the descriptor to 0x200 and limits
the size of the CaptureRegion to 1MB, similar to how the VMDK
descriptor is parsed by qemu.

VHDX:
It is a bit more involved, but similar: when looking for the
VIRTUAL_DISK_SIZE metadata, the format_inspector was creating an
unbounded CaptureRegion.

In the same way as it seems to be done in Qemu, we now limit the upper
bound of this CaptureRegion.

Closes-Bug: #2006490
Change-Id: I3ec5a33df20e1cfb6673f4ff1c7c91aacd065532
(cherry picked from commit d4d33ee)
(cherry picked from commit 06a1820)

Stream optimized VMDKs are also monolithic disks images, and start
with the same sparse extend header as normal monolithic sparse files,
so we can parse the virtual disk size in the same manner.

See "VMware Virtual Disks Virtual Disk Format 1.1" p. 17.
> Header and Footer
> The header and the footer are both described by the same SparseExtentHeader
> structure shown in Hosted Sparse Extent Header on page 8.

Closes-Bug: #2052291
Change-Id: I7d63951ff080dc699b8d11babc0a5998d90621e4
Co-Authored-By: Rajiv Mucheli <[email protected]>
(cherry picked from commit 5e7e6bf)
(cherry picked from commit 9d45e8d4b87e992be23974a831811bff563ce721)
(cherry picked from commit ef22a74)
(cherry picked from commit bcfc1e4)

Change-Id: I6326a3e85c1ba4cb1da944a4323769f2399ed2c1
Closes-Bug: #2059809
(cherry picked from commit 2ca29af4433e9fa99a0a48e230d8d25d6eaa4a87)
(cherry picked from commit c3586f3a122f6cb0663217b12b52203e74e2e4fa)
(cherry picked from commit a92c438fb5ba55440b38cae7c8b4361b58daa9dd)
(cherry picked from commit dba3bdb)
(cherry picked from commit 6a38aef)

This adds two properties to the QcowInspector that makes it able to
indicate whether the file specifies a backing_file or data_file in the
header. Both conditions are considered unsafe for our usage. To
ease checking of this condition, a classmethod is added that takes
a local filename and digests just enough of the file to assert that
both conditions are false.

Note for yoga backport:With older qemu installed one of the qemu-img
create commands fails, let's skip it from unmaintained/yoga and
below that.

Change-Id: Iaf86b525397d41bd116999cabe0954a0a7efac65
Related-Bug: #2059809
(cherry picked from commit ae536bb394793c9a7a219cb498e03d5c81dbbbb7)
(cherry picked from commit 2eba54e0821106097dfeceb424e53943fd090483)
(cherry picked from commit 89dbbc838d606f461087e1494d19ddbcf9db0a38)
(cherry picked from commit 4860024)
(cherry picked from commit f32d5b8)

This makes us check the extent filenames to make sure they don't
have any banned characters in them (i.e. slashes). It also makes
us reject VMDK files with a footer. Since we process these files
as a stream, we can't honor a footer that directs us to find the
descriptor block in a location we've already processed. Thus, if
a file indicates it has a footer, consider it a policy exception
and unsupported.

Change-Id: I4a1c6dff7854c49940a0ac7988722aa6befc04fa
(cherry picked from commit c1bf35dffb7f4c3090b1f04cf0e27cb431330c3e)
(cherry picked from commit d3f1d6159c0218ac01e8d881e2ec4da71fc952ee)
(cherry picked from commit 2dd4d138d4b8e1d9ca69fc0dda3711553a65d912)
(cherry picked from commit 812e56d)
(cherry picked from commit 05650e1)

Recent changes does not allow image conversion to convert an image
having AMI disk format for image. Passing list of disk-formats to
each import job which will now use qcow2 as default disk-format to
create the image.

Conflicts: .zuul.yaml

Change-Id: I712b304ebe168705ec2793626f5d64135ad6d458
(cherry picked from commit 244ccb6)
(cherry picked from commit f98e9ae)
(cherry picked from commit 02732c2)