Avoid exclusive expectations that cloud-init is the only agent
registering certificates in a system to
/etc/ssl/certs/ca-certificates.crt.

On Google Cloud Platform, Google Guest Agent does setup root certs
which makes performing a checksum of ca-certificates.crt incorrect due
to extra certs present in ca-certificates.crt.

Adapt test to assert that cloud-init's cert is contained in
ca-certificates.crt but not exclusive content of the file.

Fixes canonicalGH-5609

* Collect sensitive data by default since we ask for it more often
  than not
* Output warning that we're collecting sensitive data
* Glob most of /run/cloud-init, /etc/cloud, and /var/lib/cloud
* Stop creating empty directories in the tarball
* Require running as root given that the logs are root read-only
* Update apport accordingly

Fixes canonicalGH-5297

In the case of Pro, if either agent or user data is not cloud-config user-data,
combine the parse in `self.userdata_raw` as a #include file so
cloud-init transforms that internally into a multipart data.

Avoid passing strings and lists directly, which confused the processor due the
lack of a mime type.

Being explicit about only loading text/cloud-config parts also allow other
composition of cloud-init features to just work, like jinja templates.

This error was surfaced when testing with empty Landscape
data, but any non-text/cloud-config content type would trigger the same behavior.

Add merge_agent_landscape_data to process agent.yaml or Landscape data and
ignore any empty  files present in .ubuntupro/.cloud-init/

…ical#5562)

Also emphasize ''users''.

…ical#5562)

Without this fix, rendered module documentation was not rendering the
following text for some objects:

Each object in **<key_name>** list supports the following keys:

See Rsyslog Config schema tab.

…cal#5562)

Document any keys of objects in a list which allows for objects as one
of the alternative types allowed as a list item.

Also, when documenting properties, ensure we skip documentation of
either 'properties' or 'patternProperties' if those properties are
declared in the hidden key.

Fixes canonicalGH-5514

…5562)

When running tox -e doc the following environment variables are
supported:
  CLOUD_INIT_DEBUG_MODULE_DOC=cc_<module_id>
  CLOUD_INIT_DEBUG_MODULE_DOC_FILE=<file_path>

The env var CLOUD_INIT_DEBUG_MODULE_DOC can be set to either
a specific module id, such as cc_rsyslog, or 'all'.

When set the rendered module documentation RST format is printed
inline to stdout to allow for quick analysis of rendered content.

Optionally, if CLOUD_INIT_DEBUG_MODULE_DOC_FILE is set to a writable
file path, the output of the rendered content is written to that file
instead.

This supports development of docs and quick comparison of docs
generated before and after a changeset.

…mes (canonical#5562)

…ical#5562)

Remove additional \n which is not present if only one ca_cert is in the
instance.

…l#5624)

…sv (canonical#5624)

Align integration test with c28092f.

Adapt to new annotation formating from
a2193da.

…onical#5636)

Directly calling execute("cloud-init clean --logs --reboot") on
an integration instances also involves awaiting a new boot id upon
next interaction with with instance to ensure a reboot has actually
taken place already on this target machine.

Slow responding test instances/platforms may not completed the shutdown
restart sequence yet when trying to iteract with an immediate blocking
call to execut("cloud-init status --wait") which may exit early if accessing
the prior instance boot before the reboot occurred.

It is preferable to use inspect /proc/sys/kernel/random/boot_id before
issuing a reboot request and block until a delta is seen in boot_id.
This blocking wait on reboot and new boot_id is encapsulated inside
pycloudlib.BaseInstance.restart which will inspect
/proc/sys/kernel/random/boot_id before restart and block until a delta
in boot_id across the requested restart.

Fix test_status_block_through_all_boot_status to call instance.clean()
and restart() to ensure we do not beat the instance reboot race with
our post-boot assertions.

Add PPS support for azure-proxy agent and improve error logging.

…nical#5633)

Do not treat the emptiness of .cloud-init/ as an error in the logs
if agent.yaml is present.

Fixes canonicalGH-5632

…ply (canonical#5622)

Perform the same steps that cloud-init daily recipe builds performs
to assert any packaging branch updates will not break daily builds
due to quilt patch apply issues.

Steps of daily build recipe reflected in this workflow:
- checkout main
- merge packaging branch topmost commit
- quilt push -a
- run unittests (via tox -e py3)
- quilt pop -a

…cal#5641)

Reintroduce strict assert that cloud-init's cert in userdata is the
only root cert defined on the platform. Google guest agent was
installed a secondary root cert in ca_certifications.crt for a period of
time and this was determined to be less than ideal practice.

Allow cloud-init's integration tests to remain strict validation of
cert checksum to provide a signal if other platforms or agents
attempt to extend or alter the system-wide CA.

…d field (canonical#5355)

Currently cc_user_groups assumes that "useradd" never locks the password
field of newly created users. This is an incorrect assumption.

Change add_user (in both __init__.py and alpine.py) to
explicitly call either lock_passwd or unlock_passwd at all times to
achieve the desired final result.

For existing users with empty or empty locked passwords, no
password unlock will be performed and warnings will be issued.
To support empty password validation, provide functionality to
parse /etc/shadow and /var/lib/extrausers/shadow to assert
existing users do not have empty passwords before unlocking.

Additionally in this commit: 
- add NetworkBSD.ifs property to avoid subp side-effect in ___init__
  which calls ifconfig -a at every instance initialization

Useradd background:

From the useradd manpage:

'-p, --password PASSWORD
The encrypted password, as returned by crypt(3). The default is to
disable the password.'

That is, if cloud-init runs 'useradd' but does not pass it the "-p"
option (with an encrypted password) then the new user's password field
will be locked by "useradd".

cloud-init only passes the "-p" option when calling "useradd" when
user-data specifies the "passwd" option for a new user. For user-data
that specifies either the "hashed_passwd" or "plain_text_passwd"
options instead then cloud-init calls "useradd" without the "-p" option
and so the password field of such a user will be locked by "useradd".

For user-data that specifies "hashed_passwd" for a new user then
"useradd" is called with no "-p" option, so causing "useradd" to lock the
password field, however then cloud-init calls "chpasswd -e" to set the
encrypted password which also results in the password field being
unlocked.

For user-data that specifies either "plain_text_passwd" for a new user
then "useradd" is called with no "-p" option, so causing "useradd" to
lock the password. cloud-init then calls "chpasswd" to set the password
which also results in the password field being unlocked.

For user-data that specifies no password at all for a new user then
"useradd" is called with no "-p" option, so causing "useradd" to lock
the password. The password field is left locked.

In all the above scenarios "passwd -l" may be called later by
cloud-init to enforce "lock_passwd: true").

Conversely where "lock_passwd: false" applies the above "usermod"
situation (for "hash_passwd", "plain_text_passwd" or no
password) means that newly created users may have password
fields locked when they should be unlocked.

For Alpine, "adduser" does not support any form of password being
passed and it always locks the password field (the same point
applies about password field being unlocked when/if "chpasswd" is
called). Therefore in some situations (i.e. no password specified
in user-data) the password needs to be unlocked if
"lock_passwd: false".

Bump the version in cloudinit/version.py to 24.3 and
update ChangeLog.

…nical#5640)

If an instance in _routed-ip_ mode uses both IPv4 **and** IPv6, but
the IPv6 is listed (attached) first, the default IPv6 route may be
absent from the final network configuration for systems that do not
accept Router Advertisements.

This change fixes the problem by making sure that routes are
appended, not overwritten.

- Update upstream cloud-init config template for azurelinux distro
- remove modules that not currently supported azurelinux

Co-authored-by: minghe <rmhsawyer>

Co-authored-by: Marat Budkevich <[email protected]>

Co-authored-by: Chad Smith <[email protected]>