Skip to content

Commit

Permalink
Salt - caching proxy -fix whitespace in baseurl definition.
Browse files Browse the repository at this point in the history
Closes #8
  • Loading branch information
NAME authored and NAME committed Aug 19, 2022
1 parent 147c07e commit 3f59aac
Show file tree
Hide file tree
Showing 3 changed files with 27 additions and 73 deletions.
80 changes: 11 additions & 69 deletions cacher/README
Original file line number Diff line number Diff line change
Expand Up @@ -10,81 +10,23 @@ qubesctl --skip-dom0 --targets=cacher state.apply cacher.configure

To automatically use the proxy run:
qubesctl state.apply cacher.use
This will configure /etc/qubes/policy.d/30-user.policy to use the caching proxy be default.
This will configure /etc/qubes/policy.d/30-user.policy to use the caching proxy by default.

apt-cacher-ng will cache HTTPS requests if you change https:// to http://HTTPS/// in repo source lists.
To configure the templates to use the proxy in this way, run:
qubesctl --skip-dom0 --targets=Templates state.apply cacher.change_templates.sls
Or target individual templates, as you wish.
To configure all templates to use the proxy in this way, run:
qubesctl --skip-dom0 --templates state.apply cacher.change_templates
Or target individual templates, as you wish:
qubesctl --skip-dom0 --targets=TEMPLATE1,TEMPLATE2 state.apply cacher.change_templates


N.B
apt-cacher-ng works well for Debian,Ubuntu,and Arch.
It works reasonably well for Fedora, but may require tweaking of the apt-cacher-ng control file, and the fedora_mirrors lists.

Using apt-cacher-ng as caching proxy.

1. INSTALL AND CONFIGURE
apt-get install apt-cacher-ng, and mask in the template.
systemctl mask apt-cacher-ng

Create qube and give it plenty of space.

In caching qube, use bind-dirs:
binds+=( '/var/cache/apt-cacher-ng' )
binds+=( '/var/log/apt-cacher-ng' )
binds+=( '/etc/apt-cacher-ng' )

Use /rw/config/rc.local to start the apt-cacher-ng service:
systemctl unmask apt-cacher-ng
systemctl start apt-cacher-ng
/sbin/iptables -I INPUT -p tcp --dport 8082 -j ACCEPT

Edit /etc/apt-cacher-ng/acng.conf:
Port:8082

Restart service.

Set this as updateProxy in /etc/qubes-rpc/policy/qubes.UpdatesProxy

Debian templates will use this for updates with no further configuration.


2. FEDORA SUPPORT for build machines.
There is a mirrors list in /usr/lib/apt-cacher-ng/
Copy fedora_mirrors to /etc/apt-cacher-ng

Edit /etc/apt-cacher-ng/acng.conf:
Remap-fedora: file:fedora_mirrors

If requests fail because the file type is not allowed, create a pattern for
volatile data:
VfilePatternEx: .*metalink?repo=fedora*


3. TLS SUPPORT:
Two methods:
a.
Create new file in /etc/apt-cacher-ng/backends_qubes:
https://yum.qubes-os.org/

Edit /etc/apt-cacher-ng/acng.conf:
Remap-qubes: http://fake.qubes ; file:backends_qubes

Then in /etc/yum/yum.repos.d, change the repository URL to
http://fake.qubes/......
It works reasonably well for Fedora, but may require further tweaking of the apt-cacher-ng control file, and the fedora_mirrors lists.

Now the qube will use HTTP to the proxy which will use TLS to pick up the
packages and cache any responses.
To restore a template to the default state, a state is provided, restore_templates.sls.
Apply it like this:
qubesctl --skip-dom0 --targets=TEMPLATE1,TEMPLATE2 state.apply cacher.restore_templates

b.
Change the repository definition FROM:
https://yum.qubes-os.org/
TO:
http://HTTPS///yum.qubes-os.org/
The qrexec policy file at /etc/qubes/policy.d/30-user.policy should be edited so that these templates use the default system proxy.

Without any other changes to the apt-cacher configuration the qube will
use HTTP to the proxy which will use TLS to pick up the packages and
cache any responses.

This is the recommended approach.
Salt states, change_templates.sls, and restore_templates.sls, are provided to configure templates to use this scheme.
10 changes: 8 additions & 2 deletions cacher/change_templates.sls
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
- pattern: 'https://'
- repl: 'http://HTTPS///'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% endfor %}

/etc/apt/sources.list:
Expand All @@ -18,6 +20,7 @@
- pattern: 'https:'
- repl: 'http://HTTPS/'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% elif grains['os_family']|lower == 'arch' %}
pacman:
Expand All @@ -28,21 +31,24 @@
- pattern: 'https:'
- repl: 'http://HTTPS/'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% elif grains['os_family']|lower == 'redhat' %}
{% for repo in salt['file.find']('/etc/yum.repos.d/', name='*repo*') %}
{{ repo }}_baseurl:
file.replace:
- name: {{ repo }}
- pattern: 'baseurl=https://'
- repl: 'baseurl=http://HTTPS///'
- pattern: 'baseurl(.*)https://'
- repl: 'baseurl\1http://HTTPS///'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False
{{ repo }}_metalink:
file.replace:
- name: {{ repo }}
- pattern: 'metalink=https://(.*)basearch'
- repl: 'metalink=http://HTTPS///\1basearch&protocol=http'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% endfor %}
{% endif %}
10 changes: 8 additions & 2 deletions cacher/restore_templates.sls
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
- pattern: 'http://HTTPS/'
- repl: 'https:'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False
{% endfor %}

/etc/apt/sources.list:
Expand All @@ -19,6 +20,7 @@
- pattern: 'http://HTTPS/'
- repl: 'https:'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% elif grains['os_family']|lower == 'arch' %}
pacman:
Expand All @@ -29,22 +31,26 @@
- pattern: 'http://HTTPS///'
- repl: 'https://'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False


{% elif grains['os_family']|lower == 'redhat' %}
{% for repo in salt['file.find']('/etc/yum.repos.d/', name='*repo*') %}
{{ repo }}_baseurl:
file.replace:
- name: {{ repo }}
- pattern: 'http://HTTPS/'
- repl: 'https:'
- pattern: 'baseurl(.*)http://HTTPS/'
- repl: 'baseurl\1https:'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{{ repo }}_metalink:
file.replace:
- name: {{ repo }}
- pattern: 'metalink=http://HTTPS///(.*)basearch&protocol=http'
- repl: 'metalink=https://\1basearch'
- flags: [ 'IGNORECASE', 'MULTILINE' ]
- backup: False

{% endfor %}
{% endif %}

0 comments on commit 3f59aac

Please sign in to comment.