Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support multiple volumes on virtual machines #2287

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b0bb651
chore(virtual-machine): Rearrange fields in Virtual Machine Volume
adityahase Nov 13, 2024
f178888
feat(volumes): Add Server Mount DocType
adityahase Nov 13, 2024
04e41dd
feat(volumes): Fetch volumes form Virtual Machine
adityahase Nov 13, 2024
d1f01d0
feat(volumes): Set mount options
adityahase Nov 13, 2024
21c1b53
feat(volumes): Set default mounts for app and db server
adityahase Nov 13, 2024
9cb3dd1
feat(volumes): Playbook to mount all volumes
adityahase Nov 13, 2024
33c52dd
fix(virtual-machine): Allow provisioning a VM that isn't linked to a …
adityahase Nov 13, 2024
7e6e702
fix(volumes): Create mount points before mounting volumes
adityahase Nov 13, 2024
ab6593c
fix(volumes): Format volumes before mounting
adityahase Nov 13, 2024
b89a1b3
fix(volumes): Set ServerMount.status based on ansible results
adityahase Nov 13, 2024
4f4e192
feat(volumes): Set MariaDB to depend on mounts when data volumes are …
adityahase Nov 13, 2024
2938b3d
feat(volumes): Set Docker to depend on mounts when data volumes are used
adityahase Nov 13, 2024
2322838
fix(aws): Attach a new volume to VM
adityahase Nov 13, 2024
89662a2
fix(ansible): Show progress bar for completed tasks
adityahase Nov 13, 2024
afa4c5a
fix(volumes): Set mount status to Pending by default
adityahase Nov 13, 2024
267feae
fix(volumes): Set mount status after setup_server
adityahase Nov 13, 2024
bba0587
fix(volumes): Create docker systemd drop-in directory
adityahase Nov 13, 2024
4330c7e
fix(ansible): Use frappe.utils.cstr for passing variables
adityahase Nov 13, 2024
40de9b0
fix(cluster): Use most recent image to create new servers
adityahase Nov 13, 2024
82374dc
fix(volumes): Fetch volumes only if VM has volumes
adityahase Nov 14, 2024
60b663b
Merge branch 'master' of github.com:frappe/press into feat-server-vol…
adityahase Nov 15, 2024
ba26979
fix(volumes): Set Server.mount[].uuid from the output of lsblk
adityahase Nov 15, 2024
b0d0926
fix(volumes): Use UUID to mount volumes instead of volume id
adityahase Nov 15, 2024
2cc492e
chore(volumes): Rearrange fields
adityahase Nov 15, 2024
2be77d3
Merge branch 'master' of github.com:frappe/press into feat-server-vol…
adityahase Nov 15, 2024
25821fa
fix(volumes): Add user,owner,mode options to mount points
adityahase Nov 15, 2024
2ace384
fix(volumes): Explicitly set default fields on mounts
adityahase Nov 18, 2024
886a43f
chore(volumes): Refactor mount_options
adityahase Nov 18, 2024
6d7accb
fix(volumes): MariaDB data volume is owned by root
adityahase Nov 18, 2024
a14e1df
fix(oci): Sync OCI machines in batches
adityahase Nov 18, 2024
a07b282
fix(oci): Typo
adityahase Nov 18, 2024
ca6a123
Merge branch 'master' of github.com:frappe/press into feat-server-vol…
adityahase Nov 18, 2024
889912f
fix(ansible): Update task progress for current play
adityahase Nov 18, 2024
36b2194
fix(volumes): Bind mount target ownerships are the same as owner
adityahase Nov 18, 2024
80fb4c8
fix(volumes): Only show available virtual machine images
adityahase Nov 18, 2024
c5f1776
fix(volumes): Make platform editable
adityahase Nov 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions press/playbooks/database.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
roles:
- role: essentials
- role: user
- role: mount
- role: mariadb
- role: nginx
- role: agent
Expand Down
8 changes: 8 additions & 0 deletions press/playbooks/mount.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
- name: Mount Volumes
hosts: all
become: yes
become_user: root
gather_facts: no
roles:
- role: mount
2 changes: 1 addition & 1 deletion press/playbooks/roles/agent/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
- name: Generate Agent Configuration File
become: yes
become_user: frappe
command: '/home/frappe/agent/env/bin/agent setup config --name {{ server }} --workers {{ workers }} {% if proxy_ip is defined %}--proxy-ip {{ proxy_ip }}{% endif %} {% if agent_sentry_dsn is defined and agent_sentry_dsn is truthy %}--sentry-dsn {{ agent_sentry_dsn }}{% endif %}'
command: '/home/frappe/agent/env/bin/agent setup config --name {{ server }} --workers {{ workers }} {% if proxy_ip is defined and proxy_ip is truthy %}--proxy-ip {{ proxy_ip }}{% endif %} {% if agent_sentry_dsn is defined and agent_sentry_dsn is truthy %}--sentry-dsn {{ agent_sentry_dsn }}{% endif %}'
args:
chdir: /home/frappe/agent

Expand Down
19 changes: 19 additions & 0 deletions press/playbooks/roles/docker/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,27 @@
src: daemon.json
dest: /etc/docker/daemon.json

- name: Create Docker SystemD drop-in directory
file:
dest: /etc/systemd/system/docker.service.d
state: directory
owner: root
group: root
mode: 0644
recurse: true

- name: Set Docker to depend on Mounts
template:
src: mounts.conf
dest: /etc/systemd/system/docker.service.d/mounts.conf
owner: root
group: root
mode: 0644
when: docker_depends_on_mounts | default(false) | bool

- name: Restart Docker Daemon
systemd:
daemon_reload: true
name: docker
state: restarted

Expand Down
14 changes: 14 additions & 0 deletions press/playbooks/roles/docker/templates/mounts.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[Unit]
# If Docker gets activated, then the mount will be activated as well.
# If the mount fails to activate, Docker will not be started.
# If the mount is explicitly stopped (or restarted), Docker will be stopped (or restarted).

# BindsTo imposes a stronger condition than RequiresTo.
# If the mount is stopped, Docker will be stopped too.

# When used in conjunction with After
# The mount strictly has to be in active state for Docker to also be in active state.
# Reference: https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#BindsTo=

After=home-frappe-benches.mount
BindsTo=home-frappe-benches.mount
9 changes: 9 additions & 0 deletions press/playbooks/roles/mariadb/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,15 @@
insertafter: '\[Service\]'
state: present

- name: Set MariaDB to depend on Mounts
template:
src: mounts.conf
dest: /etc/systemd/system/mariadb.service.d/mounts.conf
owner: root
group: root
mode: 0644
when: mariadb_depends_on_mounts | default(false) | bool

- name: Restart MariaDB Service
systemd:
daemon_reload: true
Expand Down
14 changes: 14 additions & 0 deletions press/playbooks/roles/mariadb/templates/mounts.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[Unit]
# If MariaDB gets activated, then mounts will be activated as well.
# If one of the mounts fails to activate, MariaDB will not be started.
# If one of the mounts is explicitly stopped (or restarted), MariaDB will be stopped (or restarted).

# BindsTo imposes a stronger condition than RequiresTo.
# If one of the mounts are stopped, MariaDB will be stopped too.

# When used in conjunction with After
# The mounts strictly have to be in active state for MariaDB to also be in active state.
# Reference: https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html#BindsTo=

After=etc-mysql.mount var-lib-mysql.mount
BindsTo=etc-mysql.mount var-lib-mysql.mount
55 changes: 55 additions & 0 deletions press/playbooks/roles/mount/tasks/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
---
- name: Set JSON Variables
set_fact:
all_mounts: '{{ all_mounts_json | from_json }}'
volume_mounts: '{{ volume_mounts_json | from_json }}'
bind_mounts: '{{ bind_mounts_json | from_json }}'

- name: Create Mount Points
file:
dest: "{{ item.mount_point }}"
state: directory
owner: "{{ item.mount_point_owner }}"
group: "{{ item.mount_point_group }}"
mode: "{{ item.mount_point_mode }}"
loop: "{{ all_mounts }}"

- name: Format Volumes
filesystem:
fstype: "{{ item.filesystem }}"
dev: "{{ item.source }}"
force: false
loop: "{{ volume_mounts }}"

- name: Show Block Device UUIDs
command: 'lsblk {{ item.source }} -no UUID'
loop: "{{ volume_mounts }}"
register: block_devices

- name: Mount Volumes
mount:
src: "UUID={{ item.stdout.strip() }}"
path: "{{ item.item.mount_point }}"
fstype: "{{ item.item.filesystem }}"
opts: "{{ item.item.mount_options }}"
state: mounted
loop: "{{ block_devices.results }}"

- name: Create Mount Source Directories
file:
dest: "{{ item.source }}"
state: directory
owner: "{{ item.mount_point_owner }}"
group: "{{ item.mount_point_group }}"
mode: "{{ item.mount_point_mode }}"
loop: "{{ bind_mounts }}"

- name: Mount Bind Mounts
mount:
src: "{{ item.source }}"
path: "{{ item.mount_point }}"
fstype: none
opts: "{{ item.mount_options }}"
state: mounted
loop: "{{ bind_mounts }}"

1 change: 1 addition & 0 deletions press/playbooks/server.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- role: user
- role: nginx
- role: agent
- role: mount
- role: bench
- role: docker
- role: node_exporter
Expand Down
17 changes: 15 additions & 2 deletions press/press/doctype/ansible_play/ansible_play.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@
// For license information, please see license.txt

frappe.ui.form.on('Ansible Play', {
// refresh: function(frm) {
// }
refresh: function (frm) {
frappe.realtime.on('ansible_play_progress', (data) => {
if (data.progress && data.play === frm.doc.name) {
const progress_title = __('Ansible Play Progress');
frm.dashboard.show_progress(
progress_title,
(data.progress / data.total) * 100,
`Ansible Play Progress (${data.progress} tasks completed out of ${data.total})`,
);
if (data.progress === data.total) {
frm.dashboard.hide_progress(progress_title);
}
}
});
},
});
6 changes: 6 additions & 0 deletions press/press/doctype/database_server/database_server.js
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,12 @@ frappe.ui.form.on('Database Server', {
true,
frm.doc.is_self_hosted,
],
[
__('Mount Volumes'),
'mount_volumes',
true,
frm.doc.virtual_machine && frm.doc.mounts,
],
].forEach(([label, method, confirm, condition]) => {
if (typeof condition === 'undefined' || condition) {
frm.add_custom_button(
Expand Down
15 changes: 14 additions & 1 deletion press/press/doctype/database_server/database_server.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@
"column_break_apox",
"tags_section",
"tags",
"mounts_section",
"mounts",
"mariadb_settings_tab",
"memory_limits_section",
"memory_high",
Expand Down Expand Up @@ -517,11 +519,22 @@
"fieldtype": "Int",
"label": "Auto Add Storage Max",
"non_negative": 1
},
{
"fieldname": "mounts_section",
"fieldtype": "Section Break",
"label": "Mounts"
},
{
"fieldname": "mounts",
"fieldtype": "Table",
"label": "Mounts",
"options": "Server Mount"
}
],
"index_web_pages_for_search": 1,
"links": [],
"modified": "2024-08-13 11:02:07.399141",
"modified": "2024-10-28 18:05:38.289733",
"modified_by": "Administrator",
"module": "Press",
"name": "Database Server",
Expand Down
Loading
Loading