Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add custom-persist extension #650

Merged
merged 3 commits into from
Mar 1, 2025
Merged

add custom-persist extension #650

merged 3 commits into from
Mar 1, 2025

Conversation

Guiiix
Copy link
Member

@Guiiix Guiiix commented Jan 28, 2025

custom-persist extension (see QubesOS/qubes-issues#1006)

Adds a custom-persist feature. The service custom-persist must be activated to enable the feature.
When the feature is enabled, user custom persistent file and dirs must be configured with qvm-features. Example: qvm-features work custom-persist.var_test /var/test

@DemiMarie
Copy link
Contributor

@Guiiix Is there a companion PR to core-agent-linux?

@Guiiix
Copy link
Member Author

Guiiix commented Jan 29, 2025

Hi @DemiMarie, yes this one: QubesOS/qubes-core-agent-linux#551

I'll fix lint errors today.

@Guiiix Guiiix force-pushed the custom-persist branch 4 times, most recently from 68bcfbc to 2d64eae Compare February 2, 2025 10:18
@codecov Codecov
Copy link

codecov bot commented Feb 2, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 98.46154% with 1 line in your changes missing coverage. Please review.

Project coverage is 69.89%. Comparing base (c1cdf38) to head (78809ab).
Report is 27 commits behind head on main.

Files with missing lines Patch % Lines
qubes/ext/custom_persist.py 98.46% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #650      +/-   ##
==========================================
+ Coverage   69.71%   69.89%   +0.18%     
==========================================
  Files          58       59       +1     
  Lines       12499    12607     +108     
==========================================
+ Hits         8714     8812      +98     
- Misses       3785     3795      +10
Flag Coverage Δ
unittests 69.89% <98.46%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@marmarek
Copy link
Member

marmarek commented Feb 3, 2025

Please add some unit test

@Guiiix Guiiix force-pushed the custom-persist branch 5 times, most recently from 1f0c3ec to 929a076 Compare February 7, 2025 11:05
@Guiiix
Copy link
Member Author

Guiiix commented Feb 7, 2025

Tests added

@marmarek
Copy link
Member

I'll also add an integration test. In the meantime, can you update documentation at https://www.qubes-os.org/doc/bind-dirs/ to include this feature too? Or maybe you see some better place for that?

This was referenced Feb 13, 2025
Merged
Draft
Merged
Merged
Merged
Merged
Merged
@qubesos-bot
Copy link

qubesos-bot commented Feb 13, 2025 •
edited
Loading

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025022819-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025021804-4.3&flavor=update

  • system_tests_basic_vm_qrexec_gui

  • system_tests_extra

    • TC_00_Direct_whonix-workstation-17: test_031_sign_verify_detached (failure)
      AssertionError: subkey generation failed: b''b'2025-02-28 16:24:38....

  • system_tests_basic_vm_qrexec_gui_zfs

  • system_tests_audio

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_suspend@hw1

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/p5~T5-\d+-/...

  • system_tests_basic_vm_qrexec_gui@hw7

  • system_tests_kde_gui_interactive

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-vm-settings' mat...

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'work-text-editor' mat...

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_basic_vm_qrexec_gui_ext4

Failed tests

25 failures

  • system_tests_basic_vm_qrexec_gui

  • system_tests_extra

    • TC_00_Direct_whonix-workstation-17: test_031_sign_verify_detached (failure)
      AssertionError: subkey generation failed: b''b'2025-02-28 16:24:38....

  • system_tests_basic_vm_qrexec_gui_zfs

  • system_tests_audio

  • system_tests_basic_vm_qrexec_gui_btrfs

  • system_tests_suspend@hw1

    • suspend: wait_serial (wait serial expected)
      # wait_serial expected: qr/p5~T5-\d+-/...

    • suspend: Failed (test died + timed out)
      # Test died: command 'true' timed out at /usr/lib/os-autoinst/autot...

  • system_tests_basic_vm_qrexec_gui@hw7

  • system_tests_kde_gui_interactive

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-vm-settings' mat...

    • simple_gui_apps: unnamed test (unknown)

    • simple_gui_apps: wait_serial (wait serial expected)
      # wait_serial expected: qr/2E8vz-\d+-/...

  • system_tests_guivm_vnc_gui_interactive

    • gui_filecopy: unnamed test (unknown)
    • gui_filecopy: Failed (test died)
      # Test died: no candidate needle with tag(s) 'work-text-editor' mat...

  • system_tests_basic_vm_qrexec_gui_xfs

  • system_tests_basic_vm_qrexec_gui_ext4

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/129058#dependencies

14 fixed

  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_suspend

    • suspend: unnamed test (unknown)
    • suspend: Failed (test died)
      # Test died: no candidate needle with tag(s) 'SUSPEND-FAILED' match...

  • system_tests_qrexec

  • system_tests_audio

  • system_tests_whonix@hw7

    • whonixcheck: fail (unknown)
      Whonixcheck for sys-whonix failed...

    • whonixcheck: unnamed test (unknown)

  • system_tests_basic_vm_qrexec_gui_btrfs

    • TC_03_QvmRevertTemplateChanges: test_000_revert_linux (error)
      subprocess.CalledProcessError: Command '['sha1sum', '/var/lib/qubes...

  • system_tests_suspend@hw1

  • system_tests_kde_gui_interactive

    • clipboard_and_web: unnamed test (unknown)
    • clipboard_and_web: Failed (test died)
      # Test died: no candidate needle with tag(s) 'clipboard-paste-notif...

Unstable tests

Performance Tests

Performance degradation:

29 performance degradations
  • debian-12-xfce_exec: 8.31 :small_red_triangle_up: ( previous job: 7.15, degradation: 116.24%)
  • debian-12-xfce_exec-data-simplex: 68.47 :small_red_triangle_up: ( previous job: 48.93, degradation: 139.94%)
  • debian-12-xfce_exec-data-duplex: 69.55 :small_red_triangle_up: ( previous job: 50.76, degradation: 137.01%)
  • debian-12-xfce_exec-data-duplex-root: 85.79 :small_red_triangle_up: ( previous job: 64.91, degradation: 132.17%)
  • debian-12-xfce_socket-data-duplex: 166.11 :small_red_triangle_up: ( previous job: 81.49, degradation: 203.84%)
  • fedora-41-xfce_exec-data-simplex: 73.38 :small_red_triangle_up: ( previous job: 49.65, degradation: 147.80%)
  • fedora-41-xfce_exec-data-duplex: 72.91 :small_red_triangle_up: ( previous job: 49.08, degradation: 148.55%)
  • fedora-41-xfce_exec-data-duplex-root: 111.80 :small_red_triangle_up: ( previous job: 81.65, degradation: 136.92%)
  • fedora-41-xfce_socket-data-duplex: 152.37 :small_red_triangle_up: ( previous job: 78.62, degradation: 193.80%)
  • whonix-gateway-17_exec-data-simplex: 70.54 :small_red_triangle_up: ( previous job: 48.76, degradation: 144.65%)
  • whonix-gateway-17_exec-data-duplex: 78.20 :small_red_triangle_up: ( previous job: 48.55, degradation: 161.06%)
  • whonix-gateway-17_exec-data-duplex-root: 101.18 :small_red_triangle_up: ( previous job: 70.13, degradation: 144.28%)
  • whonix-gateway-17_socket-data-duplex: 161.79 :small_red_triangle_up: ( previous job: 82.74, degradation: 195.54%)
  • whonix-workstation-17_exec-data-simplex: 65.81 :small_red_triangle_up: ( previous job: 47.01, degradation: 139.97%)
  • whonix-workstation-17_exec-data-duplex: 70.89 :small_red_triangle_up: ( previous job: 49.48, degradation: 143.28%)
  • whonix-workstation-17_exec-data-duplex-root: 92.80 :small_red_triangle_up: ( previous job: 79.93, degradation: 116.10%)
  • whonix-workstation-17_socket-data-duplex: 157.93 :small_red_triangle_up: ( previous job: 81.71, degradation: 193.28%)
  • dom0_root_seq1m_q8t1_read 3:read_bandwidth_kb: 270530.00 :small_red_triangle_up: ( previous job: 486352.00, degradation: 55.62%)
  • dom0_root_seq1m_q8t1_write 3:write_bandwidth_kb: 184205.00 :small_red_triangle_up: ( previous job: 276742.00, degradation: 66.56%)
  • dom0_root_seq1m_q1t1_read 3:read_bandwidth_kb: 63859.00 :small_red_triangle_up: ( previous job: 423495.00, degradation: 15.08%)
  • dom0_root_seq1m_q1t1_write 3:write_bandwidth_kb: 25113.00 :small_red_triangle_up: ( previous job: 185030.00, degradation: 13.57%)
  • dom0_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 12717.00 :small_red_triangle_up: ( previous job: 100699.00, degradation: 12.63%)
  • dom0_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 1542.00 :small_red_triangle_up: ( previous job: 3277.00, degradation: 47.06%)
  • fedora-41-xfce_root_rnd4k_q32t1_write 3:write_bandwidth_kb: 2168.00 :small_red_triangle_up: ( previous job: 3785.00, degradation: 57.28%)
  • fedora-41-xfce_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 810.00 :small_red_triangle_up: ( previous job: 1126.00, degradation: 71.94%)
  • fedora-41-xfce_private_rnd4k_q32t1_write 3:write_bandwidth_kb: 2797.00 :small_red_triangle_up: ( previous job: 3885.00, degradation: 71.99%)
  • fedora-41-xfce_private_rnd4k_q1t1_write 3:write_bandwidth_kb: 1228.00 :small_red_triangle_up: ( previous job: 1613.00, degradation: 76.13%)
  • fedora-41-xfce_volatile_seq1m_q8t1_read 3:read_bandwidth_kb: 337162.00 :small_red_triangle_up: ( previous job: 392725.00, degradation: 85.85%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_write 3:write_bandwidth_kb: 1018.00 :small_red_triangle_up: ( previous job: 2693.00, degradation: 37.80%)

Remaining performance tests:

43 tests
  • debian-12-xfce_exec-root: 28.90 :small_red_triangle_up: ( previous job: 27.97, degradation: 103.32%)
  • debian-12-xfce_socket: 7.77 🟢 ( previous job: 8.33, improvement: 93.32%)
  • debian-12-xfce_socket-root: 8.84 :small_red_triangle_up: ( previous job: 8.20, degradation: 107.79%)
  • fedora-41-xfce_exec: 9.50 :small_red_triangle_up: ( previous job: 9.13, degradation: 104.05%)
  • fedora-41-xfce_exec-root: 62.94 :small_red_triangle_up: ( previous job: 61.17, degradation: 102.90%)
  • fedora-41-xfce_socket: 8.70 :small_red_triangle_up: ( previous job: 8.66, degradation: 100.43%)
  • fedora-41-xfce_socket-root: 8.70 :small_red_triangle_up: ( previous job: 8.61, degradation: 101.04%)
  • whonix-gateway-17_exec: 6.40 🟢 ( previous job: 7.87, improvement: 81.34%)
  • whonix-gateway-17_exec-root: 38.57 :small_red_triangle_up: ( previous job: 38.36, degradation: 100.54%)
  • whonix-gateway-17_socket: 7.01 🟢 ( previous job: 7.54, improvement: 92.86%)
  • whonix-gateway-17_socket-root: 7.50 🟢 ( previous job: 8.27, improvement: 90.69%)
  • whonix-workstation-17_exec: 8.07 🟢 ( previous job: 8.23, improvement: 98.12%)
  • whonix-workstation-17_exec-root: 54.08 :small_red_triangle_up: ( previous job: 52.56, degradation: 102.88%)
  • whonix-workstation-17_socket: 8.10 🟢 ( previous job: 8.21, improvement: 98.66%)
  • whonix-workstation-17_socket-root: 8.17 🟢 ( previous job: 8.20, improvement: 99.66%)
  • dom0_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 11576.00 :green_circle: ( previous job: 10163.00, improvement: 113.90%)
  • dom0_root_rnd4k_q1t1_write 3:write_bandwidth_kb: 672.00 :green_circle: ( previous job: 282.00, improvement: 238.30%)
  • dom0_varlibqubes_seq1m_q8t1_read 3:read_bandwidth_kb: 468742.00 :small_red_triangle_up: ( previous job: 475329.00, degradation: 98.61%)
  • dom0_varlibqubes_seq1m_q8t1_write 3:write_bandwidth_kb: 97087.00 :green_circle: ( previous job: 95209.00, improvement: 101.97%)
  • dom0_varlibqubes_seq1m_q1t1_read 3:read_bandwidth_kb: 433116.00 :small_red_triangle_up: ( previous job: 433474.00, degradation: 99.92%)
  • dom0_varlibqubes_seq1m_q1t1_write 3:write_bandwidth_kb: 169543.00 :green_circle: ( previous job: 164133.00, improvement: 103.30%)
  • dom0_varlibqubes_rnd4k_q32t1_read 3:read_bandwidth_kb: 104366.00 :green_circle: ( previous job: 99808.00, improvement: 104.57%)
  • dom0_varlibqubes_rnd4k_q32t1_write 3:write_bandwidth_kb: 8183.00 :small_red_triangle_up: ( previous job: 8767.00, degradation: 93.34%)
  • dom0_varlibqubes_rnd4k_q1t1_read 3:read_bandwidth_kb: 7698.00 :green_circle: ( previous job: 7053.00, improvement: 109.15%)
  • dom0_varlibqubes_rnd4k_q1t1_write 3:write_bandwidth_kb: 4422.00 :green_circle: ( previous job: 3868.00, improvement: 114.32%)
  • fedora-41-xfce_root_seq1m_q8t1_read 3:read_bandwidth_kb: 379643.00 :small_red_triangle_up: ( previous job: 396586.00, degradation: 95.73%)
  • fedora-41-xfce_root_seq1m_q8t1_write 3:write_bandwidth_kb: 149953.00 :green_circle: ( previous job: 99783.00, improvement: 150.28%)
  • fedora-41-xfce_root_seq1m_q1t1_read 3:read_bandwidth_kb: 310965.00 :small_red_triangle_up: ( previous job: 343795.00, degradation: 90.45%)
  • fedora-41-xfce_root_seq1m_q1t1_write 3:write_bandwidth_kb: 44317.00 :small_red_triangle_up: ( previous job: 44770.00, degradation: 98.99%)
  • fedora-41-xfce_root_rnd4k_q32t1_read 3:read_bandwidth_kb: 86714.00 :small_red_triangle_up: ( previous job: 86742.00, degradation: 99.97%)
  • fedora-41-xfce_root_rnd4k_q1t1_read 3:read_bandwidth_kb: 8588.00 :small_red_triangle_up: ( previous job: 8623.00, degradation: 99.59%)
  • fedora-41-xfce_private_seq1m_q8t1_read 3:read_bandwidth_kb: 414129.00 :green_circle: ( previous job: 401907.00, improvement: 103.04%)
  • fedora-41-xfce_private_seq1m_q8t1_write 3:write_bandwidth_kb: 255252.00 :green_circle: ( previous job: 116848.00, improvement: 218.45%)
  • fedora-41-xfce_private_seq1m_q1t1_read 3:read_bandwidth_kb: 352581.00 :small_red_triangle_up: ( previous job: 357875.00, degradation: 98.52%)
  • fedora-41-xfce_private_seq1m_q1t1_write 3:write_bandwidth_kb: 80465.00 :green_circle: ( previous job: 41375.00, improvement: 194.48%)
  • fedora-41-xfce_private_rnd4k_q32t1_read 3:read_bandwidth_kb: 86812.00 :small_red_triangle_up: ( previous job: 87999.00, degradation: 98.65%)
  • fedora-41-xfce_private_rnd4k_q1t1_read 3:read_bandwidth_kb: 8301.00 :small_red_triangle_up: ( previous job: 8744.00, degradation: 94.93%)
  • fedora-41-xfce_volatile_seq1m_q8t1_write 3:write_bandwidth_kb: 189909.00 :green_circle: ( previous job: 139933.00, improvement: 135.71%)
  • fedora-41-xfce_volatile_seq1m_q1t1_read 3:read_bandwidth_kb: 298145.00 :green_circle: ( previous job: 294875.00, improvement: 101.11%)
  • fedora-41-xfce_volatile_seq1m_q1t1_write 3:write_bandwidth_kb: 102052.00 :green_circle: ( previous job: 78093.00, improvement: 130.68%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_read 3:read_bandwidth_kb: 90327.00 :green_circle: ( previous job: 71108.00, improvement: 127.03%)
  • fedora-41-xfce_volatile_rnd4k_q32t1_write 3:write_bandwidth_kb: 3978.00 :green_circle: ( previous job: 3959.00, improvement: 100.48%)
  • fedora-41-xfce_volatile_rnd4k_q1t1_read 3:read_bandwidth_kb: 8159.00 :small_red_triangle_up: ( previous job: 8408.00, degradation: 97.04%)

This was referenced Feb 13, 2025
Open
Merged
Merged
@qubesos-bot qubesos-bot mentioned this pull request Feb 17, 2025
Merged
@Guiiix Guiiix force-pushed the custom-persist branch 3 times, most recently from e82a215 to 2b4d408 Compare February 19, 2025 23:08
This was referenced Feb 20, 2025
Merged
Merged
Merged
Merged
Merged
Merged
Open
Merged
Merged
Draft
Merged
Merged
Merged
Merged
Open
Merged
Merged
Open
Merged
Merged
Merged
@qubesos-bot qubesos-bot mentioned this pull request Feb 28, 2025
Merged
@marmarek marmarek merged commit b2041b8 into QubesOS:main Mar 1, 2025
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marmarek marmarek marmarek approved these changes

Assignees
No one assigned
Labels
openqa-pending
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Guiiix @DemiMarie @marmarek @qubesos-bot