Bash and script to flush writeback cache in OS suspend and shutdown

[Augusto7743]

Hello.
I have used Rapiddisk for writeback cache since version 7.2 in Ubuntu 20.04.
install_initrd.sh does an good task creating commands in OS start thus loading Rapiddisk modules creating the disk cache.
If user choose writeback in root need manually run dmsetup to flush data before OS shutdown.

I had created bash files and systemd service to flush writeback cache when OS enter suspend mode and shutdown thus avoiding data loss and file system damage.

The bash file is

====
#!/bin/bash
sleep 6s
/usr/sbin/dmsetup message /dev/mapper/rc-wb_sdaX 0 flush

exit 0

BTRFS default commit is 6 seconds.
Thus being "sleep" good value is 6.
Path to save the bash is
/usr/local/bin/rapiddisk.sh
After
chmod u+x /usr/local/bin/rapiddisk.sh

The systemd service

====
[Unit]
Description=RapidDisk Writeback Cache Flush
DefaultDependencies=no
Before=sleep.target umount.target

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/usr/local/bin/rapiddisk.sh
TimeoutStartSec=0

[Install]
WantedBy=sleep.target umount.target

Path to save the systemd service
/etc/systemd/system/rapiddisk.service

After enable the systemd service
sudo systemctl enable rapiddisk-flush

Tested since 7.2 to current version correctly working.

Is possible create more caches editing the install_initrd.sh adding commands in end of file.

===
fi
rapiddisk -a 128
rapiddisk -m rd1 -b /dev/sdaX -p wb
log_success_msg "rapiddisk: RAMDISKSIZE MB ramdisk attached to BOOTDEVICE successfully."
log_end_msg
exit 0

Example above 2 caches will be created. One being the default from the install_initrd.sh and the second the manually added in install_initrd.sh need edit the bash rapiddisk.sh.
The problem is if creating caches for others mount point example "home" "opt" or others disks when the OS is starting.
The cache will be created, but the systemd service writeback only work for suspend and shutdown.
Others mounts points than root are unmounted before root umount.
Thus need add in systemd service an configuration to run the bash file before "File System Pre" is being started because others mounts points will be unmounted not being possible flush the writeback data with high risk file system damage.
In moment I not known the name to "File System Pre" target to be added in systemd service.
When using more than one writeback cache I run manually an bash to flush all caches before of full flush in OS shutdown. Not doing that damage the file system. I saw that happen very much using BTRFS using kernel 5.10 in OS crash.
In moment testing writeback using kernel 5.17 without problems.

Would be good the install_initrd.sh allow create more than one default cache and detect if was created writeback cache automatically create one bash file with exact number of created caches and the systemd service.

[Augusto7743]

Today have happened the problem partial flush to disk using BTRFS and the file system was damaged, but is how new kernel modules have any BTRFS updates was possible easily recover the damaged partition.
I recommed update to kernel 5.17 if you use Ubuntu 20.04. Avoid 5.13 because have several users reporting power suspend errors.
Not tested in 22.04 yet.

[pkoutoupis]

@Augusto7743 Any way you can do a pull request and place these files in a subdirectory called scripts/rapiddisk-init with a README.md give a description of what each does, how to install and possibly a disclaimer of what environment (i.e. kernel) to use it in? That way we can do a more formal review, tweak the code if necessary and once everyone is happy, it can merge into master.

[matteotenca]

@Augusto7743 @pkoutoupis

I believe that a very final flush via systemd services is a problem.
From the docs:

System shutdown with systemd also consists of various target units with
some minimal ordering structure applied:

                                             (conflicts with  (conflicts with
                                               all system     all file system
                                                services)     mounts, swaps,
                                                    |           cryptsetup/
                                                    |           veritysetup
                                                    |          devices, ...)
                                                    |                |
                                                    v                v
                                             shutdown.target    umount.target
                                                    |                |
                                                    \_______   ______/
                                                            \ /
                                                             v
                                                    (various low-level
                                                         services)
                                                             |
                                                             v
                                                       final.target
                                                             |
                       _____________________________________/ \_________________________________
                      /                         |                        |                      \
                      |                         |                        |                      |
                      v                         v                        v                      v
           systemd-reboot.service   systemd-poweroff.service   systemd-halt.service   systemd-kexec.service
                      |                         |                        |                      |
                      v                         v                        v                      v
               reboot.target             poweroff.target            halt.target           kexec.target

And from here:

Immediately before executing the actual system halt/poweroff/reboot/kexec systemd-shutdown will
run all executables in /usr/lib/systemd/system-shutdown/ and pass one arguments to them: either
"halt", "poweroff", "reboot" or "kexec", depending on the chosen action. All executables in this
directory are executed in parallel, and execution of the actionis not continued before all
executables finished.

Under Ubuntu, all the executables in /lib/systemd/system-shutdown are executed. Creating the script suggested by @Augusto7743 in that folder, as for example rapiddisk-flush.sh:

#!/bin/sh
/sbin/dmsetup message /dev/mapper/rc-wb_sda1 0 flush

I achieved the "final" flush. When the debug option is passed to the kernel, it is possible to see the result of the fsck operation performed during boot:

$ sudo cat /run/initramfs/fsck.log
Log of fsck -a -V -t ext4 /dev/mapper/rc-wb_sda1
Fri Jun  3 20:08:05 2022

fsck from util-linux 2.34
[/usr/sbin/fsck.ext4 (1) -- /dev/mapper/rc-wb_sda1] fsck.ext4 -a /dev/mapper/rc-wb_sda1
/dev/mapper/rc-wb_sda1: clean, 573736/1381744 files, 4202028/5516800 blocks

Fri Jun  3 20:08:05 2022
----------------
$

I tried to remove the script and the result changed as this:

$ sudo cat /run/initramfs/fsck.log
Log of fsck -a -V -t ext4 /dev/mapper/rc-wb_sda1
Fri Jun  3 20:06:50 2022

fsck from util-linux 2.34
[/usr/sbin/fsck.ext4 (1) -- /dev/mapper/rc-wb_sda1] fsck.ext4 -a /dev/mapper/rc-wb_sda1
/dev/mapper/rc-wb_sda1: recovering journal
/dev/mapper/rc-wb_sda1: clean, 573740/1381744 files, 4193376/5516800 blocks

Fri Jun  3 20:06:50 2022
----------------

As you can see the volume needed recovering. So this solution may work.

I only tried this on a virtual machine like this:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Linux gturismo 5.4.0-110-generic #124-Ubuntu SMP Thu Apr 14 19:46:19 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[Augusto7743]

@matteotenca
Thanks very much adding all information above.
One good solution is create an script to run before "File System Pre" is being stopped.
The script need to run before the OS is unmonting all file systems not being "/".

[pkoutoupis]

@Augusto7743 Again, if you can do a pull request and place these files in a subdirectory called scripts/rapiddisk-init with a README.md that gives a description of what each does, we can start officially reviewing it and if/when it looks good, merge it into master.

[Augusto7743]

Not being possible add script.zip in "rapiddisk pull request". Using waterfox. Perhaps not compatible browser ?
The script and systemd.
Please add and review the script.
1.zip

[matteotenca]

@Augusto7743 I tried with the systemd approach (i.e. using services), with your files and with many variants found around the web, but the root filesystem with a rapiddisk device attached in writeback mode always ends up unclean on next reboot. The only working solution I found is to put the flushing script in /lib/systemd/system-shutdown under Ubuntu 20.04.4 LTS.

Other writeback caches on other volumes may be flushed just before unmounting, but I believe the root mount cannot be properly flushed before the very last moment, by systemd targets/services. And the last thing the system does before reboot/shutdown is executing the contents of /lib/systemd/system-shutdown.

The sleep event is different story, maybe an appropriate systemd target to trigger the service which flush the cache may be found, I did not tried.

[Augusto7743]

Thanks very much for your reply.

The files work here with Lubuntu (Ubuntu using LXQT) 20.04.4 LTS.

"Other writeback caches on other volumes may be flushed just before unmounting"
Need create an target to the correct systemd service when the system is being restarted or shutdown.
Is possible using an target of any systemctl before unmounting others volumes work.
I will test it.

"I believe the root mount cannot be properly flushed before the very last moment, by systemd targets/services. And the last thing the system does before reboot/shutdown is executing the contents of /lib/systemd/system-shutdown."
I have used the files for an long time and work.
All cached data is correctly flushed in root mount.

"sleep event is different story, maybe an appropriate systemd target to trigger the service which flush the cache may be found, I did not tried."
Sleep event will flush for all devices without problems.

In moment need figure what the correct name for "File System Pre" (systemd service for unmount others devices) or an service before "File System Pre" and use it in systemd service.
After the user need manually edit the flush.sh or your bash file automatically create the correct commands lines in the flush.sh

[mhtvsSFrpHdE]

Any progress on this topic? Does rapiddisk support auto flush on shutdown/reboot right now?
If so, what is the command or config file should I touch?

[Augusto7743]

Not support for auto flush.
Need create an bash script to point to run before an "target" is closed before shutdown.
Need figure what is the name of "File System Pre" and thus will be possible flush for all devices correctly.
About flush in suspend and stand by is simple to do. Read the instructions in begin of that page. Good luck.