glusterd.vol Missing: 'ERROR: parsing the volfile failed'

[michelamarie]

Hello! I've experienced a strange problem, which I have just resolved. I'm posting about it here so that I can hopefully learn why it happened, and how to avoid it in the future.

Summary of problem:
Essentially, the vol file (glusterd.vol) disappeared from one of my Gluster nodes, and I've no idea how or why that happened. This, of course, prevented that Gluster node from starting.

'journalctl -xe' output for glusterd:

Jun 29 07:57:18 host_name systemd[1]: Starting GlusterFS, a clustered file-system server...
-- Subject: Unit glusterd.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit glusterd.service has begun starting up.
Jun 29 07:57:18 host_name **GlusterFS[3560]: [glusterfsd.c:2062:parse_cmdline] 0-glusterfs: ERROR: parsing the volfile failed** [{errno=2}, {error=No such file or directory}]
Jun 29 07:57:18 host_name glusterd[3560]: USAGE: /usr/sbin/glusterd [options] [mountpoint]
Jun 29 07:57:18 host_name systemd[1]: glusterd.service: Control process exited, code=exited status=255
Jun 29 07:57:18 host_name systemd[1]: glusterd.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit glusterd.service has entered the 'failed' state with result 'exit-code'.
Jun 29 07:57:18 host_name systemd[1]: Failed to start GlusterFS, a clustered file-system server.
-- Subject: Unit glusterd.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit glusterd.service has failed.
--
-- The result is failed.

Environment details:
OS: Rocky Linux 8.6
Gluster version: 9.4, from the official CentOS Gluster 9 yum repositories
Hardware specifications: 2 vCPU, 7 GB RAM
This is a two-node Gluster cluster. I am aware of the risk of split-brain with this configuration.

Events preceding issue in question:
The only thing that's changed recently is that I resized the VM that this cluster node runs on. I am running a Gluster cluster in an OpenStack cloud platform, and issued a 'resize' command on the VM in question. When the VM came back online, Gluster would not start, and presented that missing vol file error mentioned above when running 'journalctl -xe'. I checked /etc/glusterfs/ and 'glusterd.vol' was indeed missing. Fortunately, the vol file on the other Gluster node was present. So I copied the contents of that file to a new one on the node missing the vol file. I tried starting glusterd again via systemd, and it worked instantly. Now everything appears to be fine.

I'm not sure when, exactly, the vol file disappeared, though of course, I noticed it after the reboot and subsequent glusterd unit file start attempt I've mentioned. I am quite sure no one deleted the vol file interactively.

I have searched high and low -- in the official Gluster documentation, in this forum, and elsewhere on the web -- for situations like this experienced by others. I haven't been able to find any instances of this problem anywhere else. Bizarre!

Thank you for any insight you may be able to provide on this situation!!

Michela