reading files trhough NFS(version 3,4.1) get stuck

[avishayhatton]

Hi all,
a strange thing with new version (2.0.4/2.05):
Linux platform(aarch64)

1. we were working with 0.8.5 version as a shared storage - NFS 4.1 - read/write for 3 months.
2. we upgraded to 2.0.4(remove ZFS completley, reboot, then build the new zfs)
3. imported the pool back and restart nfs-server + zpool upgrade -a
4. when reading a file locally from a dataset on the ZFS storage, everything is ok. you can read the contents
5. when you read the file using NFS protocol(tried Version3 and version 4.1) - the read operation get stuck(less/cat tools)
   cat /mnt/folder/ap_dro_7p5t_all_VT.v

only CTRL-C helps to get back to prompt
6. the same file can be read from local ZFS storage.
7. some files in the same folder can be read from remote(NFS) and some not(get stuck with cat/less)
8. when i downgrade back to 0.8.5 (imported with read only mode) - all files can be read from remote(nfs)

Thanks
Avishay

[rincebrain]

What OS/release/kernel version the server and client are running may be relevant here,

Does anything show up in the server or client's dmesg?

You say 2.0.4/2.0.5 then only mention 2.0.4 in your story - do you mean that you have also since tested 2.0.5?

[rincebrain]

1 doesn't really surprise me
I don't understand what you're trying to communicate with 2 - copied from where, to where? Are you saying you can reliably reproduce this with a certain file even with a new copy of the file? On the same or different datasets?

[avishayhatton]

I tried to copy the file, i could not read through nfs with a new name and chnaged it's ownership and permissions, to see if it changes anything(able to read it from NFS). it didn't help.(cat/less file_name gets stuck)

of course, reading the file from the local ZFS storage is fine.

[avishayhatton]

i wanted to see if i copy a file i could not read through NFS with a new name + changed ownership/permission, would change anything.
it didn't work.less/cat get stuck with the new file name as well.
of course, reading the file from the local ZFS storage is fine.

[rincebrain]

Does anything show up in dmesg when this happens?

Can you tell me anything about the file? How large it is, whether the contents is lots of text or compressed video or /dev/urandom output, what compression algorithm is in use on the dataset, is encryption enabled and if so what algorithm...

If whatever the file is would allow it, since you said you could reproduce this on multiple machines, sharing it would probably make it easier to figure out what's going on. If you can't, that's perfectly fine, but it would likely speed things up.

[avishayhatton]

1. dmesg or tail -f /var/log/messages is clean(storage side)
2. the file is a text file(it is being compressed by ZFS lz4)
3. tried also zstd(created a new data set and copied the data over to the new data set)
4. Size: 41k(on disk) but original size is 345k(ls -lha file_name), record size is default=128k
5. no encryption enabled
6. i will try to re-produce it on intel X86_64

[avishayhatton]

Hi,
tested with 0.8.5.1 on x86_64

[root@duzfs-test2 ~]# arch
x86_64

filename: /lib/modules/4.14.238-182.421.amzn2.x86_64/extra/zfs/zfs/zfs.ko
version: 0.8.5-1

1. synced the data using zfs send/recieve (from the ZFS storage with version 2.1)
2. exported the data set using nfs
3. mounted from a client using nfs 3/4.1
4. all files can be read(simple loop to read all files in the same folder)

next step: upgrade to 2.0.5

1. removed zfs rpms and rebooted
2. downloaded tar(2.0.5) and compiled (rpms, rpms-dkms)
3. rebooted
4. import without upgrading pool version
5. reading get stuck.(cat, less)
6. dmesg -Tx, tail -f /var/log/messages - clean

[avishayhatton]

du -sh havishay-ap_dro_7p5t_all_VT.v
41K havishay-ap_dro_7p5t_all_VT.v

ls -hal havishay-ap_dro_7p5t_all_VT.v
-rwxr-xr-x 1 havishay anpa 345K Jul 20 11:10 havishay-ap_dro_7p5t_all_VT.v

[avishayhatton]

1. My mistake - i followed a soft link. Now i am using the real folder

2. also, i already checked using the full path(no soft link involved - and it gets stuck)
   Object lvl iblk dblk dsize dnsize lsize %full type
   75528 2 128K 128K 40K 512 384K 100.00 ZFS plain file
   176 bonus System attributes
   dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
   dnode maxblkid: 2
   uid 2491
   gid 2001
   atime Tue Jul 20 11:10:30 2021
   mtime Tue Jul 20 11:10:30 2021
   ctime Tue Jul 20 11:33:23 2021
   crtime Tue Jul 20 11:10:30 2021
   gen 115004
   mode 100755
   size 352812
   parent 2023
   links 1
   pflags 840800000104
   Indirect blocks:
   0 L1 0:6e63a8000:1000 20000L/1000P F=3 B=467/467 cksum=8cb0c8b930:1fbd80dab73cb:39787c30bbd9e34:598caf13dbc3899f
   0 L0 0:6e639f000:2000 20000L/2000P F=1 B=467/467 cksum=1bee61c8096:8253f61896133:16098fce0c7da8d0:faeaa299eb637540
   20000 L0 0:6e63a1000:4000 20000L/4000P F=1 B=467/467 cksum=31d06a20ebf:1a994ed6c71498:917773dd1267dfb4:f4ffa1d516e0e109
   40000 L0 0:6e63a5000:3000 20000L/3000P F=1 B=467/467 cksum=2804db7557a:d9f3de07f1d09:34b68ea93b5c4178:cfb7e2a0d16ddd1

            segment [0000000000000000, 0000000000060000) size  384K
   

[rincebrain]

That's curious.

It seems to reproduce readily if you set up an Amazon Linux 2 VM on VirtualBox and try reading a copy of the ZFS git tree (for me, it cropped up on config.log).

I wonder what's going on. Let's see...

[avishayhatton]

well,
tested it with EC2 instance + Centos 7.9. it is working fine with this release(centos 7.9 + zfs 2.0.5)

[rincebrain]

That makes sense, given what I'm about to remark.

It also happens with vanilla 4.14.240, even on loopback NFS mount. So that's entertaining.