Cannot copy to dataset using rclone or robocopy #389
Comments
|
I have just run FileSpy, and it seems like the cause is both rclone and robocopy use Robocopy |
thesn10
changed the title
|
Hey, interesting but probably not the issue. All the FASTIO calls, if they fail, it will then go and do a full IRP call instead. There is a FASTIO_QUERY_OPEN but it can decide to fail it, if needed, and I think STATUS_REPARSE is one of those. |
|
Also when you FileSpy, you need to add Z: as you have, but also add path of the reparse point Z:/subfolder so you can see requests sent there. |
Done. |
|
Yeah that is it. FILE_OVERWRITE_IF and PATH_NOT_FOUND hmm. FILE_OVERWRITE_IF: So that seems wrong, we should create the file, right |
|
@lundman Yeah true, but also in line 21, |
|
Line 21 seem ok, it's checking if it is already there, to see if it needs updating, overwrite etc. But (I am guessing) the file does not exist, so the error is expected, and it gets ready to copy. |
|
Rclone is even more weird, there seems to be buffer overflows and collisions |
|
Nah that is fine, they call the function with a 0-size buffer, we reply with BUFFER_OVERFLOW and how much space it would need, they should call it again with bigger buffer. |
thesn10
changed the title
|
OK, so ZFS is on E: and E:\dataset. NTFS ZFS So interestingly, NTFS Should be equivalent. Differs here: NTFS: ZFS: It is peculiar it tries to create |
|
Whatever decisions were made, NTFS will and because it is ZFS we only ever get |
|
So this is a bug on rclones side because they do not handle the reparse correctly and tries to create |
|
@lundman Take a look at this part of the rclone source code: I think The My theory is that it cannot parse the result of the ZFS drivers |
|
Nah, it's ZFS fault, we do something wrong. Just need to figure out why. Yeah I'll check that, change how FASTIO_QUERY_NETWORK_OPEN_INFO return errors. |
|
OK note to self: Disable FastIO for ntfs so we can more easily compare FileSpy trace outputs Create and mount NTFS on E: and E:\dataset, Trace |
|
Take out same functions from ZFS Create pool on E: and lower dataset E:\dataset Trace |
|
Compare the two
|
|
First real difference is that NTFS gets the symlink name Makes me wonder where that comes from, when it gets |
Hmm, on my pc, there is no real difference and ZFS will also return a device path in ZFS returns Looks ok to me |
|
I'm currently trying to correct the "half mounted" setup we get with sub-mounts, in that they don't show properly in I have exhausted chatgpt support, trying here next: https://community.osr.com/t/mountmgr-reparsepoint-mount/58862 |
|
Does anyone happen to know if win-btrfs, or dokan, (both sources on GH - or any other GH program) when mounted on a volume (f.e.x E:) - bonus as a reparse point (E:\dataset) - will show in the output of Starting to think chatgpt doesn't know how to do the mounts either, some of the suggestions are starting to sound weird, like animal sacrifice etc. |
1 similar comment
|
Does anyone happen to know if win-btrfs, or dokan, (both sources on GH - or any other GH program) when mounted on a volume (f.e.x E:) - bonus as a reparse point (E:\dataset) - will show in the output of Starting to think chatgpt doesn't know how to do the mounts either, some of the suggestions are starting to sound weird, like animal sacrifice etc. |
Did you try mounting it from userland, the manual way? > mountvol E:\dataset \??\Volume{df46d39b-2857-11ef-82bd-58961d57db67}
For me at least Winbtrfs does. I have a ~ 38 GiB FS mounted as BTRFS (disk 1, partition 5):
... and it's presented by C:\Windows\system32> echo LIST VOL | diskpart
Microsoft DiskPart version 10.0.19041.964
Copyright (C) Microsoft Corporation.
On computer: DESKTOP-O7JE7JE
DISKPART>
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 D New Volume NTFS Partition 931 GB Healthy
Volume 1 C pagrindinis NTFS Partition 237 GB Healthy Boot
Volume 2 Partition 38 GB Healthy
Volume 3 NTFS Volume NTFS Partition 188 GB Healthy
D:\Another-volume\
Volume 4 FAT32 Partition 100 MB Healthy System
Volume 5 NTFS Partition 505 MB Healthy Hidden
DISKPART>... but the C:\Windows\system32> (echo SELECT DISK 1 && echo LIST PART) | diskpart
Microsoft DiskPart version 10.0.19041.964
Copyright (C) Microsoft Corporation.
On computer: DESKTOP-O7JE7JE
DISKPART>
Disk 1 is now the selected disk.
DISKPART>
Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 System 100 MB 1024 KB
Partition 2 Reserved 16 MB 101 MB
Partition 3 Primary 237 GB 117 MB
Partition 4 Recovery 505 MB 237 GB
Partition 5 Primary 38 GB 238 GB
Partition 6 Primary 188 GB 277 GB
DISKPART>
C:\Windows\system32>Also C:\Windows\system32> mountvol
<...>
Possible values for VolumeName along with current mount points are:
\\?\Volume{8317bf2b-9709-4f92-be0f-6f3f3dd83aa4}\
D:\
\\?\Volume{3b7e9709-77dc-4d88-ab05-e923b26d95a3}\
C:\
\\?\Volume{878041e1-48c2-484b-bcc1-507b0ff19c4c}\
*** NO MOUNT POINTS ***
\\?\Volume{c27eb7f0-3284-43ef-9f12-c3405d3bcd64}\
D:\Another-volume\
\\?\Volume{5feefb5c-468a-4da8-9fcc-0dfba19dcab8}\
*** NO MOUNT POINTS ***
\\?\Volume{0869fa25-bd66-7074-6498-7f4d08d64db0}\
E:\
C:\Windows\system32> |
|
Looks like mountvol shows it ok, but I would argue But I do need to get mountvol to show E:\dataset though, VHD will do that. |
Fair enough, Jorgen! I just wanted to state that the line is present. Now I see that even I have wiped out the partition, the line would stay the same. Sorry for not thinking about that earlier.
Sure. Soon after your comment I remembered the Ext4FSD project. I installed the version v0.71, created an additional additional 1 GiB long partition 7 on the same disk: C:\Windows\system32> (echo SELECT DISK 1 && echo LIST PART) | diskpart
Microsoft DiskPart version 10.0.19041.964
Copyright (C) Microsoft Corporation.
On computer: DESKTOP-O7JE7JE
DISKPART>
Disk 1 is now the selected disk.
DISKPART>
Partition ### Type Size Offset
------------- ---------------- ------- -------
Partition 1 System 100 MB 1024 KB
Partition 2 Reserved 16 MB 101 MB
Partition 3 Primary 237 GB 117 MB
Partition 4 Recovery 505 MB 237 GB
Partition 5 Primary 38 GB 238 GB
Partition 6 Primary 4120 MB 277 GB
Partition 7 Primary 1024 MB 281 GB
DISKPART>Ext4FSD seems to include no tool to make a new FS. So I had to elevate my MSYS2 terminal and create the FS off-site. It went somewhat along these lines <...># dd if=/dev/zero of=/dev/sdb7
# dd if=/dev/sdb7 | pv | ssh -C omn "cat > w10-sdb7.img"
# ssh omn
$ ls -l w10-sdb7.img
$ sudo losetup -f w10-sdb7.img
$ sudo losetup -l
$ mkfs.ext4 /dev/loop0
$ sudo losetup -d /dev/loop0
$ exit
# ssh -C omn "cat w10-sdb7.img" | pv | dd of=/dev/sdb7Then I toyed with Ext2 Volume Manager a bit:
... and found out that its' mountpoint management offers only letter-mounts:
So I went back to the Disk Management console & assigned the non-letter mount point by hand: And you know, with Ext4FSD this worked!
Now C:\Windows\system32> echo LIST VOL | diskpart
Microsoft DiskPart version 10.0.19041.964
Copyright (C) Microsoft Corporation.
On computer: DESKTOP-O7JE7JE
DISKPART>
Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 D New Volume NTFS Partition 931 GB Healthy
Volume 1 C pagrindinis NTFS Partition 237 GB Healthy Boot
Volume 2 Partition 38 GB Healthy
Volume 3 NTFS Volume NTFS Partition 4120 MB Healthy
Volume 4 An EXT4 vol EXT4 Partition 1024 MB Healthy
D:\Another-volume\
Volume 5 FAT32 Partition 100 MB Healthy System
Volume 6 NTFS Partition 505 MB Healthy Hidden
DISKPART>And C:\Windows\system32> mountvol
<...>
Possible values for VolumeName along with current mount points are:
\\?\Volume{8317bf2b-9709-4f92-be0f-6f3f3dd83aa4}\
D:\
\\?\Volume{3b7e9709-77dc-4d88-ab05-e923b26d95a3}\
C:\
\\?\Volume{878041e1-48c2-484b-bcc1-507b0ff19c4c}\
*** NO MOUNT POINTS ***
\\?\Volume{c27eb7f0-3284-43ef-9f12-c3405d3bcd64}\
*** NO MOUNT POINTS ***
\\?\Volume{776c48ee-fe39-4fa4-8e7c-9d07a3a0d176}\
D:\Another-volume\
\\?\Volume{5feefb5c-468a-4da8-9fcc-0dfba19dcab8}\
*** NO MOUNT POINTS ***
\\?\Volume{0869fa25-bd66-7074-6498-7f4d08d64db0}\
E:\
C:\Windows\system32> Later I unmounted the volume and retried the same using C:\Windows\system32> mountvol D:\Another-volume \\?\Volume{776c48ee-fe39-4fa4-8e7c-9d07a3a0d176}
C:\Windows\system32>HTH EDIT: Also please note, that the FSD code is license under GNU GPLv2. For the rest of the project (Ext2Srv, Ext2Mgr, Setup) I found no clear indication of the code's license. This may mean that for staying legally clear you should take breaks between reading the code and rewriting it on your own (to avoid legal uncertainties / clashes of CDDL against GPLv2:) |
|
PS. The Ext2 Volume Manager includes nice details about IRP usage, just press F8:
|
|
Ah ok, good to see it can/should work. I have something I want to try out in code today first, getting a clearer picture how it hangs together |
|
OK should re-test this after 2.2.6rc1 |
System information
Describe the problem you're observing
If i try to copy a directory that has a subfolder to the root
Z:\it will work, but if you copy to the zfs datasetZ:\dataset, it will fail.I have tried 3 tools and 2 of them fail to copy to the dataset
Z:\Z:\datasetDescribe how to reproduce the problem
Create the a directory structure that has a subfolder on your ntfs drive
Create pool and dataset:
Now if you try to copy this directory to the root
Z:\it will work, but if you copy to the zfs datasetZ:\datasetit will failRClone
Copying to root works:
Copying to dataset fails:
Robocopy
Copying to root works:
Copying to dataset fails:
The text was updated successfully, but these errors were encountered: