Different size between uploaded and local

[ilijamt]

I was wondering I uploaded some file to amazon glacier using the sync command.

There are a total of 125 files.

The local size is: 112907687148 bytes
But the uploaded size is: 112907650284 bytes

As you can see there is a difference of ~30MB between the data, can you tell me why?

To get the size from the journal I did the following

$ export f=`cut -f 5 pictures.journal`
$ echo $(( ${f//$'\n'/+} ))
112907650284

$ du -shb Pictures
112907687148    Pictures

112907687148 - 112907650284 = 36864

Running the check-local-hash says that everything is OK.
I issued a new inventory, so I can recreate the journal and run a check-local-hash again.

[vsespb]

Maybe this http://unix.stackexchange.com/a/94389 ?
12.11.2014 4:58 пользователь "Ilija Matoski" [email protected]
написал:

I was wondering I uploaded some file to amazon glacier using the sync
command.

There are a total of 125 files.

The local size is: 112907687148 bytes
But the uploaded size is: 112907650284 bytes

As you can see there is a difference of ~30MB between the data, can you
tell me why?

To get the size from the journal I did the following

$ export f=cut -f 5 pictures.journal
$ echo $(( ${f//$'\n'/+} ))
112907650284

$ du -shb Pictures
112907687148 Pictures

112907687148 - 112907650284 = 36864

Running the check-local-hash says that everything is OK.
I issued a new inventory, so I can recreate the journal and run a
check-local-hash again.

—
Reply to this email directly or view it on GitHub
#100.

[ilijamt]

I will try a little later, as I started uploading two big files, I will move them after it finishes upload, and will check then.

One note, if a file is moved before it even starts the upload with sync the whole application crashes.

[vsespb]

Need a stacktrace for crash. Maybe this is by design
12.11.2014 5:15 пользователь "Ilija Matoski" [email protected]
написал:

I will try a little later, as I started uploading two big files, I will
move them after it finishes upload, and will check then.

One note, if a file is moved before it even starts the upload with sync
the whole application crashes.

—
Reply to this email directly or view it on GitHub
#100 (comment)
.

[ilijamt]

For the stack trace, just remove any file during sync that hasn't started.
It shouldn't be like this, as if there is a different file uploading at the moment, and this crashes,, it also crashes the other upload, and you have to re-upload the file again.

$ export f=`cut -f 5 pictures.journal`
$ echo $(( ${f//$'\n'/+} ))
143196283006

$ du -shb --apparent-size 
143196319870 

143196283006 - 143196319870 = 36864

Again the same difference.

Would re-downloading the journal, find errors if there are?

[vsespb]

I think would be good to 1) compare number of files 2) compare files one by one to find file(s) with wrong sizes.

Would re-downloading the journal, find errors if there are?

I don't think so, since check-local-hash says all ok. check-local-hash takes all files from journal and comparse size + checksum for each.

Idealy, if there is no bug in mtglacier, sync with --dry-run will show if there are pending files + check-local-hash will check already uploaded files.

[vsespb]

For the stack trace, just remove any file during sync that hasn't started.

Ok, I reproduced and started issue #101
Thanks.

As you can see there is a difference of ~30MB between the data, can you tell me why?

I tested your code, it's seems du missuse.

vse@GREEN-U:/tmp$ cd test1
vse@GREEN-U:/tmp/test1$ mkdir test
vse@GREEN-U:/tmp/test1$ echo -n "hey" > test/a
vse@GREEN-U:/tmp/test1$ echo -n "there" > test/b
vse@GREEN-U:/tmp/test1$ du -shb test
4104    test
vse@GREEN-U:/tmp/test1$ du -shb --apparent-size test
4104    test
vse@GREEN-U:/tmp/test1$ du  --apparent-size test
5   test
vse@GREEN-U:/tmp/test1$ ls -la test
total 16
drwxrwxr-x 2 vse vse 4096 Nov 15 19:55 .
drwxrwxr-x 3 vse vse 4096 Nov 15 19:55 ..
-rw-rw-r-- 1 vse vse    3 Nov 15 19:55 a
-rw-rw-r-- 1 vse vse    5 Nov 15 19:55 b

In your case du reports different size - probably with directory entries.
thus closing.