Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set Upload-Length when appropriate #744

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions lib/upload.js
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ class BaseUpload {
// An array of upload URLs which are used for uploading the different
// parts, if the parallelUploads option is used.
this._parallelUploadUrls = null

// The remote upload resource's length is deferred
this._uploadLengthDeferred = this.options.uploadLengthDeferred
}

/**
Expand Down Expand Up @@ -704,9 +707,13 @@ class BaseUpload {
return
}

const deferLength = Number.parseInt(res.getHeader('Upload-Defer-Length'), 10)
this._uploadLengthDeferred = deferLength === 1

const length = Number.parseInt(res.getHeader('Upload-Length'), 10)
if (
Number.isNaN(length) &&
!this._uploadLengthDeferred &&
!this.options.uploadLengthDeferred &&
this.options.protocol === PROTOCOL_TUS_V1
) {
Expand Down Expand Up @@ -821,9 +828,10 @@ class BaseUpload {
// If the upload length is deferred, the upload size was not specified during
// upload creation. So, if the file reader is done reading, we know the total
// upload size and can tell the tus server.
if (this.options.uploadLengthDeferred && done) {
this._size = this._offset + valueSize
if (this._uploadLengthDeferred && (!this.options.uploadLengthDeferred || done)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (this._uploadLengthDeferred && (!this.options.uploadLengthDeferred || done)) {
if (this._uploadLengthDeferred || done) {

I think we can just use this._uploadLengthDeferred here and don't need to consult the original option anymore.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will change the behaviour quite a bit. As it is suggested right now, there's some problems:

  1. We don't always know the size when this._uploadLengthDeferred is true since options.uploadLengthDeferred could be true. In this case, we'd send wrong info.
  2. done will make it always pass the header at the end, which I believe isn't how it should work. It should only send the header on the next known request from what I understand.

If we change it to an AND, it'll kinda work, but only send the length header at the end.

The logic in the commit right now expands to:

  • (this._uploadLengthDeferred && !this.options.uploadLengthDeferred)
    Send header on next opportunity if the request is of known size
  • (this._uploadLengthDeferred && done)
    And send at the end if it's a stream

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, my suggestion is wrong. I used || instead of && on accident and my suggestion would be:

Suggested change
if (this._uploadLengthDeferred && (!this.options.uploadLengthDeferred || done)) {
if (this._uploadLengthDeferred && done) {

If we change it to an AND, it'll kinda work, but only send the length header at the end.

Yes, that's right and that's how tus-js-client has handled deferred lengths in the past. It always sent the length header once the file source is closed because then we know the file size for sure. I don't see a problem with keeping this behavior. Or do you?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I'm quite new to the protocol and library so I think you'd know more. But I was looking at this line over at the protocol documentation:
image

That led me to believe that it should be the first possible instance. Although, I guess both ways should work anyway?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Acconut, just want to bump this PR, any thoughts on this? Thanks 😄

this._size = done ? this._offset + valueSize : this._source.size
req.setHeader('Upload-Length', `${this._size}`)
this._uploadLengthDeferred = false
}

// The specified uploadSize might not match the actual amount of data that a source
Expand Down
107 changes: 107 additions & 0 deletions test/spec/test-common.js
Original file line number Diff line number Diff line change
Expand Up @@ -1322,5 +1322,112 @@ describe('tus', () => {
expect(options.onError).not.toHaveBeenCalled()
expect(options.onSuccess).toHaveBeenCalled()
})

it('should send upload length on the next request when server length is deferred and we know the total size', async () => {
const testStack = new TestHttpStack()
const file = getBlob('hello world')
const options = {
httpStack: testStack,
endpoint: 'http://tus.io/uploads',
uploadUrl: 'http://tus.io/uploads/resuming',
chunkSize: 4,
}

const upload = new tus.Upload(file, options)
upload.start()

let req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('HEAD')

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Defer-Length': 1,
'Upload-Offset': 5,
},
})

req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('PATCH')
expect(req.requestHeaders['Upload-Offset']).toBe('5')
expect(req.requestHeaders['Upload-Length']).toBe('11')
expect(req.body.size).toBe(4)

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Offset': 9,
},
})

req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('PATCH')
expect(req.requestHeaders['Upload-Offset']).toBe('9')
expect(req.requestHeaders['Upload-Length']).toBe(undefined)

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Offset': 11,
},
})
})

it('should send upload length at the end when server length is deferred and the file is a stream', async () => {
const testStack = new TestHttpStack()
const file = getBlob('hello world')
const options = {
httpStack: testStack,
uploadLengthDeferred: true,
endpoint: 'http://tus.io/uploads',
uploadUrl: 'http://tus.io/uploads/resuming',
chunkSize: 4,
}

const upload = new tus.Upload(file, options)
upload.start()

let req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('HEAD')

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Defer-Length': 1,
'Upload-Offset': 5,
},
})

req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('PATCH')
expect(req.requestHeaders['Upload-Offset']).toBe('5')
expect(req.requestHeaders['Upload-Length']).toBe(undefined)
expect(req.body.size).toBe(4)

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Offset': 9,
},
})

req = await testStack.nextRequest()
expect(req.url).toBe('http://tus.io/uploads/resuming')
expect(req.method).toBe('PATCH')
expect(req.requestHeaders['Upload-Offset']).toBe('9')
expect(req.requestHeaders['Upload-Length']).toBe('11')

req.respondWith({
status: 204,
responseHeaders: {
'Upload-Offset': 11,
},
})
})
})
})