Content-Length might not always exist

[potados99]

TL; DR

I found a problem which can be caused by some mirror servers(similar to this FAQ).

It would be nice if this module can handle null value of content-length header in a response.

The full story

In the middle of installing nodegui, I received this error:

> @nodegui/[email protected] setupqt /Users/potados/WebstormProjects/nodegui-starter/node_modules/@nodegui/nodegui
> cross-env node ./scripts/setupMiniQt.js

Minimal Qt 5.14.1 setup:
/Users/potados/WebstormProjects/nodegui-starter/node_modules/progress/lib/node-progress.js:160
  complete = Array(Math.max(0, completeLength + 1)).join(this.chars.complete);
             ^

RangeError: Invalid array length
    at ProgressBar.render (/Users/potados/WebstormProjects/nodegui-starter/node_modules/progress/lib/node-progress.js:160:14)
...

After some hours of digging, I found that it was because the completeLength did not have a valid number.

The artifact-installer module seemed to use node-progress to display download progress. It tries to read a content-length header from a response and uses it as a total amount of the file size. However, in some cases response.headers.get('content-length') returned null.

artifact-installer/src/downloader.ts

Line 31 in dc95c5e

const total = parseInt(`${response.headers.get('content-length')}`, 10);

There are at least one mirror server that does not allow the js code to read the content-length header.

In my region, http://download.qt.io redirects to http://mirrors.sjtug.sjtu.edu.cn. Below is a curl output of fetching qt base from that mirror:

$ curl -v  http://mirrors.sjtug.sjtu.edu.cn/qt/online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z
*   Trying 202.120.58.155...
* TCP_NODELAY set
* Connected to mirrors.sjtug.sjtu.edu.cn (202.120.58.155) port 80 (#0)
> GET /qt/online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z HTTP/1.1
> Host: mirrors.sjtug.sjtu.edu.cn
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 27030533
< Etag: "q4lwmhg3cw5"
< Last-Modified: Fri, 24 Jan 2020 10:18:17 GMT
< Server: Caddy
< X-Sjtug-Mirror-Id: zhiyuan
< Date: Fri, 16 Apr 2021 17:17:41 GMT
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 1246)
* Closing connection 0

It clearly gives the line Content-Length: 27030533. However when run inside js with node-fetch:

> fetch = require('node-fetch');
<ref *1> [Function: fetch] {
  isRedirect: [Function (anonymous)],
  Promise: [Function: Promise],
  default: [Circular *1],
  Headers: [class Headers],
  Request: [class Request],
  Response: [class Response],
  FetchError: [Function: FetchError]
}
> fetch("http://mirrors.sjtug.sjtu.edu.cn/qt/online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z").then(response => console.log(response.headers.raw()))
Promise { <pending> }
> [Object: null prototype] {
  'content-encoding': [ 'gzip' ],
  etag: [ '"q4lwmhg3cw5"' ],
  'last-modified': [ 'Fri, 24 Jan 2020 10:18:17 GMT' ],
  server: [ 'Caddy' ],
  vary: [ 'Accept-Encoding' ],
  'x-sjtug-mirror-id': [ 'zhiyuan' ],
  date: [ 'Fri, 16 Apr 2021 17:20:14 GMT' ],
  connection: [ 'close' ],
  'transfer-encoding': [ 'chunked' ]
}

The Content-Length header is missing.

This is compared to the one in other mirrors: for example, https://qt.mirror.constant.com. It gives Content-Length in both situations:

curl:

$ curl -v http://qt.mirror.constant.com/online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z
*   Trying 108.61.5.83...
* TCP_NODELAY set
* Connected to qt.mirror.constant.com (108.61.5.83) port 80 (#0)
> GET /online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z HTTP/1.1
> Host: qt.mirror.constant.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Fri, 16 Apr 2021 17:39:31 GMT
< Content-Type: application/x-7z-compressed
< Content-Length: 27030533
< Last-Modified: Fri, 24 Jan 2020 10:18:17 GMT
< Connection: keep-alive
< ETag: "5e2ac469-19c7405"
< Expires: Sat, 17 Apr 2021 17:39:31 GMT
< Cache-Control: max-age=86400
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Accept-Ranges: bytes
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 1064)
* Closing connection 0

node-fetch

> fetch("http://qt.mirror.constant.com/online/qtsdkrepository/mac_x64/desktop/qt5_5141/qt.qt5.5141.clang_64/5.14.1-0-202001241000qtbase-MacOS-MacOS_10_13-Clang-MacOS-MacOS_10_13-X86_64.7z").then(response => console.log(response.headers.raw()));
Promise { <pending> }
> [Object: null prototype] {
  server: [ 'nginx' ],
  date: [ 'Fri, 16 Apr 2021 17:28:59 GMT' ],
  'content-type': [ 'application/x-7z-compressed' ],
  'content-length': [ '27030533' ],
  'last-modified': [ 'Fri, 24 Jan 2020 10:18:17 GMT' ],
  connection: [ 'close' ],
  etag: [ '"5e2ac469-19c7405"' ],
  expires: [ 'Sat, 17 Apr 2021 17:28:59 GMT' ],
  'cache-control': [ 'max-age=86400' ],
  'x-frame-options': [ 'DENY' ],
  'x-content-type-options': [ 'nosniff' ],
  'accept-ranges': [ 'bytes' ]
}

It would be nice if this module can handle null content length.