Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The processlogs script sometimes throws "RangeError [ERR_HTTP_INVALID_STATUS_CODE]" #3769

Closed
trivikr opened this issue Jun 19, 2024 · 2 comments
Closed

The processlogs script sometimes throws "RangeError [ERR_HTTP_INVALID_STATUS_CODE]" #3769

trivikr opened this issue Jun 19, 2024 · 2 comments

Comments

@trivikr
Copy link
Member

trivikr commented Jun 19, 2024

Describe the issue

The processlogs script sometimes throws "RangeError [ERR_HTTP_INVALID_STATUS_CODE]"

Steps to reproduce

View if you have access https://cloudlogging.app.goo.gl/rHMd176nXCG1XqgH9

Observed behavior

ERROR IN DOWNLOAD 20240607/20240607T032920Z_20240607T033027Z_2555cc7c.log.gz GaxiosError: Request failed with status code 502
at Gaxios._request (/usr/src/app/node_modules/gaxios/build/src/gaxios.js:129:23)
at processTicksAndRejections (node:internal/process/task_queues:94:5)
at async GoogleToken.requestToken (/usr/src/app/node_modules/gtoken/build/src/index.js:231:23)
at async GoogleToken.getTokenAsync (/usr/src/app/node_modules/gtoken/build/src/index.js:142:20)
at async JWT.refreshTokenNoCache (/usr/src/app/node_modules/google-auth-library/build/src/auth/jwtclient.js:158:23)
at async JWT.getRequestMetadataAsync (/usr/src/app/node_modules/google-auth-library/build/src/auth/oauth2client.js:261:17)
at async JWT.getRequestHeaders (/usr/src/app/node_modules/google-auth-library/build/src/auth/oauth2client.js:238:26)
at async GoogleAuth.authorizeRequest (/usr/src/app/node_modules/google-auth-library/build/src/auth/googleauth.js:593:25) {
response: {
config: {
method: 'POST',
url: 'https://www.googleapis.com/oauth2/v4/token',
data: [Object],
headers: [Object],
responseType: 'json',
paramsSerializer: [Function: paramsSerializer],
body: 'grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=<redacted>',
validateStatus: [Function: validateStatus]
},
data: '<!DOCTYPE html>\n' +
'<html lang=en>\n' +
' <meta charset=utf-8>\n' +
' <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">\n' +
' <title>Error 502 (Server Error)!!1</title>\n' +
' <style>\n' +
' *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n' +
' </style>\n' +
' <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n' +
' <p><b>502.</b> <ins>That’s an error.</ins>\n' +
' <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>\n',
headers: {
'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
connection: 'close',
'content-length': '1613',
'content-type': 'text/html; charset=UTF-8',
date: 'Fri, 07 Jun 2024 03:30:39 GMT',
'referrer-policy': 'no-referrer'
},
status: 502,
statusText: 'Bad Gateway',
request: { responseURL: 'https://www.googleapis.com/oauth2/v4/token' }
},
config: {
method: 'POST',
url: 'https://www.googleapis.com/oauth2/v4/token',
data: {
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
assertion: '<redacted>'
},
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
Accept: 'application/json'
},
responseType: 'json',
paramsSerializer: [Function: paramsSerializer],
body: 'grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=<redacted>',
validateStatus: [Function: validateStatus]
},
code: '502',
bufferedData: <Buffer >
}
node:internal/process/promises:245
triggerUncaughtException(err, true /* fromPromise */);
^
RangeError [ERR_HTTP_INVALID_STATUS_CODE]: Invalid status code: undefined
    at new NodeError (node:internal/errors:329:5)
    at ServerResponse.writeHead (node:_http_server:282:11)
    at ServerResponse._implicitHeader (node:_http_server:273:8)
    at ServerResponse.end (node:_http_outgoing:874:10)
    at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:221:10)
    at /usr/src/app/process-cloudflare.js:217:24
    at /usr/src/app/process-cloudflare.js:155:7
    at /usr/src/app/node_modules/onetime/index.js:19:28
    at processTicksAndRejections (node:internal/process/task_queues:94:5) {

Expected behavior

No RangeError

Additional context

Noticed while reviewing logs in #3697 (comment)
@trivikr
Copy link
Member Author

trivikr commented Jun 19, 2024

This happens as callback expects status to be defined

build/ansible/roles/metrics/files/process-cloudflare/process-cloudflare.js

Lines 216 to 218 in e3e4b4a

processLogs(bucket, filename, function (status) {
res.status(status).send()
})

But when there's error while downloading file, the callback is called without a status code

build/ansible/roles/metrics/files/process-cloudflare/process-cloudflare.js

Lines 152 to 156 in e3e4b4a

if (err) {
console.log('ERROR IN DOWNLOAD ', filename, err)
// callback(500)
callback()
} else {

@trivikr
Copy link
Member Author

trivikr commented Jun 27, 2024

Example log which throws error

EVENT TYPE: OBJECT_FINALIZE
Node version is: v20.15.0
BUCKET cloudflare-logs-nodejs
FILENAME 20240625/20240625T193311Z_20240625T193421Z_ce3b0ea7.log.gz
PROCESSEDFILENAME 20240625/20240625T193311Z_20240625T193421Z
INSIDE CREATE PIPELINE
ERROR IN DOWNLOAD 20240625/20240625T193311Z_20240625T193421Z_ce3b0ea7.log.gz [Error: ENOENT: no such file or directory, open '/usr/src/app/metrics-processor-service-key.json'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/usr/src/app/metrics-processor-service-key.json'
}
node:_http_server:352
throw new ERR_HTTP_INVALID_STATUS_CODE(originalStatusCode);
^
ERROR 2024-06-25T19:34:33.666202Z RangeError [ERR_HTTP_INVALID_STATUS_CODE]: Invalid status code: undefined at ServerResponse.writeHead (node:_http_server:352:11) at ServerResponse._implicitHeader (node:_http_server:338:8) at ServerResponse.end (node:_http_outgoing:1072:10) at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:233:10) at /usr/src/app/process-cloudflare.js:220:24 at /usr/src/app/process-cloudflare.js:155:7 at callback (/usr/src/app/node_modules/@google-cloud/storage/build/src/file.js:1624:17) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
code: 'ERR_HTTP_INVALID_STATUS_CODE'
}

This error happened when Dockerfile was deployed without metrics-processor-service-key.json while experimenting.
The documentation instructions in #3779 explain how/where the file should be added.

In future, we should consider providing credentials in some different manner.

@trivikr trivikr closed this as completed Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@trivikr