Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement flushSync to flush logs when running inside of Cloud functions #33

Draft
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

egorpavlikhin
Copy link

@egorpavlikhin egorpavlikhin commented Mar 4, 2024

When flushing logs pino doesn't actually call _final and wait for the logs to flush when using streams. Instead it calls a poorly documented flushSync method that is not part of the stream.Writable interface. As the name suggests the function is not async and will block the event loop until the logs are flushed. Normally this wouldn't be an issue, but when running inside a Cloud function the function will often exit before the logs are flushed, so this implementation is in line with what pino recommends https://getpino.io/#/docs/asynchronous?id=aws-lambda

https://github.com/pinojs/pino/blob/c109804c6ce4b4545f3dfe1d021ceb698582fe15/lib/multistream.js#L82

In practice, without this change we had about 20% of our logs disappear as pino-seq would still be trying to send logs to seq when our Azure functions has already finished executing (eventually failing after a 30 seconds timeout).

How to review: I've been trying to find a good way to run async functions synchronously in javascript but couldn't. There are libraries that spawn a separate nodejs process and wait for completion there, but since this library doesn't necessarily run in nodejs context I thought it's a bit of an overkill. Happy to hear thoughts on how this should ideally look like.

@liammclennan
Copy link
Contributor

Hi @egorpavlikhin,

It looks like you should be able to just await _logger.flush() or _logger.close()

@richard-stafflink
Copy link

Hi! I would really love to get a hold of this

// As the name suggests the function is not async and will block the event loop until the logs are flushed
// Normally this wouldn't be an issue, but when running inside a Cloud function the function will often exit before the logs are flushed,
// so this implementation is in line with what pino recommends https://getpino.io/#/docs/asynchronous?id=aws-lambda
flushSync() {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The .d.ts file should also be updated to expose this method on the return type

@richard-stafflink
Copy link

Hi @egorpavlikhin,

It looks like you should be able to just await _logger.flush() or _logger.close()

Could you expose the _logger on the .d.ts please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants