Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readable never recieving 'end' event on COPY stream #3284

Open
Ivor808 opened this issue Jul 26, 2024 · 0 comments
Open

Readable never recieving 'end' event on COPY stream #3284

Ivor808 opened this issue Jul 26, 2024 · 0 comments

Comments

@Ivor808
Copy link

Ivor808 commented Jul 26, 2024

Hey all,

I'm using node pg + the copy extension to run COPY TO STDOUT on the postgres server to write to a file locally. I've noticed that certain tables and row combinations result in the readable stream never recieving the "end" event (even when data events stop coming through). Thus the stream hangs open forever. I then checked the pg_stat_progress_copy table and that had 0 results even when the stream was still hanging. I then checked the pg_stat_activity and can see the copy process in there in an idle state. The data size in my repo case is 300 lines from the DB aka very small. I actually noticed this issue with the postgresjs package too. So I'm not sure if its specific to nodepg or postgres.

Here is an example of my COPY code

 const client = await pgPool.connect();
  const readable = client.query(copyTo(copyQuery));

  try {
    await pipeline(
      readable,
      createWriteStream(`${os.homedir()}/cms/${fileName}`)
    );
  } finally {
    client.release();
  }

Here I attached end and close event listeners to the readable stream. You can see the first "downloading" gets a end and close event. While the second readable stream does not and hangs.
image

Edit: I'm noticing it actually getting stuck on the same table which never sends the close event but doesnt read any more data in

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant