Post queue is now empty - upload speed will be throttled

[BlepingMatrix]

Hi there,

When I'm posting on usenet my upload speed starts out at around 80MB/s ...then I get this message "Post queue is now empty - upload speed will be throttled" and it dials down to 30MB/s and just stays there. Is it an issue? or am I doing something wrong? Mind helping me out?

Thanks,
BlepingMatrix

[animetosho]

When posting, Nyuu reads data off disk, yEncodes it and places it in a post queue. Uploading connections pull items off the queue to post. If a connection tries to grab an item, but finds the queue empty, that warning gets displayed as the uploader has to wait for data to become available.
Essentially, the issue arises if your system is unable to get data quickly enough to send over your connection - for most systems, the upload bandwidth is the bottleneck, not the system itself. The purpose of the queue is to buffer against any potential spikes in system activity and minimize speed degradation from such.

Some questions:

• is your internet connection actually able to sustain 80MB/s?
• is the system under any other load whilst posting?
• what's the CPU usage and (most importantly) disk I/O usage like when posting, before/after you get that message?
• what's the hardware specifications of the system doing the uploading? In particular, what's the disk that the data is being read from? Is it over a network/NAS or a USB based drive, and what bandwidth (sequential read speed) can it sustain?
• does this issue occur consistently?

Nyuu may provide some ability to mitigate some issues, but we'll need to figure out the problem first.

[BlepingMatrix]

If I'm say seeding something I can easily get around 100MB/s upload if I'm the sole seed. So yes it can sustain it.
Around 5% or so of the CPU and 4% of the memory is in use (before posting)
Paring/raring maxes it all out. But when it reaches the Uploading bit. It comes down to around 50% usage then a few seconds later the post queue thing shows up and it goes down to 30% and mostly just says within ±5 of that.
I have a Xeon(R) CPU X3440 processor
About 16gigs of ram
4 TOSHIBA HDWE140 HDDs Totalling 16TB
Yes this appears every time. Pretty consistent.

Tho I should mention I'm using a prebuilt nyuu binary and didn't use npm to compile it.

[animetosho]

Thanks for the info.

Have a few more questions unfortunately:

• Are you running PAR/RAR, or seeding whilst uploading?
• How long does it run before that warning pops up (like, within a few seconds of starting, halfway through, etc)?
• Are you able to check the disk I/O usage? On Linux, you can use the iostat tool, something like running iostat -x 5 for a while and post the output. On Windows, Task Manager may be able to give an indication Disk usage.
• I presume those disks are connected directly to the PC via SATA, and you're using some RAID config on them?

It comes down to around 50% usage then a few seconds later the post queue thing shows up and it goes down to 30% and mostly just says within ±5 of that.

Are you referring to CPU usage there? Nyuu is largely single threaded, so shouldn't use more than around 12.5% of an 8 thread CPU (if you're using SSL, I'm not sure whether node offloads this to another thread, so I suppose there could be 2 threads, i.e. max possible CPU load is around 25%) - are you able to confirm this on your end?

Your CPU usage there seems to suggest that something else is running, but more importantly, the decrease suggests a bottleneck elsewhere (i.e. not your CPU).
My gut says that your disk is the bottleneck, but I'm just making a wild guess here. Try adding --disk-req-size 28000K --post-queue-size 100 to your command line and see if that makes any difference.

[BlepingMatrix]

"--disk-req-size 28000K --post-queue-size 100" That didn't make much of a difference, but limiting the connections increased the speed by around 6MB/s.
How long does it run before that warning pops up---->its always exactly when the third par or whatever uploads.. so 7-10 seconds?
Are you referring to CPU usage there?---> yep, took a screenshot:

"41.9"
Here is an iostat log while Nyuu is running till after it ends. https://pastebin.com/aDJuHz3q
Have my HDDs in a raid0 setup yeah

EDIT: worth noting I further decreased connections to 8 and now Im at 55MB/s upload. But Nyuu CPU usage increases as I lower connections ...as seen here.

[animetosho]

Thanks for that.

I see, you're referring to CPU usage of the process itself - I believe your reading there is based on 100% = 1 core, so Nyuu isn't fully utilising a core there.

Your iostat log gives a lot of useful info. It looks like the disk isn't the bottleneck there, and the speed doesn't vary too much over the period you were logging, so my guess was incorrect there.

Your CPU doesn't support AES acceleration, so using SSL is likely slow - I assume you are using SSL? If so, you could trying adding --ssl-ciphers RC4-MD5 or if that doesn't work (it may be unsupported), try --ssl-ciphers AES128-SHA or --ssl-ciphers ECDHE-RSA-AES128-SHA. This could reduce the CPU usage slightly.
If you're willing to do so, you'll probably see a further boost by disabling SSL entirely.

Interesting that lowering connections increases speed and utilises the CPU more efficiently, thanks for finding that out. I haven't done much SSL testing, so am not sure how well it plays with multiple connections, but your result is a little surprising there. I'm not sure what your bottleneck there is, but it might be CPU (even though it's not fully using a core), so see if the SSL settings above makes a difference.

[BlepingMatrix]

Alright I'll give the above ssl ciphers a whirl and report back. I found the sweet spot if this still doesn't fixes it tho...
At 5 connections --->

EDIT: Okay so RC4-MD5 brought it up to around 70MB/s ..as expected. The throttling error is still there though.

[animetosho]

Thanks for trying that out. That does seem to show that your speed is somewhat related to CPU usage (although, weirdly, it still doesn't consume a whole core).

I think I'll need to look more into testing SSL to understand its performance characteristics.

If you're still interested in trying stuff out, you could try building Nyuu on a different version of node.js (the pre-built binary is using the 4.x line). It's possible that newer node.js versions improve SSL performance, but I've never done any testing myself, so wouldn't have a clue. (or you could use something like stunnel to offload SSL to another process, but the setup gets complicated)

Otherwise, 70MB/s might be the best you can get from Nyuu for now.

[scriptzteam]

I got over 100MB/sec easily :)

But getting the same "error" anyway i dont care as it upload :D

[BlepingMatrix]

Woohoo! Managed to get a stable 105 MB/s after changing providers (noticed a funny thing tho. If cpu% in htop is 86% the speed is 85-86MB/s ..if its 104% I get 103-104MB/s)

Note: The throttling error remains... (Not sure if I should close this cause of the error even tho it now seems harmless)

[animetosho]

Thanks for the info!

I've had a sneaking suspicion of whether the provider's CPU may be maxed out, since yours isn't and using a faster cipher resulted in a faster download (but there's no way really verify this theory). If switching provider increased speeds for you, perhaps there's something to that theory. Are you still changing SSL ciphers on the new provider or just using the default?

The post queue empty warning is still something interesting to investigate, since it indicates that Nyuu was stalled at least once when waiting for data to upload. Ideally this shouldn't be happening (i.e. I'd like the network to be the bottleneck, not the CPU/disk of the system).

[BlepingMatrix]

I'm using this cipher ECDHE-RSA-AES128-GCM-SHA256 as Tweak doesn't seem to support RC4-MD5. Using the default reduces the speed to around 75MB/s

[animetosho]

Ah I see, thanks. Many CPUs now support AES acceleration (yours doesn't unfortunately) but maybe your old provider's servers didn't.

[animetosho]

I think I'll need to look more into testing SSL to understand its performance characteristics.

It turns out that node.js doesn't implement threading for SSL/encryption (even worse, it appears to be on the main thread) which means that your upload speed will be limited by how fast a single CPU core can encrypt. I may try to see if I can do something to push SSL connections off to separate threads to make better use of CPU.