100% cpu usage when send data with channel.data #266
Comments
|
Could you please take a profile of your program to see where it's looping? |
|
How can I do that? |
|
Is it enough? |
|
There is also additional context of this problem. Probably root reason is target device. |
|
I'm facing the same issue - 100% CPU usage (single core) when awaiting In comparison, Minimal example: use async_trait::async_trait;
use russh::client;
use russh::client::Config;
use russh_keys::key;
use std::sync::Arc;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let addrs = ("10.0.0.123", 22);
let mut session = client::connect(Arc::new(Config::default()), addrs, Client).await?;
let auth_ok = session.authenticate_password("user", "pass").await?;
if !auth_ok {
anyhow::bail!("auth failed");
}
let channel = session.channel_open_session().await?;
channel.exec(true, "cat > /dev/null").await?;
let infinite_data = tokio::io::repeat(b'A');
println!("sending data");
channel.data(infinite_data).await?;
unreachable!();
}
struct Client;
#[async_trait]
impl client::Handler for Client {
type Error = russh::Error;
async fn check_server_key(
&mut self,
_server_public_key: &key::PublicKey,
) -> Result<bool, Self::Error> {
Ok(true)
}
}(I'm using russh Flamegraph: flamegraph.svg.gz I generated the flamegraph using I also turned on debuginfo in the release profile: # Cargo.toml
[profile.release]
debug = true |
|
Here's a more realistic example: OpenSSH
russh
For me, Code: use async_trait::async_trait;
use russh::client;
use russh::client::Config;
use russh_keys::key;
use std::sync::Arc;
use tokio::time::Instant;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
let addrs = ("localhost", 22);
let mut session = client::connect(Arc::new(Config::default()), addrs, Client).await?;
let auth_ok = session.authenticate_password("eugene", "").await?;
if !auth_ok {
anyhow::bail!("auth failed");
}
let channel = session.channel_open_session().await?;
channel.exec(true, "cat > /dev/null").await?;
let mut bytes_sent = 0;
let block = [0; 1024 * 1024];
let t = Instant::now();
for _ in 0..1000 {
channel.data(&block[..]).await?;
bytes_sent += block.len();
}
let elapsed = t.elapsed();
let elapsed = elapsed.as_secs() as f64;
println!("sent {} bytes in {} seconds", bytes_sent, elapsed);
println!(
"{} MB per second",
(bytes_sent as f64 / elapsed / 1_000_000.0) as u64
);
Ok(())
}
struct Client;
#[async_trait]
impl client::Handler for Client {
type Error = russh::Error;
async fn check_server_key(
&mut self,
_server_public_key: &key::PublicKey,
) -> Result<bool, Self::Error> {
Ok(true)
}
} |
|
(also, as most CPU is spent in encryption, different cipher selection can account for huge performance deviations, e.g. 50% between Chacha20-Poly1305 and AES) |
|
I have same problem when the connection is lost while sending the large amount of data. |
I replaced
Sending data over a slow network takes orders of magnitude more time than any encryption, even in debug mode. The expected behavior is that the SSH implementation sleeps and does nothing while the data are being transmitted over the wire. BenchmarkOpenSSH$ head -c 67108864 /dev/zero | time ssh [email protected] "cat > /dev/null"
ssh [email protected] "cat > /dev/null" 0.26s user 0.27s system 1% cpu 33.915 total
|
Recently I got strange behavior on my app, it can stuck with 100% cpu usage forever, and only restart of proccess can fix that.
After some investigations, I found that it happens here:
Debug log:
log.txt
The text was updated successfully, but these errors were encountered: