Plotting of large data sets fails

[bwbioinfo]

When plotting ~200M or more points, the plotting fails with the following error :

<--- Last few GCs --->

[4015030:0x5b7f43c3f000]     5983 ms: Mark-Compact (reduce) 1398.7 (1402.5) -> 1397.8 (1399.6) MB, pooled: 0 MB, 120.51 / 0.00 ms  (+ 0.1 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 121 ms) (average mu = 0.50[4015030:0x5b7f43c3f000]     6160 ms: Mark-Compact 1400.8 (1401.6) -> 1399.4 (1404.2) MB, pooled: 0 MB, 175.27 / 0.00 ms  (average mu = 0.294, current mu = 0.007) allocation failure; scavenge might not succeed


<--- JS stacktrace --->



#
# Fatal JavaScript out of memory: Reached heap limit
#

I don't see an option to adjust the memory settings but this might be an avenue to explore.

[LukaOber]

I don't want to be that guy, but why are you trying to create a plot with that many points? I would recommend using something like a lttb algorithm to reduce the number of points to plot. Can you provide us with more information about your use case? What renderer are you using? Can you provide an example? What version of charming are you using?

[bwbioinfo]

I'm trying to visualize chromosome-wide genomics events. I could downsample, but if I understand that error it seems to be failing at less than 2 gb, so we're not talking about an obscene amount of resources here.

I'm using the most recent version, pulled from the repo.

This is the function used for plotting :

pub fn plot_chromosome(data: DataFrame, output_path: &str) -> Result<(), Box<dyn Error>> {

    // Extract the columns we need
    let start_position: Vec<Option<i64>> =  data.column("chromStart").unwrap().i64().unwrap().into_iter().collect();
    let percent_modified = data.column("percent_modified").unwrap().f64().unwrap().to_vec();
    // Transpose the data into a format that can be used by the chart
    let datapoints = dz!(start_position, percent_modified);

    // get the chromosome name as a single string
    let chromosome_name = 
        data
        .column("chrom")
        .unwrap()
        .str()
        .unwrap()
        .get(0)
        .unwrap();

    // Create the chart
    let chart = Chart::new()
    // set the chromosome name as the title
    .title(Title::new().text(format!("Chromosome {} Modification Frequency", chromosome_name)))
    .x_axis(Axis::new())
    .y_axis(Axis::new())
    .series(
        Scatter::new()
        .symbol_size(5)
        .data(datapoints)
    );

    // Render the chart
    let mut renderer = 
        ImageRenderer::new(1500, 200)
        .theme(Theme::Default);
    // Render the chart as SVG string.
    renderer.render(&chart).unwrap();
    // Render the chart as PNG bytes.
    renderer.render_format(ImageFormat::Png, &chart).unwrap();
    // Save the chart as SVG file.
    // renderer.save(&chart, output_path).unwrap();
    // Save the chart as PNG file.
    _ = renderer.save_format(ImageFormat::Png, &chart, output_path);


    println!("Saved to: {}", output_path);

    Ok(())
}

[LukaOber]

The v8 engine seems to have a default memory limit per process of ~1.4GB on 64-bit machines. Can you try running your code while setting this env? V8_FLAGS: "--max-old-space-size=4096"

[LukaOber]

@bwbioinfo did it work? Do you need more help?