Performance issue with clustering duplicates #298

bczech · 2023-10-30T11:00:27Z

Describe the bug

During clustering of columns of [1, 483] matrix we detected the performance issue: Error: C stack usage 7969316 is too close to the limit.
In our case we used a Bioc-devel docker image with 8 GB of memory.

To Reproduce

data <- structure(c(1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, -0.63721185561396, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.08197242820826, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, -4, -4, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.25412348577246, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, -0.514168290397842, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.26334567055877, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, -0.938041635162273, 
1.30257021196862, 1.42046951549569, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, -0.194594761524711, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.36755834126481, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966, 1.47712125471966, 
1.47712125471966, 1.47712125471966, 1.47712125471966), dim = c(1L, 
483L), dimnames = list(NULL, NULL))

heatmaply::heatmaply(x = data,
                     plot_method = "plotly",
                     Rowv = FALSE,
                     Colv = TRUE)

Expected behavior

Display the heatmap with dendrogram for columns

Screenshots

N/A

Additional context

The text was updated successfully, but these errors were encountered:

alanocallaghan · 2023-10-30T11:22:34Z

I think you just need more resources or to increase the C stack limit. It's probably an R/dendrogram issue rather than a heatmaply issue, unless the traceback suggests otherwise

talgalili · 2023-10-30T12:31:24Z

This might be a dendextend::as.ggdend issue. Could you please check/confirm? If so, there might be a solution (but it would take me a couple of weeks to implement).

…

On Mon, 30 Oct 2023, 13:22 alanocallaghan, ***@***.***> wrote: I think you just need more resources or to increase the C stack limit. It's probably an R/dendrogram issue rather than a heatmaply issue, unless the traceback suggests otherwise — Reply to this email directly, view it on GitHub <#298 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHOJBSQV2VTFN7PU5DG5CTYB6EYLAVCNFSM6AAAAAA6V2BCT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHE4DCOBXGQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

bczech · 2024-01-31T13:35:29Z

The only solution was to manually turn off the dendrogram drawing

talgalili · 2024-01-31T13:47:04Z

Thanks. Is it sufficient for your needs now?

…

On Wed, 31 Jan 2024, 15:35 Bartek, ***@***.***> wrote: The only solution was to manually turn off the dendrogram drawing — Reply to this email directly, view it on GitHub <#298 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHOJBTHM4AWDU7WIP4C5BDYRJCCZAVCNFSM6AAAAAA6V2BCT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGEYTKOJVGU> . You are receiving this because you commented.Message ID: ***@***.***>

bczech · 2024-01-31T13:49:13Z

Yes, we can close this issue for now.

bczech added the bug label Oct 30, 2023

bczech closed this as completed Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance issue with clustering duplicates #298

Performance issue with clustering duplicates #298

bczech commented Oct 30, 2023 •

edited

Loading

alanocallaghan commented Oct 30, 2023

talgalili commented Oct 30, 2023 via email

bczech commented Jan 31, 2024

talgalili commented Jan 31, 2024 via email

bczech commented Jan 31, 2024

Performance issue with clustering duplicates #298

Performance issue with clustering duplicates #298

Comments

bczech commented Oct 30, 2023 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Screenshots

Additional context

alanocallaghan commented Oct 30, 2023

talgalili commented Oct 30, 2023 via email

bczech commented Jan 31, 2024

talgalili commented Jan 31, 2024 via email

bczech commented Jan 31, 2024

bczech commented Oct 30, 2023 •

edited

Loading