Disconnections from Monti cause memory leak

[claudiovicar]

Hey folks,
this is a follow up on this issue: #37.
It seems like we had this exact same problem a month ago, using version 2.45.2-beta.2.
Our servers were forced to restart due to a memory error some time after a Monti connection failure.

2022-06-21T06:30:55.965598170Z Monti APM Error: while sending payload to Monti APM: Reached maximum retry limit for Request failed: 500

Then, around 10 minutes later (the error kept happening in the meantime, btw):

2022-06-21T06:42:36.417383217Z Monti APM Error: while sending payload to Monti APM: Reached maximum retry limit for Request failed: 500
2022-06-21T06:42:53.332755460Z <--- JS stacktrace --->
2022-06-21T06:42:53.332760121Z 
2022-06-21T06:42:53.332764375Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

The error only happened during the time Monti was enabled, just to make it clear :)
Could it be related to the linked issue? Any help is appreciated!

Thanks in advance

[zodern]

On Wednesday we finished deploying some changes that made the api the agent uses much more stable which should avoid the situation that causes this memory leak.

I will look into the memory leak. It seems more likely that the agent is temporarily using too much memory when it can't connect rather than being a memory leak. How much ram do your servers have, and how much is normally used?

[claudiovicar]

Hi, zodern, glad to hear it :)
That instance has 8GB but most of the time it sits under 50%.
And thanks again for looking into it