Process runs out of memory

[fabiankrack]

Hi,

the process just died leaving only the following info in the logs:

Error: spawn ENOMEM
   at errnoException (child_process.js:1001:11)
   at ChildProcess.spawn (child_process.js:948:11)
   at exports.spawn (child_process.js:736:9)
   at spawn (/var/lib/sphere/sphere-express-pdf/node_modules/phantom/node_modules/win-spawn/index.js:54:10)
   at startPhantomProcess (/var/lib/sphere/sphere-express-pdf/node_modules/phantom/phantom.js:17:12)
   at Server.<anonymous> (/var/lib/sphere/sphere-express-pdf/node_modules/phantom/phantom.js:95:14)
   at Server.emit (events.js:92:17)
   at net.js:1055:10
   at process._tickDomainCallback (node.js:463:13)
strong-agent[19760] cannot connect to collector: getaddrinfo ENOTFOUND

So it looks like the app tried to spawn a child process and got an error back from the system because there was no memory left. After restarting the app there is plenty free memory left, so i guess the issue could be a memory leak here.
Any hint how to get more insights into this would be much appreciated.

[emmenko]

@onibox

Well, the pdf is running on the 1GB sftp node, so it's not necessary a problem with the pdf app if the sftp itself is under load.

Nevertheless, handling phantom processes is not really easy. What I'm doing currently is ensuring a process is always running because spawning one on demand for every request is expensive.
Also if the process dies, it should recover.

There is also a monit check that should restart the app if it crashes (I think)
See sphere-express_monitrc.erb file (no link)

But of course there could be a memory leak somewhere. If I have time I will try to investigate a bit more.

[fabiankrack]

@emmenko thx for the quick feedback. I'm aware of our monitoring checks, but i don't think this kind of info belongs in a public repository like this ;-)

Would be cool if you could have a look into this.

[svenmueller]

let's monitor the memory usage on the node...do we already have the metrics to do that?