-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime-manager: lock down pooled threads #120
Comments
Some unit tests around this: https://github.com/OpenFn/kit/blob/worker-tests/packages/runtime-manager/test/worker-pool.test.ts |
You know, the runtime currently creates a new context object (basically a global scope) for each run. So actually, subsequent jobs in the same worker thread should already have their own contexts. This needs checking and testing. When we create a context, it's probably a good idea to freeze or harden it before running the job. With slightly tighter control over what goes in that scope (see #104) it's probably a pretty secure starting point. |
Ok, confirmed. Each runtime creates its own context, so two runs can't actually interfere with each other. This means we don't need to sandbox threads any more (unless someone can break out of the vm context into the parenting thread scope) See unit tests around this stuff here: https://github.com/OpenFn/kit/blob/security-tests/packages/runtime/test/security.test.ts Note that while the scope is isolated, it may still not be entirely secure. We're doing nothing to harden it, so a job can override eg the array prototype. To what end I am not sure as it can't affect any other job... |
So I think this one is gonna stay open. I've been testing around this most of the day. Where we are now is: Jobs running in pooled threads are totally dependent on the runtime sandbox to preserve their environment. We can't really freeze the global scope to prevent writes (freezing Having broken out, though, I don't think there's any way to break back in to the next job. I can confirm that the parent process is not available to the child. The child worker can optionally see the parent's I don't know. I'm not particularly worried about this, but I would like to invest a bit more time. If we can run jobs in a) a secure, sandboxed worker thread and b) a secure sandbox inside that worker, then that's a really tight double layer of security. Double jail. So I'd still like to take some action here. |
More thoughts on this. In workerpool, the environment is not terribly secure if you break out of the sandbox. But this week we have learned more and added more mitgations:
Also, the sandbox itself is a reasonable level of protection; threads cannot read or communicate with other threads or their parent process; and per the above the environment is protected (and the parent environment private). Closing as I do not plan to do any more work on this (beyond the other open issues) |
The runtime manager uses thread pooling to run jobs. This reduces the overhead of starting up a new process for every job - idle threads will be re-used.
There is a security issue here: because threads are shared, their environment is ALSO shared. That means a job could corrupt a worker's environment and compromise the next job to run in that thread.
There are two solutions which we should explore more nearer the time:
Object.freeze()
orses.harden()
to freeze the global scopevm.runInNewContext()
on every job (probably inside the worker'srun
function) to isolate each global scope.Breakouts are presumably a theoretical concern in both cases. I don't know which is more secure. If we can't find a reasonable solution then we'll have to abandon node pooling (which makes you wonder if threading is even worth it)
The text was updated successfully, but these errors were encountered: