Does the usercode/grpc container support scaling?

[spandan-sharma]

On AWS Fargate I can deploy 3 Fargate services, one running the grpc service, one dagit and one daemon. If I then use DefaultRunLauncher all job runs are scheduled in the grpc service's task, ie. there's no new task per job run being spawned.

1. Assuming the jobs and independent steps within each job are still running concurrently (via multiprocessing), is it one executor controlling all the jobs or is it one executor per job running in the grpc container with whom the run-launcher is communicating?

2. What happens if I scale this task? ie if I have say 2 such task containers for the same user repository can I put them behind the application load balancer? Can I auto-scale this based on some criteria (like memory etc)? The idea is to still run as many jobs in a single task as possible without spawning new task per job but then scale out if that Fargate task is now overwhelmed and scale in when there's not enough load. But to do that, I need to understand how these jobs are run in the first place.

I'm not clear if the DefaultRunLauncher will put something in the RDS (database) which then the executor picks up and starts running the job and communicates back to the launcher by updating state(s) in RDS or if there's tcp-connection between the launcher (running in the daemon task) and the grpc container and they are communicating directly. For the latter, I suppose an ALB would be needed. For the former if the executors are polling the database for new jobs to run, would running multiple grpc containers (each with executor(s) running in them) cause them to step on each other? Is dagster designed for such scaling activity?

FWIW, I tried to create 2 tasks for the daemon service and they do interfere with each other, in the sense that every sensor now runs twice, once due to each of the daemon instances. So this make me feel that there should be exactly one daemon container/task. Dagit task is fine even if there are multiple of them from what i can see, please correct me if wrong. What I'm not sure about is the grpc/usercode container. Can I have multiple of those for the same codebase/repository for the purposes of scaling out as described earlier or that'll just not work?