Hot Reload

[HRashidi]

Enhancement Description

• Right now, if any part of the sdk is changed, the whole sdk needs to be restarted. This is not ideal because it takes a lot of time to restart the SDK especially if some of the models take a long time to load.

Advantages

• Using this improvement, we don't need to restart the entire sdk to add or remove new models or features especially when updating the model weights.

Possible Implementation

• Split the sdk into different pars. so each deployments are launched separately. This also allow us to change number of replicas or auto scaling based on the loads.

[movchan74]

We already have an issue for that: #29
If you have a suggestion on how to implement it, please add it as a comment there. Please check if an issue already exists before opening a new one.

[HRashidi]

duplicated