Fixes a memory leak during model unloading

[tanmayv25]

Before

 ==21418== Thread 32:
==21418== Invalid read of size 8
==21418== at 0x4CCA20F: nvidia::inferenceserver::RateLimiter::ModelInstanceContext::RequestRemoval() (in\
/opt/tritonserver/lib/libtritonserver.so)
==21418== by 0x4CCA285: nvidia::inferenceserver::RateLimiter::ModelInstanceContext::WaitForRemoval() (in\
/opt/tritonserver/lib/libtritonserver.so)
==21418== by 0x4CD1480: nvidia::inferenceserver::RateLimiter::UnregisterModel(nvidia::inferenceserver::T\
ritonModel const*) (in /opt/tritonserver/lib/libtritonserver.so)
==21418== by 0x4DC498A: nvidia::inferenceserver::TritonModel::~TritonModel() (in /opt/tritonserver/lib/l\
ibtritonserver.so)
==21418== by 0x4DC504C: nvidia::inferenceserver::TritonModel::~TritonModel() (in /opt/tritonserver/lib/l\
ibtritonserver.so)
==21418== by 0x4C61E46: std::thread::_State_impl<std::thread::_Invoker<std::tuple<nvidia::inferenceserve\
r::(anonymous namespace)::ModelDeleter::operator()(nvidia::inferenceserver::Model*)::{lambda()#1}> > >::M\
run() (in /opt/tritonserver/lib/libtritonserver.so)
==21418== by 0x5BE3DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==21418== by 0x578A608: start_thread (pthread_create.c:477)
==21418== by 0x5F7D292: clone (clone.S:95)
==21418== Address 0xf3d5d650 is 80 bytes inside a block of size 312 free'd
==21418== at 0x483F651: operator delete(void*) (vg_replace_malloc.c:923)
==21418== by 0x4DC48F0: nvidia::inferenceserver::TritonModel::~TritonModel() (in /opt/tritonserver/lib/l\
ibtritonserver.so)
==21418== by 0x4DC504C: nvidia::inferenceserver::TritonModel::~TritonModel() (in /opt/tritonserver/lib/l\
ibtritonserver.so)
==21418== by 0x4C61E46: std::thread::_State_impl<std::thread::_Invoker<std::tuple<nvidia::inferenceserve\
r::(anonymous namespace)::ModelDeleter::operator()(nvidia::inferenceserver::Model*)::{lambda()#1}> > >::M\
run() (in /opt/tritonserver/lib/libtritonserver.so)
==21418== by 0x5BE3DE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==21418== by 0x578A608: start_thread (pthread_create.c:477)
==21418== by 0x5F7D292: clone (clone.S:95)

[deadeyegoodwin]

Is this related to #3777. If so, be sure to link it to this PR.

[tanmayv25]

It fixes that specific invalid read but I still see the issue described in #3777 even with this fix.
I am still working on resolving the main issue.

[GuanLuo]

So the order between instance destruction and rate limiter removing the instance is arbitrary?

[tanmayv25]

I am not sure what you mean by arbitrary. The TritonModelInstance destruction will not remove the entry of the ModelInstanceContext from the RateLimiter. If we remove the context before destroying the instance, backend thread waits over the condition variable that gets destroyed with the context. This can lead to undefined behaviour.

So, it is not arbitrary. Removing context from rate limiter should happen after instance destruction(backend thread is stopped and joined) as is happening in this case. The context is a wrapper over TritonModelInstance.