debug initial_data mistral issue on p1

[emiliecowen]

from log in argo:

INFO:main:Downloading base model from Minio to /home/python/app/core/model_cache/mistrallite.Q4_K_M.gguf
WARNING:urllib3.connectionpool:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf
WARNING:urllib3.connectionpool:Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectionResetError(104, 'Connection reset by peer')': /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/python/app/core/minio.py", line 72, in download_file_from_minio
data = s3.get_object(
^^^^^^^^^^^^^^
File "/home/python/python-packages/minio/api.py", line 1153, in get_object
return self._execute(
^^^^^^^^^^^^^^
File "/home/python/python-packages/minio/api.py", line 408, in _execute
return self._url_open(
^^^^^^^^^^^^^^^
File "/home/python/python-packages/minio/api.py", line 271, in _url_open
response = self._http.urlopen(
^^^^^^^^^^^^^^^^^^^
File "/home/python/python-packages/urllib3/poolmanager.py", line 443, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/python-packages/urllib3/connectionpool.py", line 875, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/python/python-packages/urllib3/connectionpool.py", line 875, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/home/python/python-packages/urllib3/connectionpool.py", line 875, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/python/python-packages/urllib3/connectionpool.py", line 845, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/home/python/python-packages/urllib3/util/retry.py", line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='minio.minio', port=9000): Max retries exceeded with url: /minio-transformers-il4-mission-staging-rs248x0ngi5v4of6/mistrallite.Q4_K_M.gguf (Caused by ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/python/app/initial_data.py", line 560, in
main()
File "/home/python/app/initial_data.py", line 553, in main
init_large_objects_p1(s3, db)
File "/home/python/app/initial_data.py", line 513, in init_large_objects_p1
llm_pretrained_obj = init_llm_db_obj_staging_prod(s3, db, LlmFilenameEnum.Q4_K_M)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/app/initial_data.py", line 308, in init_llm_db_obj_staging_prod
download_file_from_minio(model_name, s3, filename=local_path)
File "/home/python/app/core/minio.py", line 97, in download_file_from_minio
data.close()
^^^^