Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docs: Feedback for Vector search tutorial #1350

Open
colljos opened this issue Oct 25, 2024 · 9 comments
Open

Docs: Feedback for Vector search tutorial #1350

colljos opened this issue Oct 25, 2024 · 9 comments
Assignees
@wrmay @naadhira
Labels
bug Something isn't working

Comments

@colljos
Copy link

colljos commented Oct 25, 2024

Hi, I have some feedback about this page

Tutorial Setup step 2 continuously fails:

Download the CLIP model

docker compose run download-model

Error output:

Joses-MacBook-Pro:hazelcast-image-search colljos$ docker compose run download-model
[+] Building 14.6s (6/8)                                                                                                                              docker:desktop-linux
 => [download-model internal] load build definition from Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 194B                                                                                                                                  0.0s
 => [download-model internal] load metadata for docker.io/library/python:3                                                                                            0.8s
 => [download-model auth] library/python:pull token for registry-1.docker.io                                                                                          0.0s
 => [download-model internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => CACHED [download-model 1/4] FROM docker.io/library/python:3@sha256:a31cbb4db18c6f09e3300fa85b77f6d56702501fcb9bdb8792ec702a39ba6200                               0.0s
 => ERROR [download-model 2/4] RUN pip install sentence-transformers jupyter pytest                                                                                  13.7s
------                                                                                                                                                                     
 > [download-model 2/4] RUN pip install sentence-transformers jupyter pytest:                                                                                              
1.493 Collecting sentence-transformers                                                                                                                                     
1.604   Downloading sentence_transformers-3.2.1-py3-none-any.whl.metadata (10 kB)                                                                                          
1.646 Collecting jupyter                                                                                                                                                   
1.669   Downloading jupyter-1.1.1-py2.py3-none-any.whl.metadata (2.0 kB)                                                                                                   
1.779 Collecting pytest
1.800   Downloading pytest-8.3.3-py3-none-any.whl.metadata (7.5 kB)
1.986 Collecting transformers<5.0.0,>=4.41.0 (from sentence-transformers)
2.009   Downloading transformers-4.46.0-py3-none-any.whl.metadata (44 kB)
2.254 Collecting tqdm (from sentence-transformers)
2.275   Downloading tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
2.397 Collecting torch>=1.11.0 (from sentence-transformers)
2.417   Downloading torch-2.5.0-cp313-cp313-manylinux1_x86_64.whl.metadata (28 kB)
2.596 Collecting scikit-learn (from sentence-transformers)
2.615   Downloading scikit_learn-1.5.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
2.837 Collecting scipy (from sentence-transformers)
2.858   Downloading scipy-1.14.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
2.982 Collecting huggingface-hub>=0.20.0 (from sentence-transformers)
3.003   Downloading huggingface_hub-0.26.1-py3-none-any.whl.metadata (13 kB)
3.406 Collecting Pillow (from sentence-transformers)
3.427   Downloading pillow-11.0.0-cp313-cp313-manylinux_2_28_x86_64.whl.metadata (9.1 kB)
3.520 Collecting notebook (from jupyter)
3.541   Downloading notebook-7.2.2-py3-none-any.whl.metadata (10 kB)
3.588 Collecting jupyter-console (from jupyter)
3.607   Downloading jupyter_console-6.6.3-py3-none-any.whl.metadata (5.8 kB)
3.703 Collecting nbconvert (from jupyter)
3.722   Downloading nbconvert-7.16.4-py3-none-any.whl.metadata (8.5 kB)
3.814 Collecting ipykernel (from jupyter)
3.833   Downloading ipykernel-6.29.5-py3-none-any.whl.metadata (6.3 kB)
3.910 Collecting ipywidgets (from jupyter)
3.935   Downloading ipywidgets-8.1.5-py3-none-any.whl.metadata (2.3 kB)
4.174 Collecting jupyterlab (from jupyter)
4.194   Downloading jupyterlab-4.2.5-py3-none-any.whl.metadata (16 kB)
4.250 Collecting iniconfig (from pytest)
4.269   Downloading iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB)
4.325 Collecting packaging (from pytest)
4.345   Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
4.395 Collecting pluggy<2,>=1.5 (from pytest)
4.413   Downloading pluggy-1.5.0-py3-none-any.whl.metadata (4.8 kB)
4.471 Collecting filelock (from huggingface-hub>=0.20.0->sentence-transformers)
4.489   Downloading filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
4.554 Collecting fsspec>=2023.5.0 (from huggingface-hub>=0.20.0->sentence-transformers)
4.572   Downloading fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
4.698 Collecting pyyaml>=5.1 (from huggingface-hub>=0.20.0->sentence-transformers)
4.717   Downloading PyYAML-6.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
4.790 Collecting requests (from huggingface-hub>=0.20.0->sentence-transformers)
4.811   Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
4.891 Collecting typing-extensions>=3.7.4.3 (from huggingface-hub>=0.20.0->sentence-transformers)
4.909   Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
4.984 Collecting networkx (from torch>=1.11.0->sentence-transformers)
5.003   Downloading networkx-3.4.2-py3-none-any.whl.metadata (6.3 kB)
5.055 Collecting jinja2 (from torch>=1.11.0->sentence-transformers)
5.074   Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
5.113 Collecting nvidia-cuda-nvrtc-cu12==12.4.127 (from torch>=1.11.0->sentence-transformers)
5.131   Downloading nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
5.170 Collecting nvidia-cuda-runtime-cu12==12.4.127 (from torch>=1.11.0->sentence-transformers)
5.189   Downloading nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
5.222 Collecting nvidia-cuda-cupti-cu12==12.4.127 (from torch>=1.11.0->sentence-transformers)
5.240   Downloading nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
5.279 Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch>=1.11.0->sentence-transformers)
5.297   Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
5.337 Collecting nvidia-cublas-cu12==12.4.5.8 (from torch>=1.11.0->sentence-transformers)
5.355   Downloading nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
5.393 Collecting nvidia-cufft-cu12==11.2.1.3 (from torch>=1.11.0->sentence-transformers)
5.411   Downloading nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
5.450 Collecting nvidia-curand-cu12==10.3.5.147 (from torch>=1.11.0->sentence-transformers)
5.472   Downloading nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
5.510 Collecting nvidia-cusolver-cu12==11.6.1.9 (from torch>=1.11.0->sentence-transformers)
5.529   Downloading nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
5.574 Collecting nvidia-cusparse-cu12==12.3.1.170 (from torch>=1.11.0->sentence-transformers)
5.593   Downloading nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)
5.631 Collecting nvidia-nccl-cu12==2.21.5 (from torch>=1.11.0->sentence-transformers)
5.650   Downloading nvidia_nccl_cu12-2.21.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
5.691 Collecting nvidia-nvtx-cu12==12.4.127 (from torch>=1.11.0->sentence-transformers)
5.709   Downloading nvidia_nvtx_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.7 kB)
5.747 Collecting nvidia-nvjitlink-cu12==12.4.127 (from torch>=1.11.0->sentence-transformers)
5.765   Downloading nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
6.066 Collecting setuptools (from torch>=1.11.0->sentence-transformers)
6.085   Downloading setuptools-75.2.0-py3-none-any.whl.metadata (6.9 kB)
6.152 Collecting sympy==1.13.1 (from torch>=1.11.0->sentence-transformers)
6.174   Downloading sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
6.227 Collecting mpmath<1.4,>=1.1.0 (from sympy==1.13.1->torch>=1.11.0->sentence-transformers)
6.247   Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
6.583 Collecting numpy>=1.17 (from transformers<5.0.0,>=4.41.0->sentence-transformers)
6.602   Downloading numpy-2.1.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
7.350 Collecting regex!=2019.12.17 (from transformers<5.0.0,>=4.41.0->sentence-transformers)
7.377   Downloading regex-2024.9.11-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
7.628 Collecting safetensors>=0.4.1 (from transformers<5.0.0,>=4.41.0->sentence-transformers)
7.649   Downloading safetensors-0.4.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
7.999 Collecting tokenizers<0.21,>=0.20 (from transformers<5.0.0,>=4.41.0->sentence-transformers)
8.019   Downloading tokenizers-0.20.1.tar.gz (339 kB)
8.215   Installing build dependencies: started
12.89   Installing build dependencies: finished with status 'done'
12.89   Getting requirements to build wheel: started
12.95   Getting requirements to build wheel: finished with status 'done'
12.95   Preparing metadata (pyproject.toml): started
12.99   Preparing metadata (pyproject.toml): finished with status 'error'
12.99   error: subprocess-exited-with-error
12.99   
12.99   × Preparing metadata (pyproject.toml) did not run successfully.
12.99   │ exit code: 1
12.99   ╰─> [6 lines of output]
12.99       
12.99       Cargo, the Rust package manager, is not installed or is not on PATH.
12.99       This package requires Rust and Cargo to compile extensions. Install it through
12.99       the system's package manager or via https://rustup.rs/
12.99       
12.99       Checking for Rust toolchain....
12.99       [end of output]
12.99   
12.99   note: This error originates from a subprocess, and is likely not a problem with pip.
13.19 error: metadata-generation-failed
13.19 
13.19 × Encountered error while generating package metadata.
13.19 ╰─> See above for output.
13.19 
13.19 note: This is an issue with the package mentioned above, not pip.
13.19 hint: See above for details.
------
failed to solve: process "/bin/sh -c pip install sentence-transformers jupyter pytest" did not complete successfully: exit code: 1

Here is my environment:

Joses-MacBook-Pro:hazelcast-image-search colljos$ pip --version
pip 24.2 from /usr/local/lib/python3.13/site-packages/pip (python 3.13)
Joses-MacBook-Pro:hazelcast-image-search colljos$ cargo --version
cargo 1.82.0
@colljos
Copy link
Author

colljos commented Nov 17, 2024 •
edited
Loading

Now got this working by tweaking the Python install command in the python-container sub-directory Dockerfile as follows:

RUN python3 -m venv tmp; export PATH=$PATH:tmp/bin/activate:/Users/colljos/Library/Python/3.9/lib/python/site-packages; python3 -m pip install sentence-transformers jupyter pytest

Output of running docker compose run download-model:

Joses-MacBook-Pro:hazelcast-image-search colljos$ docker compose run download-model
[+] Building 1657.2s (10/10) FINISHED                                                                                                                 docker:desktop-linux
 => [download-model internal] load build definition from Dockerfile                                                                                                   0.0s
 => => transferring dockerfile: 321B                                                                                                                                  0.0s
 => [download-model internal] load metadata for docker.io/library/python:3                                                                                            1.7s
 => [download-model auth] library/python:pull token for registry-1.docker.io                                                                                          0.0s
 => [download-model internal] load .dockerignore                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                       0.0s
 => [download-model 1/4] FROM docker.io/library/python:3@sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0                                    172.8s
 => => resolve docker.io/library/python:3@sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0                                                     0.0s
 => => sha256:b2b31b28ee3c96e96195c754f8679f690db4b18e475682d716122016ef056f39 49.58MB / 49.58MB                                                                     56.6s
 => => sha256:47db274575b70d4943bb3d408cdd590b8285b5811f10c2e7167ba7f0cada9f2c 2.32kB / 2.32kB                                                                        0.0s
 => => sha256:c41ea82733659810377a74cad7ee2a4d1d6f582395de8d12091be78c315cfee2 6.05kB / 6.05kB                                                                        0.0s
 => => sha256:2112e5e7c3ff699043b282f1ff24d3ef185c080c28846f1d7acc5ccf650bc13d 64.39MB / 64.39MB                                                                     92.8s
 => => sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0 9.72kB / 9.72kB                                                                        0.0s
 => => sha256:c3cc7b6f04730c072f8b292917e0d95bb886096a2b2b1781196170965161cd27 24.06MB / 24.06MB                                                                     27.0s
 => => sha256:af247aac076473044d24960a352a8ec6f154cf0a28f4fbf35fe5d43b52687ba2 211.29MB / 211.29MB                                                                  160.4s
 => => sha256:ef45f15f570bfc171a0a3444eb77562e1fdf865d7339942ce60eafbc8c84f198 6.16MB / 6.16MB                                                                       63.9s
 => => extracting sha256:b2b31b28ee3c96e96195c754f8679f690db4b18e475682d716122016ef056f39                                                                             3.4s
 => => extracting sha256:c3cc7b6f04730c072f8b292917e0d95bb886096a2b2b1781196170965161cd27                                                                             0.9s
 => => sha256:4d87d670c3ff8f79d31e0987e6da501ab7f7b5902398c8b3bfbd6c4566a54f26 27.19MB / 27.19MB                                                                     95.1s
 => => extracting sha256:2112e5e7c3ff699043b282f1ff24d3ef185c080c28846f1d7acc5ccf650bc13d                                                                             4.0s
 => => sha256:3ed3f00b0d2c8d764889e205f62819c43c3548445d2b31da54527d5b68248978 248B / 248B                                                                           93.3s
 => => extracting sha256:af247aac076473044d24960a352a8ec6f154cf0a28f4fbf35fe5d43b52687ba2                                                                            10.3s
 => => extracting sha256:ef45f15f570bfc171a0a3444eb77562e1fdf865d7339942ce60eafbc8c84f198                                                                             0.4s
 => => extracting sha256:4d87d670c3ff8f79d31e0987e6da501ab7f7b5902398c8b3bfbd6c4566a54f26                                                                             1.3s
 => => extracting sha256:3ed3f00b0d2c8d764889e205f62819c43c3548445d2b31da54527d5b68248978                                                                             0.0s
 => [download-model 2/4] RUN python3 -m venv tmp; export PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:tmp/bin/activate:/Users  1426.1s
 => [download-model 3/4] WORKDIR /root                                                                                                                                0.1s 
 => [download-model 4/4] RUN git clone https://github.com/hazelcast/hazelcast-python-client.git                                                                       3.9s 
 => [download-model] exporting to image                                                                                                                              52.3s 
 => => exporting layers                                                                                                                                              52.3s 
 => => writing image sha256:8c2615b4a4fcd43cebaf8e741cdca684a12d2a0cfbd8f919cebf6a7b5a80fbbc                                                                          0.0s 
 => => naming to docker.io/library/hazelcast-image-search-download-model                                                                                              0.0s 
 => [download-model] resolving provenance for metadata file                                                                                                           0.0s
downloading model ...
modules.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 122/122 [00:00<00:00, 470kB/s]
config_sentence_transformers.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<00:00, 466kB/s]
README.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.91k/1.91k [00:00<00:00, 12.7MB/s]
0_CLIPModel/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.03k/4.03k [00:00<00:00, 14.0MB/s]
0_CLIPModel/preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 2.10MB/s]
0_CLIPModel/special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 389/389 [00:00<00:00, 4.00MB/s]
0_CLIPModel/tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 604/604 [00:00<00:00, 5.63MB/s]
0_CLIPModel/merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 932kB/s]
0_CLIPModel/vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 961k/961k [00:01<00:00, 783kB/s]
pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 605M/605M [04:44<00:00, 2.13MB/s]
done. saving ...n: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 605M/605M [04:44<00:00, 2.40MB/s]
done
Joses-MacBook-Pro:hazelcast-image-search colljos$ pip --version
pip 24.2 from /Users/colljos/Library/Python/3.9/lib/python/site-packages/pip (python 3.9)
Joses-MacBook-Pro:hazelcast-image-search colljos$ cargo --version
cargo 1.82.0

@colljos
Copy link
Author

colljos commented Nov 18, 2024

I am now experiencing a run-time exception upon executing the Deploy Pipeline step (4):

deploy the pipeline: docker compose run submit-image-loader

Output:

Joses-MacBook-Pro:hazelcast-image-search colljos$ docker compose run submit-image-loader
WARN[0000] Found orphan containers ([hazelcast-image-search-download-model-run-9f897877ad42]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
Verbose mode is on, setting logging level to INFO
Submitting JAR '/project/image-ingest-pipeline/target/image-ingest-pipeline-1.0-SNAPSHOT.jar' with arguments [/project/www, .jpg, http://www:8000, /project/image-embedding-service, clip_image_encoder]
17:41:04.811 [ INFO] [c.h.i.c.AbstractConfigLocator] Loading configuration '/opt/hazelcast/config/hazelcast-client.xml' from System property 'hazelcast.client.config'
17:41:04.840 [ INFO] [c.h.i.c.AbstractConfigLocator] Using configuration file at /opt/hazelcast/config/hazelcast-client.xml
17:41:05.767 [ INFO] [c.h.c.i.s.ClientInvocationService] hz.client_1 [dev] [5.5.2] Running with 2 response threads, dynamic=true
17:41:05.837 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is STARTING
17:41:05.841 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is STARTED
17:41:05.907 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Trying to connect to cluster: dev
17:41:05.914 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Trying to connect to [hz]:5701
17:41:05.945 [ INFO] [c.h.c.i.s.ClientClusterService] hz.client_1 [dev] [5.5.2] 

Members [1] {
	Member [172.24.0.2]:5701 - ba03f9d3-cda1-4651-be4f-1a82cf8f34ae
}

17:41:05.947 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is CLIENT_CONNECTED
17:41:05.948 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Authenticated with server [172.24.0.2]:5701:ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, server version: 5.5.2, local address: /172.24.0.6:47699
17:41:05.951 [ INFO] [c.h.i.d.Diagnostics] hz.client_1 [dev] [5.5.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
17:41:06.081 [ INFO] [c.h.c.i.s.ClientStatisticsService] Client statistics is enabled with period 5 seconds.
17:41:06.110 [ INFO] [c.h.i.i.e.CommandLineExecuteJar] Found mainClassName :"hazelcast.platform.labs.image.similarity.ImageIngestPipeline" and main method
17:41:06.613 [ERROR] [c.h.i.i.e.CommandLineExecuteJar] Exception caught while executing the jar 
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.invokeMain(CommandLineExecuteJar.java:117)
	at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.executeJar(CommandLineExecuteJar.java:83)
	at com.hazelcast.instance.impl.HazelcastBootstrap.executeJarOnCLI(HazelcastBootstrap.java:96)
	at com.hazelcast.client.console.HazelcastCommandLine.submit(HazelcastCommandLine.java:196)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2070)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2540)
	at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2542)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2536)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2496)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2268)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2668)
	at com.hazelcast.client.console.HazelcastCommandLine.runCommandLine(HazelcastCommandLine.java:557)
	at com.hazelcast.client.console.HazelcastCommandLine.main(HazelcastCommandLine.java:134)
Caused by: java.lang.IllegalArgumentException: These transforms have nothing attached to them: [Watch for Changes]
	at com.hazelcast.jet.impl.pipeline.Planner.validateNoLeakage(Planner.java:244)
	at com.hazelcast.jet.impl.pipeline.Planner.createDag(Planner.java:85)
	at com.hazelcast.jet.impl.pipeline.PipelineImpl.toDag(PipelineImpl.java:133)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitJob$1(JobCoordinationService.java:259)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522)
	at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64)
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
17:41:06.615 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is SHUTTING_DOWN
17:41:06.643 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Removed connection to endpoint: [172.24.0.2]:5701:ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.24.0.6:47699->hz/172.24.0.2:5701}, remoteAddress=[172.24.0.2]:5701, remoteUuid=ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, clusterUuid=78cdcc19-ec89-4706-89de-af20f78f1762, lastReadTime=2024-11-18 17:41:06.609, lastWriteTime=2024-11-18 17:41:06.542, closedTime=2024-11-18 17:41:06.617}
17:41:06.644 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is CLIENT_DISCONNECTED
17:41:06.648 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is SHUTDOWN
picocli.CommandLine$ExecutionException: Error while calling command (public void com.hazelcast.client.console.HazelcastCommandLine.submit(com.hazelcast.client.console.HazelcastCommandLine$GlobalMixin,java.lang.String,java.lang.String,java.lang.String,java.io.File,java.util.List) throws java.io.IOException,java.lang.ClassNotFoundException,java.lang.reflect.InvocationTargetException,java.lang.IllegalAccessException): java.lang.reflect.InvocationTargetException
	at picocli.CommandLine.executeUserObject(CommandLine.java:2082)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2540)
	at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2542)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2536)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2496)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2268)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2668)
	at com.hazelcast.client.console.HazelcastCommandLine.runCommandLine(HazelcastCommandLine.java:557)
	at com.hazelcast.client.console.HazelcastCommandLine.main(HazelcastCommandLine.java:134)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.invokeMain(CommandLineExecuteJar.java:117)
	at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.executeJar(CommandLineExecuteJar.java:83)
	at com.hazelcast.instance.impl.HazelcastBootstrap.executeJarOnCLI(HazelcastBootstrap.java:96)
	at com.hazelcast.client.console.HazelcastCommandLine.submit(HazelcastCommandLine.java:196)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2070)
	... 9 more
Caused by: java.lang.IllegalArgumentException: These transforms have nothing attached to them: [Watch for Changes]
	at com.hazelcast.jet.impl.pipeline.Planner.validateNoLeakage(Planner.java:244)
	at com.hazelcast.jet.impl.pipeline.Planner.createDag(Planner.java:85)
	at com.hazelcast.jet.impl.pipeline.PipelineImpl.toDag(PipelineImpl.java:133)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitJob$1(JobCoordinationService.java:259)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501)
	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522)
	at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64)
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
Joses-MacBook-Pro:hazelcast-image-search colljos$

@colljos
Copy link
Author

colljos commented Nov 18, 2024 •
edited
Loading

OK ... so I can see from re-reading the tutorial instructions, that step 3) states:

3) Using your Java IDE, open ImagesIngestPipeline.java in the image-ingest-pipeline module. Follow the guidance and instructions in the file.

The createPipeline method is basically incomplete.

    private static Pipeline createPipeline(
            String dir,
            String suffix,
            String wwwServer,
            String pythonServiceBaseDir,
            String pythonServiceModule){
        Pipeline pipeline = Pipeline.create();

        /*
         * This custom source emits (event type, changed file name) tuples.  Currently, the
         * only event type returned is ADD.
         *
         * This source is non-distributed, which means it will run on one arbitrarily chosen node in the cluster.
         */
        StreamSource<Tuple2<DirectoryWatcher.EventType, String>> directoryWatcher =
                DirectoryWatcherSourceBuilder.newDirectoryWatcher(dir, suffix);
        StreamStage<Tuple2<DirectoryWatcher.EventType, String>> changeEvents =
                pipeline.readFrom(directoryWatcher).withoutTimestamps().setName("Watch for Changes");

        /*
         * TODO: The events all originate on a single server. Unless forced,  subsequent processing will
         *        remain on the same server.  In order to distribute the work, rebalance the stream.
         *        See: https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/pipeline/StreamStage.html#rebalance()
         */
        StreamStage<Tuple2<DirectoryWatcher.EventType, String>> rebalancedEvents = null;

        /*
         * TODO: Change the format of the input to match what is expected by the python embedding
         *       service.  Use a "map" stage (https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/pipeline/StreamStage.html#map(com.hazelcast.function.FunctionEx)
         *       to change the filename into a URL.
         *       Example: if the filename  = "myimage.jpg", the URL would be wwwServer + "/" + filename
         *
         *       Note that the input event is a Tuple2 (https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/datamodel/Tuple2.html)
         *       The filename is in the second part of the tuple and can be accessed with tuple.f1()
         */
        StreamStage<String> inputs = null;

        /*
         * Now use mapUsingPython to call the python embedding service.
         * See the example here: https://docs.hazelcast.com/hazelcast/5.4/pipelines/python
         *
         * Notes: create the PythonServiceConfig with the following arguments
         *        baseDir: This is the name of the folder (on the client that submits the job) where the
         *                 python service code resides.  In this case, it is passed in to the createPipeline
         *                 method as an argument.
         *
         *        handlerModule: the name of the python file (without .py) containing the transform_list function.
         *                 This is also passed to the createPipeline method.
         *
         *        You can set the number of python instances to run on each node using the "setLocalParallelism"
         *        method of the StreamStage.  Set local parallelism to 2.
         *
         *        After creating a PythonServiceConfig, invoke the python service with code similar to
         *        the example below.
         *
         *        inputs.apply(PythonTransforms.mapUsingPython(myPythonServiceConfig).setLocalParallelism(2);
         */
        PythonServiceConfig pythonService = null;    // create the python service config
        StreamStage<String> outputs = null;          // call map using python

        /*
         * For debugging purposes, finish the pipeline with a logging sink and then test it by deploying
         * it per the instructions in the README.  Remove this sink in the final pipeline.
         */
        //outputs.writeTo(Sinks.logger());

        /*
         * Use mapUsingService to decode the output.  The python stage emits json encoded strings.
         * Use an instance of EmbeddingServiceCodec to translate the json into a (filename, float[]) 2-tuple.
         *
         * See: https://docs.hazelcast.com/hazelcast/5.4/pipelines/transforms#mapusingservice for
         * an example
         *
         * First use ServiceFactories.sharedService to create an instance of EmbeddingServiceCodec.  One instance
         * will be created on each node in the cluster.  Then use a mapUsingService stage to perform the decoding.
         * Your map method will be passed the shared instance of EmbeddingServiceCodec and an event.  You should
         * call embeddingCodec.decodeOutput(myEvent)
         */
        ServiceFactory<?, EmbeddingServiceCodec> postProcessor = null;
        StreamStage<Tuple2<String, float[]>> vectors = null;

        /*
         * Finally, create a vector collection sink and write the pipeline events to it.
         * See: https://docs.hazelcast.com/hazelcast/5.5-snapshot/integrate/vector-collection-connector#vector-collection-as-a-sink
         *
         * Call the VectorSinks.vectorCollection method with the following argument
         *      name of the collection (see hazelcast.yaml)
         *      function to extract the key - in this case, use tuple.f0() to return the URL of the image
         *      function to extract the value - pass the URL of the image as the value as well
         *      function to build an instance of VectorValue from the array of floats in tuple.f1()
         *
         *  You can construct a VectorValue instance with VectorValues.Of(an-array-of-floats)
         *
         * When you have created a vector sink, finish the pipeline by writing it to the sink.
         */
        Sink<Tuple2<String, float[]>> vectorCollectionSink = null;

        //vectors.writeTo(vectorCollectionSink);

        return pipeline;
    }

This feels impossible to complete by a novice to the platform.
What previous tutorials should I have completed before attempting this one?
It would be useful to provide the complete solution as well (to experience running the demo without too much prior knowledge).

NB: one of the references within the code comments refer to an invalid URL (404: Page Not Found)
https://docs.hazelcast.com/hazelcast/5.5-snapshot/integrate/vector-collection-connector#vector-collection-as-a-sink

@naadhira
Copy link
Contributor

Thank you for your feedback. We're looking into this.

@wrmay
Copy link

wrmay commented Nov 19, 2024

So, two issues here. The first is docker compose run download-model doesn't work for Jose. To address this, I've added specific versions of python and the python packages in python-container/Dockerfile. That fixed it for me (M1 Mac) but there could be differences with platforms. We should at least test with an Intel Mac. Could someone with an Intel Mac please pull the latest and try it? Note - Docker will cache the broken image so before you retest, run docker image ls | grep hazelcast-image-search and remove any images associated with this project with docker image rm <image-id-to-delete>. You may have to run docker container prune before you can delete the image. Anyway, to recap, it works for me on my m1 mac but we need someone with an Intel Mac to pull the latest and retest.

For the second issue, for those that don't want to write the image ingest pipeline themselves, there is a solution in the project here: image-ingest-pipeline/src/main/java/hazelcast/platform/labs/image/similarity/solution/ImageIngestPipelineSolution.java and there is also a task to submit the solution in the docker compose file. The solution can be run using docker compose run submit-image-loader-solution. I think we just need to document that somewhere in the tutorial.

@colljos
Copy link
Author

colljos commented Nov 20, 2024

@wrmay Thanks for the feedback on this.

I have now applied the Solution code to the pipeline but am seeing a PythonService initialisation error upon submitting the image loader command.

Sequence of commands:

 docker compose run submit-image-loader-solution
 docker compose logs --follow hz

Log output of hz container:

hz-1  | 2024-11-20 11:07:54,767 [ INFO] [hz.keen_lichterman.cached.thread-21] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 262
hz-1  | 2024-11-20 11:07:54,767 [ INFO] [hz.keen_lichterman.cached.thread-19] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 261
hz-1  | 2024-11-20 11:08:12,915 [ERROR] [hz.keen_lichterman.cached.thread-17] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1  | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]

I've checked the configuration and validated the expected Python file clip_image_encoder is available on the Docker image:

Screenshot 2024-11-20 at 11 20 43

@wrmay
Copy link

wrmay commented Nov 20, 2024

The solution also didn't work for me. We may have a regression. @colljos Thanks for bringing this to our attention. This was developed against a pre-release snapshot. It worked then but it appears broken now. I'll need a little time figure out why it no longer works. If you could include the whole stack trace from the Hazelcast logs that would help me to confirm whether you are seeing the same thing as I am.

@colljos
Copy link
Author

colljos commented Nov 20, 2024

Please find attached full stack trace:

hz-1  | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-19] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 678
hz-1  | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-20] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 677
hz-1  | 2024-11-20 11:19:48,681 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1  | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1  | 	at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | Caused by: java.io.IOException: Python process died before completing initialization
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	... 21 more
hz-1  | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	... 21 more
hz-1  | 2024-11-20 11:19:48,684 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1  | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1  | 	at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | Caused by: java.io.IOException: Python process died before completing initialization
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	... 21 more
hz-1  | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	... 21 more
hz-1  | 2024-11-20 11:19:50,285 [ERROR] [hz.keen_lichterman.cached.thread-19] [c.h.j.i.MasterJobContext]: [172.24.0.3]:5701 [dev] [5.5.2] Execution of job '0ca6-91a3-6540-0001', execution 0ca6-91a3-d280-0001 failed
hz-1  | 	Start time: 2024-11-20T11:18:00.260
hz-1  | 	Duration: 00:01:50.025
hz-1  | 	To see additional job metrics enable JobConfig.storeMetricsAfterJobCompletion
hz-1  | com.hazelcast.jet.JetException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:235) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.submitCooperativeTasklets(TaskletExecutionService.java:196) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.beginExecute(TaskletExecutionService.java:154) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ExecutionContext.beginExecution(ExecutionContext.java:233) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.JobExecutionService.beginExecution0(JobExecutionService.java:568) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.JobExecutionService.beginExecution(JobExecutionService.java:563) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.operation.StartExecutionOperation.doRun(StartExecutionOperation.java:50) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.operation.AsyncOperation.run(AsyncOperation.java:56) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:192) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:291) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:262) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:216) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:459) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:519) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:617) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:596) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:557) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:240) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:71) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.MasterContext.invokeOnParticipant(MasterContext.java:479) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:462) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:433) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.MasterJobContext.invokeStartExecution(MasterJobContext.java:638) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.MasterJobContext.lambda$onInitStepCompleted$10(MasterJobContext.java:606) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | Caused by: com.hazelcast.jet.impl.execution.TaskletExecutionException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:234) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	... 33 more
hz-1  | Caused by: com.hazelcast.jet.JetException: PythonService initialization failed
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1  | 	... 8 more
hz-1  | Caused by: java.io.IOException: Python process died before completing initialization
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1  | 	... 8 more
hz-1  | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1  | 	at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1  | 	at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1  | 	at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1  | 	at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1  | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1  | 	... 8 more

@wrmay
Copy link

wrmay commented Nov 21, 2024 via email

@Rob-Hazelcast Rob-Hazelcast added the bug Something isn't working label Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wrmay wrmay

@naadhira naadhira

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@colljos @wrmay @naadhira @Rob-Hazelcast