创建GIE不成功，而且有WARNING][coordinator:297]: Connect to analytical engine failed, engine may not started or closed. code: UNAVAILABLE, details: failed to connect to all addresses

[sunnyshark2018]

1、创建interactive报错，无法创建GIE：
interactive = sess.gremlin(g)

2021-09-15 18:04:46,253 [INFO][cluster:375]: Create GIE instance with command: /opt/graphscope/bin/giectl create_gremlin_instance_on_k8s /tmp/gs/fwkpmu/session_fsnvsldl 243021195744897 /tmp/graph_P07U20o0.json gs-engine-fwkpmu-hnbk4,gs-engine-fwkpmu-mwn9j engine gaia.engine.port:40476 False coordinator-fwkpmu
2021-09-15 06:05:46,371 [ERROR][rpc:232]: Runstep failed with code: INTERACTIVE_ENGINE_INTERNAL_ERROR, message: Error occurred during preprocessing, The traceback is: Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 676, in _create_interactive_instance
outs, errs = proc.communicate(timeout=60)
File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib64/python3.6/subprocess.py", line 1535, in _communicate
self._check_timeout(endtime, orig_timeout)
File "/usr/lib64/python3.6/subprocess.py", line 891, in _check_timeout
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['/opt/graphscope/bin/giectl', 'create_gremlin_instance_on_k8s', '/tmp/gs/fwkpmu/session_fsnvsldl', '243021195744897', '/tmp/graph_P07U20o0.json', 'gs-engine-fwkpmu-hnbk4,gs-engine-fwkpmu-mwn9j', 'engine', 'gaia.engine.port:40476', 'False', 'coordinator-fwkpmu']' timed out after 60 seconds

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 512, in RunStep
request.session_id, dag_def, op_results
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 428, in run_on_interactive_engine
op_result = self._create_interactive_instance(op)
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 731, in _create_interactive_instance
raise RuntimeError("Create interactive instance failed.") from e
RuntimeError: Create interactive instance failed.

2、创建session总报错
sess = graphscope.session(k8s_coordinator_cpu=2, k8s_coordinator_mem="4Gi", k8s_volumes=k8s_volumes)
2021-09-15 05:52:14,915 [INFO][session:640]: Initializing graphscope session with parameters: {'addr': None, 'mode': 'eager', 'cluster_type': 'k8s', 'num_workers': 2, 'preemptive': True, 'k8s_namespace': None, 'k8s_service_type': 'NodePort', 'k8s_gs_image': 'registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.7.0', 'k8s_etcd_image': 'quay.io/coreos/etcd:v3.4.13', 'k8s_image_pull_policy': 'IfNotPresent', 'k8s_image_pull_secrets': [], 'k8s_coordinator_cpu': 2, 'k8s_coordinator_mem': '4Gi', 'k8s_etcd_num_pods': 1, 'k8s_etcd_cpu': 1.0, 'k8s_etcd_mem': '512Mi', 'k8s_vineyard_daemonset': 'none', 'k8s_vineyard_cpu': 0.2, 'k8s_vineyard_mem': '512Mi', 'vineyard_shared_mem': '4Gi', 'k8s_engine_cpu': 0.2, 'k8s_engine_mem': '1Gi', 'k8s_mars_worker_cpu': 0.2, 'k8s_mars_worker_mem': '512Mi', 'k8s_mars_scheduler_cpu': 0.2, 'k8s_mars_scheduler_mem': '512Mi', 'with_mars': False, 'enable_gaia': False, 'reconnect': False, 'k8s_volumes': {'data': {'type': 'hostPath', 'field': {'path': '/kg/test_data/', 'type': 'Directory'}, 'mounts': {'mountPath': '/home/graphscope'}}}, 'k8s_waiting_for_delete': False, 'timeout_seconds': 600, 'dangling_timeout_seconds': 600, 'k8s_client_config': {}}
2021-09-15 05:52:19,660 [INFO][cluster:308]: Launching coordinator...
2021-09-15 05:52:26,555 [INFO][utils:167]: coordinator-fwkpmu-6f684945f8-ckcht: Successfully assigned gs-gmknss/coordinator-fwkpmu-6f684945f8-ckcht to k8s-master
2021-09-15 05:52:59,975 [INFO][utils:167]: coordinator-fwkpmu-6f684945f8-ckcht: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.7.0" already present on machine
2021-09-15 05:53:25,171 [INFO][utils:167]: coordinator-fwkpmu-6f684945f8-ckcht: Created container coordinator
2021-09-15 05:53:29,419 [INFO][utils:167]: coordinator-fwkpmu-6f684945f8-ckcht: Started container coordinator
2021-09-15 17:53:37,173 [INFO][cluster:614]: Launching etcd ...
2021-09-15 05:53:41,689 [INFO][utils:167]: coordinator-fwkpmu-6f684945f8-ckcht: Readiness probe failed: dial tcp 10.244.0.43:59990: connect: connection refused
2021-09-15 17:53:43,796 [INFO][cluster:817]: Etcd is ready, endpoint is 10.101.120.243:58729
2021-09-15 17:53:43,797 [INFO][cluster:820]: Creating interactive engine service...
2021-09-15 17:53:43,797 [INFO][cluster:766]: Launching zetcd proxy service ...
2021-09-15 17:53:43,799 [INFO][cluster:781]: zetcd cmd /usr/local/bin/zetcd --zkaddr 0.0.0.0:2181 --endpoints http://gs-etcd-service-fwkpmu:58729,http://gs-etcd-fwkpmu-0:58729
Running zetcd proxy
Version: Version not provided (use make instead of go build)
SHA: SHA not provided (use make instead of go build)
2021-09-15 17:53:44,839 [INFO][cluster:810]: ZEtcd is ready, endpoint is 10.244.0.43:2181
2021-09-15 17:53:44,840 [INFO][cluster:828]: Creating engine replicaset...
2021-09-15 17:53:44,841 [INFO][cluster:505]: Launching GraphScope engines pod ...
2021-09-15 17:53:56,217 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Successfully assigned gs-gmknss/gs-engine-fwkpmu-hnbk4 to k8s-master
2021-09-15 17:53:57,097 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Successfully assigned gs-gmknss/gs-engine-fwkpmu-mwn9j to k8s-master
2021-09-15 17:54:55,970 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.7.0" already present on machine
2021-09-15 17:55:14,603 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Container image "registry.cn-hongkong.aliyuncs.com/graphscope/graphscope:0.7.0" already present on machine
2021-09-15 17:55:32,688 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Created container engine
2021-09-15 17:55:45,774 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Started container engine
2021-09-15 17:55:52,243 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Created container engine
2021-09-15 17:56:01,819 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Started container engine
2021-09-15 17:56:22,701 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Created container vineyard
2021-09-15 17:56:35,629 [INFO][cluster:879]: [gs-engine-fwkpmu-mwn9j]: Started container vineyard
2021-09-15 17:56:39,644 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Created container vineyard
2021-09-15 17:56:39,648 [INFO][cluster:879]: [gs-engine-fwkpmu-hnbk4]: Started container vineyard
2021-09-15 17:56:50,550 [INFO][cluster:915]: GraphScope engines pod is ready.
2021-09-15 17:56:51,283 [INFO][cluster:1052]: Engines pod name list: ['gs-engine-fwkpmu-hnbk4', 'gs-engine-fwkpmu-mwn9j']
2021-09-15 17:56:51,283 [INFO][cluster:1053]: Engines pod ip list: ['10.244.0.45', '10.244.0.44']
2021-09-15 17:56:51,284 [INFO][cluster:1054]: Engines pod host ip list: ['192.168.3.10', '192.168.3.10']
2021-09-15 17:56:51,284 [INFO][cluster:1056]: Vineyard service endpoint: 192.168.3.10:30794
2021-09-15 17:56:51,284 [INFO][cluster:941]: Starting GAE rpc service on 10.244.0.45:56032 ...
2021-09-15 17:56:57,074 [INFO][coordinator:1395]: Coordinator server listen at 0.0.0.0:59990
2021-09-15 05:57:06,978 [INFO][cluster:556]: Coordinator pod start successful with address 192.168.3.10:30184, connecting to service ...
2021-09-15 05:57:07,014 [WARNING][rpc:124]: Heart beat analytical engine failed, code: DEADLINE_EXCEEDED, details: Connect to analytical engine failed, engine may not started or closed. code: UNAVAILABLE, details: failed to connect to all addresses
2021-09-15 05:57:08,031 [WARNING][rpc:124]: Heart beat analytical engine failed, code: DEADLINE_EXCEEDED, details: Connect to analytical engine failed, engine may not started or closed. code: UNAVAILABLE, details: failed to connect to all addresses

[lidongze0629]

Hi, @sunnyshark2018

[WARNING][rpc:124]: Heart beat analytical engine failed, code: DEADLINE_EXCEEDED, details: Connect to analytical engine failed, engine may not started or closed. code: UNAVAILABLE, details: failed to connect to all addresses

This warning is expected as GAE engine may takes some time to launch. After that, line GraphScope coordinator service connected will output and means the client connect to the service successful.

About the failure of launching GIE, could you give more details such as the schema of datasets, and how to deploy your kubernetes cluster, thanks.

[sunnyshark2018]

现在 heart beat告警已经不怎么出现了，但是还是GIE不能被拉起来，报一样的错了

[Pipidaddy]

Hi, bro, these two problems has tortured me for two days, sloved now?

[Pipidaddy]

对不起不知道为啥习惯性打了英文···

[sighingnow]

Its ok to discuss issues in English @Pipidaddy, I believe @lidongze0629 has some insights about why the error happened.

[sunnyshark2018]

没呢，还没找到原因，0.7.0没跑起来。
我还换成0.4.0，GIE起来了，但是过一段时间coordinator和namespace自动给删除了，好像也是说心跳失败。

[sunnyshark2018]

手动在coordinator的容器上，执行命令拉起GIE，是可以成功的。
/opt/graphscope/bin/giectl close_gremlin_instance_on_k8s /tmp/gs/gmdqrc/session_ckmezkan 161189477531346 gs-engine-gmdqrc-dkjdb,gs-engine-gmdqrc-tkgkv engine

可以看到GIE的excutor在两个pod上拉起来了。
但是通过执行interactive = sess.gremlin(g)，拉不起来，这个就比较奇怪。
怀疑要吗是超时60s被删除了，或者是没来起来

[sunnyshark2018]

参数改一下，看到起来了又被停止了：
sess = graphscope.session(num_workers=1,k8s_coordinator_cpu=4,k8s_coordinator_mem="4Gi",k8s_engine_cpu=4,k8s_engine_mem="4Gi",k8s_volumes=k8s_volumes)

报错：
2021-10-02 08:31:04,793 [INFO][cluster:375]: Create GIE instance with command: /opt/graphscope/bin/giectl create_gremlin_instance_on_k8s /tmp/gs/rxrbtc/session_uinbeqfs 14673353446156 /tmp/graph_41mtzxlY.json gs-engine-rxrbtc-t4jb8 engine gaia.engine.port:40048 False coordinator-rxrbtc
2021-10-01 20:31:36,017 [ERROR][rpc:232]: Runstep failed with code: INTERACTIVE_ENGINE_INTERNAL_ERROR, message: Error occurred during preprocessing, The traceback is: Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 726, in _create_interactive_instance
"Error code: {0}, message {1}".format(return_code, outs)
RuntimeError: Error code: 1, message [2021-10-02T08:31:35+0800]: Launch interactive engine(executor) in per engine pod.
[2021-10-02T08:31:35+0800]: Expose gremlin server.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 512, in RunStep
request.session_id, dag_def, op_results
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 428, in run_on_interactive_engine
op_result = self._create_interactive_instance(op)
File "/home/graphscope/.local/lib/python3.6/site-packages/gscoordinator/coordinator.py", line 731, in _create_interactive_instance
raise RuntimeError("Create interactive instance failed.") from e
RuntimeError: Create interactive instance failed.

中间看到executor起来了，后来又被停止了：
UID PID PPID C STIME TTY TIME CMD
graphsc+ 1 0 0 08:51 ? 00:00:00 tail -f /dev/null
graphsc+ 89 0 0 08:52 ? 00:00:00 /bin/sh -c cat /tmp/hosts_of_nodes | sudo tee -a /etc/hosts && orted -mca ess "env" -mca ess_base_jobid "2413756416" -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca orte_node_regex "co
graphsc+ 97 89 0 08:52 ? 00:00:00 orted -mca ess env -mca ess_base_jobid 2413756416 -mca ess_base_vpid 1 -mca ess_base_num_procs 2 -mca orte_node_regex coordinator-litkhf-[2:84]d549c6f4-v7sg7,gs-engine-litkhf-[2:74]q5n@0(2) -
graphsc+ 100 97 0 08:52 ? 00:00:01 grape_engine --host 0.0.0.0 --port 56652 -v 10 --vineyard_socket /tmp/vineyard_workspace/vineyard.sock
graphsc+ 1015 0 1 08:54 pts/1 00:00:00 /bin/bash
graphsc+ 1050 0 0 08:54 ? 00:00:00 /opt/graphscope/bin/executor --config /tmp/gs/litkhf/session_htjmwqrk/config/19002757382210/executor.vineyard.properties maxgraph19002757382210executor 1
graphsc+ 1052 1015 0 08:54 pts/1 00:00:00 ps -ef
root 1056 0 0 08:54 ? 00:00:00 runc init
graphsc+ 1074 1050 0 08:54 ? 00:00:00 [executor]