graphd crashing when counting edges

[joruxs]

Hi

I have been ingesting data to my three server nebula cluster (meta, graph and storage all running on each server). When I try to count the edges the graphd process crashes, seemingly due to memory shortage:

[17300.759108] Out of memory: Killed process 12885 (nebula-graphd) total-vm:24413600kB, anon-rss:12674900kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:36956kB oom_score_adj:0

As I understand it, this is the operating system killing the process due to taking too much memory. The machines have 15,5 gb memory each. What should I do to make my query stable? I am running nebula 3.2.0.

Details:

CREATE TAG User (userId string NOT NULL);
CREATE EDGE liked (likeCount int NOT NULL);

after ingesting data I have done this to be able to count the objects:

CREATE TAG INDEX idx_User ON User ();
CREATE EDGE INDEX idx_liked ON liked ();
REBUILD TAG INDEX idx_User;
REBUILD EDGE INDEX idx_liked;

Then I do:

MATCH ()-[e:liked]->() return count(e);

But get:

022/08/08 12:13:06 Loop error, failed to reconnect, failed to open transport, error: dial tcp 10.83.0.102:9669: connect: connection refused

Bye root!
Mon, 08 Aug 2022 12:13:06 UTC

panic: Loop error, failed to reconnect, failed to open transport, error: dial tcp 10.83.0.102:9669: connect: connection refused

goroutine 1 [running]:
log.Panicf(0x7ea177, 0xe, 0xc000247e58, 0x1, 0x1)
	/usr/local/go/src/log/log.go:345 +0xc0
main.main()
	/usr/src/main.go:572 +0x586

[wey-gu]

Dear @joruxs

For now, the equivalent query of above MATCH:

LOOKUP ON liked YIELD dst(edge) AS DstVID | YIELD count(*)

is more optimal from a resource-consuming perspective(ref).

But still as there is no count pushdown in storage side, they are all expensive in both query patterns now.

#4137

Another option is to leverage the stats job(async job), but it's limited to getting stats per edge type/tag:

(root@nebula) [basketballplayer]> SUBMIT JOB STATS
+------------+
| New Job Id |
+------------+
| 51         |
+------------+
Got 1 rows (time spent 49832/60813 us)

Tue, 09 Aug 2022 09:31:29 CST

(root@nebula) [basketballplayer]> SHOW JOBS
+--------+----------------------+------------+----------------------------+----------------------------+
| Job Id | Command              | Status     | Start Time                 | Stop Time                  |
+--------+----------------------+------------+----------------------------+----------------------------+
| 51     | "STATS"              | "FINISHED" | 2022-08-09T01:31:29.000000 | 2022-08-09T01:31:29.000000 |
| 50     | "REBUILD_EDGE_INDEX" | "FINISHED" | 2022-08-09T01:23:03.000000 | 2022-08-09T01:23:04.000000 |
+--------+----------------------+------------+----------------------------+----------------------------+
Got 2 rows (time spent 2531/9708 us)

(root@nebula) [basketballplayer]> SHOW STATS
+---------+------------+-------+
| Type    | Name       | Count |
+---------+------------+-------+
| "Tag"   | "player"   | 51    |
| "Tag"   | "team"     | 30    |
| "Edge"  | "follow"   | 81    |
| "Edge"  | "serve"    | 152   |
| "Space" | "vertices" | 81    |
| "Space" | "edges"    | 233   |
+---------+------------+-------+
Got 6 rows (time spent 1321/7138 us)

ref: https://docs.nebula-graph.io/3.2.0/3.ngql-guide/7.general-query-statements/6.show/14.show-stats/

[joruxs]

Thanks! The stats job alternative works and is very fast. There are 87 million something "liked" edges in my db.

The LOOKUP alternative seems to collect less memory but it fails everytime with this message:

[ERROR (-1005)]: Storage Error: part: 100, error: E_RPC_FAILURE(-3).

[wey-gu]

@joruxs np :) Yes, we should use SHOW STATS when possible in NebulaGraph, and when not, use lookup with count :).

For RPC failure, you could tune the configure of graphd storage_client_timeout_ms by enlarging it to a timer larger than the default value :), the error applies that the RPC call from graphD to storageD for this query timed out.

ref:

• https://docs.nebula-graph.io/3.2.0/5.configurations-and-logs/1.configurations/3.graph-config/
• https://docs.nebula-graph.io/3.2.0/20.appendix/0.FAQ/#how_to_resolve_the_error_storage_error_e_rpc_failure

[joruxs]

Ah, great explanation, cheers for that!

[wey-gu]

Thanks @joruxs, for your kind word, polished discussion composed, and your contribution to the nebulaGraph community :).