Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](export)fix core when export jsonb field #29365

Merged
merged 1 commit into from
Jan 3, 2024

Conversation

ryanzryu
Copy link
Contributor

@ryanzryu ryanzryu commented Jan 1, 2024

Proposed changes

export table whit jsonb field will be core,get stack like this:

*** Query id: 0-0 ***
*** Aborted at 1702436737 (unix time) try "date -d @1702436737" if you are using GNU date ***
*** Current BE git commitID: 0b1372c ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 449814 (TID 0x7fee0bb14700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /data/doris-1.x/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
4# 0x00007FEE4AB51400 in /lib64/libc.so.6
5# memcpy at /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# doris::FieldTypeTraits<(doris::FieldType)13>::direct_copy(void*, void const*) at /data/doris-1.x/be/src/olap/types.h:1439
7# void doris::direct_copy_row<doris::RowCursor, doris::RowCursor>(doris::RowCursor*, doris::RowCursor const&) at /data/doris-1.x/be/src/olap/row.h:120
8# doris::TupleReader::_unique_key_next_row(doris::RowCursor*, doris::MemPool*, doris::ObjectPool*, bool*) at /data/doris-1.x/be/src/olap/tuple_reader.cpp:190
9# doris::OlapScanner::get_batch(doris::RuntimeState*, doris::RowBatch*, bool*) at /data/doris-1.x/be/src/exec/olap_scanner.cpp:372
10# doris::OlapScanNode::scanner_thread(doris::OlapScanner*) at /data/doris-1.x/be/src/exec/olap_scan_node.cpp:1751
11# doris::PriorityWorkStealingThreadPool::work_thread(int) at /data/doris-1.x/be/src/util/priority_work_stealing_thread_pool.hpp:136
12# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
13# start_thread in /lib64/libpthread.so.0
14# __clone in /lib64/libc.so.6

and I gdb this core

(gdb) f 8
#8 0x000055a595362f65 in doris::RowBlock::init (this=0x7f0bdc238c20, block_info=...) at /data/doris-1.x/be/src/olap/row_block.cpp:61
61 /data/doris-1.x/be/src/olap/row_block.cpp: No such file or directory.
(gdb) p _schema._M_ptr._cols._M_impl._M_finish-_schema._M_ptr._cols._M_impl._M_start
$37 = 53 (表的列数)
(gdb) p _field_offset_in_memory._M_impl._M_finish - _field_offset_in_memory._M_impl._M_start
$38 = 53 (记录遍历每个列需要申请的内存数,一直递增,这里一共进行了5次跳变,每次 2G,最后变成了10G)
(gdb) p (_field_offset_in_memory._M_impl._M_start)@53
$39 = {0, 2, 6, 15, 18, 27, 44, 46, 65582, 131118, 131120, 196656, 196665, 196674, 196691, 196708, 196725, 196742, 196744, 262280, 2147745928, 4295229576, 4295229578, 4295229595, 4295229612, 4295229617, 4295229634, 4295229643,
6442713291, 6442713293, 8590196941, 8590196958, 8590196975, 8590196984, 8590196986, 8590197003, 8590262539, 8590328075, 10737811723, 10737877259, 10737942795, 10738008331, 10738008333, 10738073869, 10738073874, 10738073879,
10738073884, 10738073886, 10738073903, 10738073912, 10738073929, 18446744073709551615, 18446744073709551615} ------这里中间跳变了5次2G大小的长度
(gdb) p _mem_buf_bytes
$40 = 10995787712512 (_compute_layout 函数的计算结果,有下面的 _info.row_num
_mem_row_bytes 得到,结果是10T,这里直接导致了 _mem_buf = new char[_mem_buf_bytes] core掉 )
(gdb) p _info.row_num
$41 = 1024
(gdb) p _mem_row_bytes
$42 = 10738073938

so this pr support jsonb as a variable length for avoid this core

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@ryanzryu
Copy link
Contributor Author

ryanzryu commented Jan 1, 2024

after this pr,we will get jsonb not supoort error msg instead of core when export table with jsonb filed

@lide-reed lide-reed self-requested a review January 2, 2024 10:57
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2024
Copy link
Contributor

github-actions bot commented Jan 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jan 2, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 8f2b829 into apache:branch-1.2-lts Jan 3, 2024
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/1.2.8-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants