[fix](export)fix core when export jsonb field #29365
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
export table whit jsonb field will be core,get stack like this:
*** Query id: 0-0 ***
*** Aborted at 1702436737 (unix time) try "date -d @1702436737" if you are using GNU date ***
*** Current BE git commitID: 0b1372c ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 449814 (TID 0x7fee0bb14700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /data/doris-1.x/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /usr/local/jdk/jre/lib/amd64/server/libjvm.so
4# 0x00007FEE4AB51400 in /lib64/libc.so.6
5# memcpy at /data/doris-1.x/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# doris::FieldTypeTraits<(doris::FieldType)13>::direct_copy(void*, void const*) at /data/doris-1.x/be/src/olap/types.h:1439
7# void doris::direct_copy_row<doris::RowCursor, doris::RowCursor>(doris::RowCursor*, doris::RowCursor const&) at /data/doris-1.x/be/src/olap/row.h:120
8# doris::TupleReader::_unique_key_next_row(doris::RowCursor*, doris::MemPool*, doris::ObjectPool*, bool*) at /data/doris-1.x/be/src/olap/tuple_reader.cpp:190
9# doris::OlapScanner::get_batch(doris::RuntimeState*, doris::RowBatch*, bool*) at /data/doris-1.x/be/src/exec/olap_scanner.cpp:372
10# doris::OlapScanNode::scanner_thread(doris::OlapScanner*) at /data/doris-1.x/be/src/exec/olap_scan_node.cpp:1751
11# doris::PriorityWorkStealingThreadPool::work_thread(int) at /data/doris-1.x/be/src/util/priority_work_stealing_thread_pool.hpp:136
12# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
13# start_thread in /lib64/libpthread.so.0
14# __clone in /lib64/libc.so.6
and I gdb this core
(gdb) f 8
#8 0x000055a595362f65 in doris::RowBlock::init (this=0x7f0bdc238c20, block_info=...) at /data/doris-1.x/be/src/olap/row_block.cpp:61
61 /data/doris-1.x/be/src/olap/row_block.cpp: No such file or directory.
(gdb) p _schema._M_ptr._cols._M_impl._M_finish-_schema._M_ptr._cols._M_impl._M_start
$37 = 53 (表的列数)
(gdb) p _field_offset_in_memory._M_impl._M_finish - _field_offset_in_memory._M_impl._M_start
$38 = 53 (记录遍历每个列需要申请的内存数,一直递增,这里一共进行了5次跳变,每次 2G,最后变成了10G)
(gdb) p (_field_offset_in_memory._M_impl._M_start)@53
$39 = {0, 2, 6, 15, 18, 27, 44, 46, 65582, 131118, 131120, 196656, 196665, 196674, 196691, 196708, 196725, 196742, 196744, 262280, 2147745928, 4295229576, 4295229578, 4295229595, 4295229612, 4295229617, 4295229634, 4295229643,
6442713291, 6442713293, 8590196941, 8590196958, 8590196975, 8590196984, 8590196986, 8590197003, 8590262539, 8590328075, 10737811723, 10737877259, 10737942795, 10738008331, 10738008333, 10738073869, 10738073874, 10738073879,
10738073884, 10738073886, 10738073903, 10738073912, 10738073929, 18446744073709551615, 18446744073709551615} ------这里中间跳变了5次2G大小的长度
(gdb) p _mem_buf_bytes
$40 = 10995787712512 (_compute_layout 函数的计算结果,有下面的 _info.row_num_mem_row_bytes 得到,结果是10T,这里直接导致了 _mem_buf = new char[_mem_buf_bytes] core掉 )
(gdb) p _info.row_num
$41 = 1024
(gdb) p _mem_row_bytes
$42 = 10738073938
so this pr support jsonb as a variable length for avoid this core
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...