[fix](load) Fix undefined behavior in the JSON reader caused by multiple calls to the get_string method#58107
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 33954 ms |
TPC-DS: Total hot run time: 188440 ms |
ClickBench: Total hot run time: 27.83 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
There was a problem hiding this comment.
map will lead to performance issue, and this is the critical part.maybe only modify _simdjson_write_columns_by_jsonpath
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
…ple calls to the get_string method
|
run buildall |
TPC-H: Total hot run time: 34143 ms |
TPC-DS: Total hot run time: 188237 ms |
ClickBench: Total hot run time: 27.33 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…ple calls to the get_string method (#58107) ### What problem does this PR solve? <img width="2118" height="1222" alt="image" src="https://github.com/user-attachments/assets/aeefb709-f59d-406e-824f-0c9b250bb5af" /> ```text doris_be: /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359: simdjson_result<std::string_view> simdjson::fallback::ondemand::json_iterator::unescape(raw_json_string, bool): Assertion `!parser->string_buffer_overflow(_string_buf_loc)' failed. *** Query id: 24468c5d0b372cf0-3e6c777580238e96 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1763545679 (unix time) try "date -d @1763545679" if you are using GNU date *** *** Current BE git commitID: cbc3d21 *** *** SIGABRT unknown detail explain (@0x3f8001618b8) received by PID 1448120 (TID 1450358 OR 0x7b0fb9f4e700) from PID 1448120; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F183C476D10 in /lib64/libpthread.so.0 2# __GI_raise in /lib64/libc.so.6 3# __GI_abort in /lib64/libc.so.6 4# _nl_load_domain.cold.0 in /lib64/libc.so.6 5# 0x00007F183BCC8E86 in /lib64/libc.so.6 6# simdjson::fallback::ondemand::json_iterator::unescape(simdjson::fallback::ondemand::raw_json_string, bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359 7# simdjson::fallback::ondemand::raw_json_string::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/raw_json_string-inl.h:160 8# simdjson::simdjson_result<simdjson::fallback::ondemand::raw_json_string>::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const in /root/doris/be/output/lib/doris_be 9# simdjson::fallback::ondemand::value_iterator::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value_iterator-inl.h:514 10# simdjson::fallback::ondemand::value::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value-inl.h:48 11# doris::Status doris::vectorized::NewJsonReader::_simdjson_write_data_to_column<false>(simdjson::fallback::ondemand::value&, std::shared_ptr<doris::vectorized::IDataType const> const&, doris::vectorized::IColumn*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<doris::vectorized::DataTypeSerDe>, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1080 12# doris::vectorized::NewJsonReader::_simdjson_write_columns_by_jsonpath(simdjson::fallback::ondemand::object*, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, doris::vectorized::Block&, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1460 13# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:806 14# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:753 15# doris::vectorized::NewJsonReader::_read_json_column(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:485 16# doris::vectorized::NewJsonReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:209 17# doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:480 18# doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:417 19# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:113 20# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:85 21# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 22# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 23# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…ple calls to the get_string method (apache#58107) <img width="2118" height="1222" alt="image" src="https://github.com/user-attachments/assets/aeefb709-f59d-406e-824f-0c9b250bb5af" /> ```text doris_be: /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359: simdjson_result<std::string_view> simdjson::fallback::ondemand::json_iterator::unescape(raw_json_string, bool): Assertion `!parser->string_buffer_overflow(_string_buf_loc)' failed. *** Query id: 24468c5d0b372cf0-3e6c777580238e96 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1763545679 (unix time) try "date -d @1763545679" if you are using GNU date *** *** Current BE git commitID: cbc3d21 *** *** SIGABRT unknown detail explain (@0x3f8001618b8) received by PID 1448120 (TID 1450358 OR 0x7b0fb9f4e700) from PID 1448120; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F183C476D10 in /lib64/libpthread.so.0 2# __GI_raise in /lib64/libc.so.6 3# __GI_abort in /lib64/libc.so.6 4# _nl_load_domain.cold.0 in /lib64/libc.so.6 5# 0x00007F183BCC8E86 in /lib64/libc.so.6 6# simdjson::fallback::ondemand::json_iterator::unescape(simdjson::fallback::ondemand::raw_json_string, bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359 7# simdjson::fallback::ondemand::raw_json_string::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/raw_json_string-inl.h:160 8# simdjson::simdjson_result<simdjson::fallback::ondemand::raw_json_string>::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const in /root/doris/be/output/lib/doris_be 9# simdjson::fallback::ondemand::value_iterator::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value_iterator-inl.h:514 10# simdjson::fallback::ondemand::value::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value-inl.h:48 11# doris::Status doris::vectorized::NewJsonReader::_simdjson_write_data_to_column<false>(simdjson::fallback::ondemand::value&, std::shared_ptr<doris::vectorized::IDataType const> const&, doris::vectorized::IColumn*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<doris::vectorized::DataTypeSerDe>, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1080 12# doris::vectorized::NewJsonReader::_simdjson_write_columns_by_jsonpath(simdjson::fallback::ondemand::object*, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, doris::vectorized::Block&, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1460 13# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:806 14# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:753 15# doris::vectorized::NewJsonReader::_read_json_column(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:485 16# doris::vectorized::NewJsonReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:209 17# doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:480 18# doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:417 19# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:113 20# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:85 21# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 22# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()apache#1}::operator()() const::{lambda()apache#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 23# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()apache#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
…ple calls to the get_string method (apache#58107) ### What problem does this PR solve? <img width="2118" height="1222" alt="image" src="https://github.com/user-attachments/assets/aeefb709-f59d-406e-824f-0c9b250bb5af" /> ```text doris_be: /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359: simdjson_result<std::string_view> simdjson::fallback::ondemand::json_iterator::unescape(raw_json_string, bool): Assertion `!parser->string_buffer_overflow(_string_buf_loc)' failed. *** Query id: 24468c5d0b372cf0-3e6c777580238e96 *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1763545679 (unix time) try "date -d @1763545679" if you are using GNU date *** *** Current BE git commitID: cbc3d21 *** *** SIGABRT unknown detail explain (@0x3f8001618b8) received by PID 1448120 (TID 1450358 OR 0x7b0fb9f4e700) from PID 1448120; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F183C476D10 in /lib64/libpthread.so.0 2# __GI_raise in /lib64/libc.so.6 3# __GI_abort in /lib64/libc.so.6 4# _nl_load_domain.cold.0 in /lib64/libc.so.6 5# 0x00007F183BCC8E86 in /lib64/libc.so.6 6# simdjson::fallback::ondemand::json_iterator::unescape(simdjson::fallback::ondemand::raw_json_string, bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/json_iterator-inl.h:359 7# simdjson::fallback::ondemand::raw_json_string::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/raw_json_string-inl.h:160 8# simdjson::simdjson_result<simdjson::fallback::ondemand::raw_json_string>::unescape(simdjson::fallback::ondemand::json_iterator&, bool) const in /root/doris/be/output/lib/doris_be 9# simdjson::fallback::ondemand::value_iterator::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value_iterator-inl.h:514 10# simdjson::fallback::ondemand::value::get_string(bool) at /root/doris/thirdparty/installed/include/simdjson/generic/ondemand/value-inl.h:48 11# doris::Status doris::vectorized::NewJsonReader::_simdjson_write_data_to_column<false>(simdjson::fallback::ondemand::value&, std::shared_ptr<doris::vectorized::IDataType const> const&, doris::vectorized::IColumn*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<doris::vectorized::DataTypeSerDe>, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1080 12# doris::vectorized::NewJsonReader::_simdjson_write_columns_by_jsonpath(simdjson::fallback::ondemand::object*, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, doris::vectorized::Block&, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:1460 13# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:806 14# doris::vectorized::NewJsonReader::_simdjson_handle_flat_array_complex_json(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:753 15# doris::vectorized::NewJsonReader::_read_json_column(doris::RuntimeState*, doris::vectorized::Block&, std::vector<doris::SlotDescriptor*, std::allocator<doris::SlotDescriptor*> > const&, bool*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:485 16# doris::vectorized::NewJsonReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:209 17# doris::vectorized::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:480 18# doris::vectorized::FileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/file_scanner.cpp:417 19# doris::vectorized::Scanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:113 20# doris::vectorized::Scanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/scan/scanner.cpp:85 21# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:182 22# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()apache#1}::operator()() const::{lambda()apache#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:96 23# doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_0::operator()() const::{lambda()apache#1}::operator()() const at /root/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:95 ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)