LLDB does not flush IO after expression evaluation which makes output of expression overlapped

[ikey4u]

When debugging clang static analyzer, the output of evaluating dump of ProgramState of ExplodedNode seems does not work in LLDB, the issue is weirded that I cannot describe it in simple one or two statements. As a result, I spend a lot of time to create a docker environment that easy you to reproduce the problem, and the problem is described in the last.

Host

The following operations are executed in host.

• Create docker base image

    docker pull centos:7.6.1810
  

• Create docker container

    mkdir -p ~/share/lldb-bug
    docker run --name lldb-bug -d -i -t -v ~/share/lldb-bug:/lldb-bug centos:7.6.1810 bash
  

Docker

The following operations are executed in docker, you can use following command
to get into docker on your host:

docker exec -it lldb-bug bash

• Update yum

    cp -a /etc/yum.repos.d /etc/yum.repos.d.bak
    sed -e "s|^mirrorlist=|#mirrorlist=|g" \
        -e "s|^#baseurl=http://mirror.centos.org/centos/\$releasever|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810|g" \
        -e "s|^#baseurl=http://mirror.centos.org/\$contentdir/\$releasever|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810|g" \
        -i.bak \
        /etc/yum.repos.d/CentOS-*.repo
  
    echo sslverify=false >> /etc/yum.conf
    cat >> /etc/yum.repos.d/CentOS-Base.repo <<EOF
    [sclo]
    name=CentOS-7.6.1810 -Sclo -mirrors.tuna.tsinghua.edu.cn/centos-vault
    failovermethod=priority
    baseurl=http://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810/sclo/\$basearch/rh
    gpgcheck=0
    enabled=1
    gpgkey=http://mirrors.tuna.tsinghua.edu.cn/centos-vault/RPM-GPG-KEY-CentOS-7
    EOF
  
    yum makecache
  

• Install dependencies

    source /opt/rh/devtoolset-8/enable
  
    yum install -y pcre2-devel devtoolset-8 git make git patch openssl-devel zlib-devel readline-devel sqlite-devel bzip2-devel zlib libffi-devel
  
    cd /lldb-bug
    curl https://pyenv.run | bash
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
    echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
    echo 'eval "$(pyenv init -)"' >> ~/.bashrc
    source ~/.bashrc
    PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.5
    pyenv global 3.9.5
  
    cd /lldb-bug
    curl -LO http://prdownloads.sourceforge.net/swig/swig-4.2.1.tar.gz
    tar zxvf swig-4.2.1.tar.gz
    cd swig-4.2.1
    ./configure --prefix=${PWD}/out
    make
    make install
  
    cd /lldb-bug
    curl -LO https://www.thrysoee.dk/editline/libedit-20240517-3.1.tar.gz
    tar zxvf libedit-20240517-3.1.tar.gz
    cd libedit-20240517-3.1/
    ./configure --prefix=${PWD}/out
    make
    make install
  
    cd /lldb-bug
    curl -LO https://github.com/Kitware/CMake/releases/download/v3.20.6/cmake-3.20.6-linux-x86_64.tar.gz
    tar zxvf cmake-3.20.6-linux-x86_64.tar.gz
  
    cd /lldb-bug
    curl -LO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip
    unzip ninja-linux.zip -d ninja
  

• Build lldb

    source /opt/rh/devtoolset-8/enable
  
    cd /lldb-bug
    git clone https://github.com/llvm/llvm-project.git
    cd llvm-project && git checkout tags/llvmorg-18.1.8
  
    # build clang in release mode using gcc 8.3 which will be used to build
    # clang it self later since gcc 8.3 will crash randomly when build clang
    # in debug mode
    cd /lldb-bug
    export PKG_CONFIG_PATH=/lldb-bug/libedit-20240517-3.1/out/lib/pkgconfig:${PKG_CONFIG_PATH}
    export PATH=/lldb-bug/ninja:/lldb-bug/swig-4.2.1/out/bin:/lldb-bug/cmake-3.20.6-linux-x86_64/bin:${PATH}
    cmake -G "Ninja" -B /lldb-bug/llvm-project/build/Release \
        -D CMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -D LLVM_ENABLE_PROJECTS="clang;" \
        -D LLVM_PARALLEL_LINK_JOBS=1 \
        -D LLVM_PARALLEL_COMPILE_JOBS=32 \
        -D CMAKE_BUILD_TYPE=Release \
        -S /lldb-bug/llvm-project/llvm
    time ninja -C /lldb-bug/llvm-project/build/Release
  
    # build clang and lldb in debug mode
    cd /lldb-bug
    export PKG_CONFIG_PATH=/lldb-bug/libedit-20240517-3.1/out/lib/pkgconfig:${PKG_CONFIG_PATH}
    export PATH=/lldb-bug/llvm-project/build/Release/bin/:/lldb-bug/ninja:/lldb-bug/swig-4.2.1/out/bin:/lldb-bug/cmake-3.20.6-linux-x86_64/bin:${PATH}
    cmake -G "Ninja" -B /lldb-bug/llvm-project/build/Debug \
        -D CMAKE_CXX_COMPILER=clang++ \
        -D CMAKE_C_COMPILER=clang \
        -D CMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -D LLVM_ENABLE_PROJECTS="clang;lldb;" \
        -D LLVM_PARALLEL_LINK_JOBS=1 \
        -D LLVM_PARALLEL_COMPILE_JOBS=32 \
        -D CMAKE_BUILD_TYPE=Debug \
        -S /lldb-bug/llvm-project/llvm
    time ninja -C /lldb-bug/llvm-project/build/Debug
  

• The time has finally come to reproduce the bug

  Create a div zero file in home directory:

    // /lldb-bug/divzero.cpp
    int foo(int x) {
        return x/0;
    }
  

  Load it using lldb:

    /lldb-bug/llvm-project/build/Debug/bin/lldb -- /lldb-bug/llvm-project/build/Debug/bin/clang++ --analyze -Xanalyzer -analyzer-checker=core.DivideZero /lldb-bug/divzero.cpp
  

  Disable ASLR in lldb when using lldb in a docker to avoid error
  personality set failed: Operation not permitted:

    settings set target.disable-aslr false
  

  And set a breakpoint in lldb:

    breakpoint set --name ExprEngine::Visit
  

  Then run lldb:

    r
  

  When the breakpoint hits, checke the ExplodeNode state using following
  command:

    * thread #1, name = 'clang++', stop reason = breakpoint 1.1
        frame #0: 0x0000558522a13fc6 clang++`clang::ento::ExprEngine::Visit(this=0x00007ffcf1baf400, S=0x000055852d8bb390, Pred=0x000055852d8df550, DstTop=0x00007ffcf1baeb48) at ExprEngine.cpp:1714:33
       1711
       1712 void ExprEngine::Visit(const Stmt *S, ExplodedNode *Pred,
       1713                        ExplodedNodeSet &DstTop) {
    -> 1714   PrettyStackTraceLoc CrashInfo(getContext().getSourceManager(),
       1715                                 S->getBeginLoc(), "Error evaluating statement");
       1716   ExplodedNodeSet Dst;
       1717   StmtNodeBuilder Bldr(Pred, DstTop, *currBldrCtx);
  
    (lldb)         p Pred->Location->dump()
    "kind": "Statement", "stmt_kind": "DeclRefExpr", "stmt_id": 606, "pointer": "0x55852d8bb390", "pretty": "x", "location": { "line": 2, "column": 12, "file": "/lldb-bug/divzero.cpp" }, "stmt_point_kind": "PreStmtPurgeDeadSymbols"  Evaluated this expression after applying Fix-It(s):
        Pred->Location.dump()
  

  Note that lldb auto fix the command, and let's type the right command, the
  bug appears:

    (lldb) p Pred->Location.dump()
    (lldb)  "Statement", "stmt_kind": "DeclRefExpr", "stmt_id": 606, "pointer": "0x55852d8bb390", "pretty": "x", "location": { "line": 2, "column": 12, "file": "/lldb-bug/divzero.cpp" }, "stmt_point_kind": "PreStmtPurgeDeadSymbols"(lldb)
  

  Note that the last (lldb) which is not right, and the content between the
  two (lldb) will not appear correctly on the terminal which seems is eat
  out by the terminal.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: zhq (ikey4u)

When debugging clang static analyzer, the output of evaluating dump of ProgramState of ExplodedNode seems does not work in LLDB, the issue is weirded that I cannot describe it in simple one or two statements. As a result, I spend a lot of time to create a docker environment that easy you to reproduce the problem, and the problem is described in the last.

Host

The following operations are executed in host.

• Create docker base image

    docker pull centos:7.6.1810
  

• Create docker container

    mkdir -p ~/share/lldb-bug
    docker run --name lldb-bug -d -i -t -v ~/share/lldb-bug:/lldb-bug centos:7.6.1810 bash
  

Docker

The following operations are executed in docker, you can use following command
to get into docker on your host:

docker exec -it lldb-bug bash

• Update yum

    cp -a /etc/yum.repos.d /etc/yum.repos.d.bak
    sed -e "s|^mirrorlist=|#mirrorlist=|g" \
        -e "s|^#baseurl=http://mirror.centos.org/centos/\$releasever|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810|g" \
        -e "s|^#baseurl=http://mirror.centos.org/\$contentdir/\$releasever|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810|g" \
        -i.bak \
        /etc/yum.repos.d/CentOS-*.repo
  
    echo sslverify=false >> /etc/yum.conf
    cat >> /etc/yum.repos.d/CentOS-Base.repo <<EOF
    [sclo]
    name=CentOS-7.6.1810 -Sclo -mirrors.tuna.tsinghua.edu.cn/centos-vault
    failovermethod=priority
    baseurl=http://mirrors.tuna.tsinghua.edu.cn/centos-vault/7.6.1810/sclo/\$basearch/rh
    gpgcheck=0
    enabled=1
    gpgkey=http://mirrors.tuna.tsinghua.edu.cn/centos-vault/RPM-GPG-KEY-CentOS-7
    EOF
  
    yum makecache
  

• Install dependencies

    source /opt/rh/devtoolset-8/enable
  
    yum install -y pcre2-devel devtoolset-8 git make git patch openssl-devel zlib-devel readline-devel sqlite-devel bzip2-devel zlib libffi-devel
  
    cd /lldb-bug
    curl https://pyenv.run | bash
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
    echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
    echo 'eval "$(pyenv init -)"' >> ~/.bashrc
    source ~/.bashrc
    PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.5
    pyenv global 3.9.5
  
    cd /lldb-bug
    curl -LO http://prdownloads.sourceforge.net/swig/swig-4.2.1.tar.gz
    tar zxvf swig-4.2.1.tar.gz
    cd swig-4.2.1
    ./configure --prefix=${PWD}/out
    make
    make install
  
    cd /lldb-bug
    curl -LO https://www.thrysoee.dk/editline/libedit-20240517-3.1.tar.gz
    tar zxvf libedit-20240517-3.1.tar.gz
    cd libedit-20240517-3.1/
    ./configure --prefix=${PWD}/out
    make
    make install
  
    cd /lldb-bug
    curl -LO https://github.com/Kitware/CMake/releases/download/v3.20.6/cmake-3.20.6-linux-x86_64.tar.gz
    tar zxvf cmake-3.20.6-linux-x86_64.tar.gz
  
    cd /lldb-bug
    curl -LO https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip
    unzip ninja-linux.zip -d ninja
  

• Build lldb

    source /opt/rh/devtoolset-8/enable
  
    cd /lldb-bug
    git clone https://github.com/llvm/llvm-project.git
    cd llvm-project && git checkout tags/llvmorg-18.1.8
  
    # build clang in release mode using gcc 8.3 which will be used to build
    # clang it self later since gcc 8.3 will crash randomly when build clang
    # in debug mode
    cd /lldb-bug
    export PKG_CONFIG_PATH=/lldb-bug/libedit-20240517-3.1/out/lib/pkgconfig:${PKG_CONFIG_PATH}
    export PATH=/lldb-bug/ninja:/lldb-bug/swig-4.2.1/out/bin:/lldb-bug/cmake-3.20.6-linux-x86_64/bin:${PATH}
    cmake -G "Ninja" -B /lldb-bug/llvm-project/build/Release \
        -D CMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -D LLVM_ENABLE_PROJECTS="clang;" \
        -D LLVM_PARALLEL_LINK_JOBS=1 \
        -D LLVM_PARALLEL_COMPILE_JOBS=32 \
        -D CMAKE_BUILD_TYPE=Release \
        -S /lldb-bug/llvm-project/llvm
    time ninja -C /lldb-bug/llvm-project/build/Release
  
    # build clang and lldb in debug mode
    cd /lldb-bug
    export PKG_CONFIG_PATH=/lldb-bug/libedit-20240517-3.1/out/lib/pkgconfig:${PKG_CONFIG_PATH}
    export PATH=/lldb-bug/llvm-project/build/Release/bin/:/lldb-bug/ninja:/lldb-bug/swig-4.2.1/out/bin:/lldb-bug/cmake-3.20.6-linux-x86_64/bin:${PATH}
    cmake -G "Ninja" -B /lldb-bug/llvm-project/build/Debug \
        -D CMAKE_CXX_COMPILER=clang++ \
        -D CMAKE_C_COMPILER=clang \
        -D CMAKE_EXPORT_COMPILE_COMMANDS=1 \
        -D LLVM_ENABLE_PROJECTS="clang;lldb;" \
        -D LLVM_PARALLEL_LINK_JOBS=1 \
        -D LLVM_PARALLEL_COMPILE_JOBS=32 \
        -D CMAKE_BUILD_TYPE=Debug \
        -S /lldb-bug/llvm-project/llvm
    time ninja -C /lldb-bug/llvm-project/build/Debug
  

• The time has finally come to reproduce the bug

  Create a div zero file in home directory:

    // /lldb-bug/divzero.cpp
    int foo(int x) {
        return x/0;
    }
  

  Load it using lldb:

    /lldb-bug/llvm-project/build/Debug/bin/lldb -- /lldb-bug/llvm-project/build/Debug/bin/clang++ --analyze -Xanalyzer -analyzer-checker=core.DivideZero /lldb-bug/divzero.cpp
  

  Disable ASLR in lldb when using lldb in a docker to avoid error
  personality set failed: Operation not permitted:

    settings set target.disable-aslr false
  

  And set a breakpoint in lldb:

    breakpoint set --name ExprEngine::Visit
  

  Then run lldb:

    r
  

  When the breakpoint hits, checke the ExplodeNode state using following
  command:

    * thread #<!-- -->1, name = 'clang++', stop reason = breakpoint 1.1
        frame #<!-- -->0: 0x0000558522a13fc6 clang++`clang::ento::ExprEngine::Visit(this=0x00007ffcf1baf400, S=0x000055852d8bb390, Pred=0x000055852d8df550, DstTop=0x00007ffcf1baeb48) at ExprEngine.cpp:1714:33
       1711
       1712 void ExprEngine::Visit(const Stmt *S, ExplodedNode *Pred,
       1713                        ExplodedNodeSet &amp;DstTop) {
    -&gt; 1714   PrettyStackTraceLoc CrashInfo(getContext().getSourceManager(),
       1715                                 S-&gt;getBeginLoc(), "Error evaluating statement");
       1716   ExplodedNodeSet Dst;
       1717   StmtNodeBuilder Bldr(Pred, DstTop, *currBldrCtx);
  
    (lldb)         p Pred-&gt;Location-&gt;dump()
    "kind": "Statement", "stmt_kind": "DeclRefExpr", "stmt_id": 606, "pointer": "0x55852d8bb390", "pretty": "x", "location": { "line": 2, "column": 12, "file": "/lldb-bug/divzero.cpp" }, "stmt_point_kind": "PreStmtPurgeDeadSymbols"  Evaluated this expression after applying Fix-It(s):
        Pred-&gt;Location.dump()
  

  Note that lldb auto fix the command, and let's type the right command, the
  bug appears:

    (lldb) p Pred->Location.dump()
    (lldb)  "Statement", "stmt_kind": "DeclRefExpr", "stmt_id": 606, "pointer": "0x55852d8bb390", "pretty": "x", "location": { "line": 2, "column": 12, "file": "/lldb-bug/divzero.cpp" }, "stmt_point_kind": "PreStmtPurgeDeadSymbols"(lldb)
  

  Note that the last (lldb) which is not right, and the content between the
  two (lldb) will not appear correctly on the terminal which seems is eat
  out by the terminal.