Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lldb-mi freezes on MacOS when breaking near uninitialised variables #7240

Open
andrewfindon opened this issue Mar 26, 2021 · 30 comments
Open

Comments

@andrewfindon
Copy link

Type: Debugger

Describe the bug

  • OS and Version: MacOS 11.2.3 (Big Sur)
  • VS Code Version: 1.5.4.3
  • C/C++ Extension Version: 1.2.2
  • I have recreated on both Intel and Apple Silicon (Rosetta) MacBook Pros.

During debugging, if stopping on (or stepping in to) a frame containing certain types of uninitialised variables, lldb-mi gets stuck at 100% cpu and starts to consume multiple GBs of memory.

I've encountered this a a few times in different code, I've condensed the latest example in to a tiny snippet (below), using a std::initialiser_list<int>.

I couldn't reproduce using the lldb command line. Xcode is fine too, but I notice it uses lldb-rpc-server to communicate with debugserver instead of lldb-mi.

I also tried using the CodeLLDB extension (https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) which doesn't suffer with the same problem (and doesn't use lldb-mi) - but interestingly, it does try to inspect the values of the initializer_list but times out after a short time.

If debugging is stopped while this is going on, the lldb-mi process continue to spin with high cpu usage in the background.

This may be related to the following:

The issue may not be directly related to cpptools (could be an lldb-mi bug?), but the extension suffers as a result - in that case, maybe there is something between the interaction with lldb-mi and cpptools that could avoid triggering the bug.

To Reproduce

(I've also prepared a tiny repo for easily reproducing the problem here: https://github.com/andrewfindon/lldbmi-freeze)

To trigger the problem, step in to the freeze_up() function in the example below. The lldb-mi process should spike in cpu/memory usage and vscode will appear to freeze waiting for local variables to resolve.

main.cpp

#include <initializer_list>

void freeze_up()
{
	auto bad_var = {1, 2, 3};		// A breakpoint here will trigger the issue.
}						// A breakpoint here is fine (bad_var is initialised).

int main() {
	freeze_up();
	
	return 0;
}

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/freeze",
            "args": [],
            "stopAtEntry": false,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "lldb",
            "preLaunchTask": "clang++ build",

            "logging": {
                "engineLogging": true,
                "trace": true,
                "traceResponse": true
            }
        }
    ]
}

tasks.json

{
    "version": "2.0.0",
    "tasks": [
        {
            "type": "shell",
            "label": "clang++ build",
            "command": "/usr/bin/clang++",
            "args": [
                "main.cpp",
                "-arch",
                "x86_64",
                "-std=c++17",
                "-Wall",
                "-g",
                "-o",
                "freeze",
            ],
            "problemMatcher": [
                "$gcc"
            ],
            "group": "build",
        }
    ]
}

The tail of my engine logging output at the point where the debugger has frozen on a breakpoint, looks like:

--> E (stopped): {"type":"event","event":"stopped","body":{"reason":"step","threadId":1,"allThreadsStopped":true,"source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":6,"column":1},"seq":641}
<--   C (threads-12): {"command":"threads","type":"request","seq":12}
--> R (threads-12): {"type":"response","request_seq":12,"success":true,"command":"threads","body":{"threads":[{"id":1,"name":"Thread #1"}]},"seq":644}
<--   C (stackTrace-13): {"command":"stackTrace","arguments":{"threadId":1,"startFrame":0,"levels":20},"type":"request","seq":13}
--> R (stackTrace-13): {"type":"response","request_seq":13,"success":true,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"freeze!freeze_up()","source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":6,"column":1,"moduleId":1},{"id":1001,"name":"freeze!main","source":{"name":"main.cpp","path":"/Users/andrewfindon/Documents/lldbmi-crash/main.cpp","sources":[],"checksums":[]},"line":12,"column":1,"moduleId":1},{"id":1002,"name":"libdyld.dylib!start","line":0,"column":0,"moduleId":12},{"id":1003,"name":"libdyld.dylib!start","line":0,"column":0,"moduleId":12}],"totalFrames":4},"seq":647}
<--   C (scopes-14): {"command":"scopes","arguments":{"frameId":1000},"type":"request","seq":14}
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33618) <-1018-stack-list-variables 0 --thread 1 --frame 0\n"},"seq":650}
1: (33618) <-1018-stack-list-variables 0 --thread 1 --frame 0
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33631) ->1018^done,variables=[{name=\"bad_var\"}]\n"},"seq":652}
1: (33631) ->1018^done,variables=[{name="bad_var"}]
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33631) ->(gdb)\n"},"seq":654}
1: (33631) ->(gdb)
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33632) 1018: elapsed time 13\n"},"seq":656}
1: (33632) 1018: elapsed time 13
--> E (output): {"type":"event","event":"output","body":{"category":"console","output":"1: (33657) <-1019-var-create - - \"bad_var\" --thread 1 --frame 0\n"},"seq":658}
1: (33657) <-1019-var-create - - "bad_var" --thread 1 --frame 0

Here's a trace of lldb-mi in Instruments - it's clear on the timeline when I stepped in to the freeze_up function:
image

Example debugging session:

image

@WardenGnaw
Copy link
Member

lldb-mi seems to already have some issues being slow debugging as per #1899 (comment).

For Apple Silicon, it will be slow since both the Debug Adapter (MIEngine) and lldb-mi will both be running in emulation.

To fix the latter, it is dependent on #7035

@andrewfindon
Copy link
Author

Please note, this issue applies equally to Intel machines (I mention Apple Silicon only to be thorough).

Also, I have left It in this high cpu, high memory state for "some time" (multiple minutes) - it doesn't seem to recover (so less about it being slow, and more about it getting stuck).

cpptools no longer responds, except for cancelling debugging (but this leaves lldb-mi running in the background eating cpu and rapidly draining battery).

This ~6 line sample is enough to break the debugging experience entirely, but personally I've encountered these symptoms a couple of times in actual code, although then I think it was an uninitialised std::vector that was the trigger.

Anecdotally it seems like the debugger is reaching in to the uninitiated memory forever to resolve values for the local variables view.

@ni4
Copy link

ni4 commented Jun 19, 2021

+1 for this issue, macOS Catalina 10.15.7, Intel CPU.
However:

  • all variables in the function I'm trying to stop in seems to be initialized
  • when put breakpoint in called function, I'm able to navigate, and when returning to the original function see this:

Screenshot 2021-06-19 at 17 35 05

  • at the moment of stop all of those variables are initialized
  • if it helps, I'm building with -fsanitize=address,undefined. Removing this doesn't change the behaviour.

@Wdong04
Copy link

Wdong04 commented Aug 30, 2021

Same problem here. Would like to know how to address it.

@shiretu
Copy link

shiretu commented Dec 20, 2021

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

@Ayasra
Copy link

Ayasra commented Mar 19, 2022

Same problem here

@PenceGL
Copy link

PenceGL commented Apr 7, 2022

I just experienced this a few minutes ago. I am on MacOS Monterey, v12.3.1, Intel CPU.
I was able to hit breakpoints, but then realized my debugger controls (Continue, Step Over, Step In) were completely unresponsive. I was able to press stop to quit debugging as @andrewfindon mentioned. But then my laptop tried to enter orbit... fans on full blast, CPU utilization pegged at 100%. Activity Monitor showed several instances (I guess because I had hit stop and rerun the debugger a few times) of lldb-mi using lots of memory and 100% CPU. Stopping them manually through Activity Monitor resolved the issue and the fan/utilization went back to normal levels.

A few quick Google searches led me here. I'm glad to see that it's an issue that's still getting attention.

@AgraVator
Copy link

AgraVator commented Apr 24, 2022

This helped me to avoid the problem for now.
Looking forward for a permanent fix though.

@lp35
Copy link

lp35 commented Aug 10, 2022

Hey guys, do you have a solution for Linux users?
Cheers!

@vlebourl
Copy link

Hey, any news on this?

@shiretu
Copy link

shiretu commented Oct 5, 2022

Hey, any news on this?

I do not think this is a VSCode issue. This is truly a problem with the lldb. See my patch above. lldb is out-of-bounds-ing on walking uninitialized variables with garbage values on things like internal counters and sizes. My quick yet brutal fix is to limit these things to 64.

Unfortunately, VSCode tries to render all local variables in the current scope. And when it reaches one of those not-yet-initialized vars, it behaves like that.

@jpcofr
Copy link

jpcofr commented Mar 12, 2023

As a workaround, I installed CodeLLDB. It fixed the issue. Still, vscode's own debugger got lldb to use all the memory in my machine, which in turn made it unresponsive and I had to restart it.

@garfieldnate
Copy link

I ran into this today. Since @shiretu says this is an lldb-mi issue, I guess the place to track it is here?

@heafox
Copy link

heafox commented Sep 3, 2023

CodeLLDB is fine

@H-G-Hristov
Copy link

H-G-Hristov commented Sep 27, 2023

CodeLLDB is fine

I have a bad feeling that CodeLLDB doesn't work now too.
vadimcn/codelldb#987
vadimcn/codelldb#998

@H-G-Hristov
Copy link

@bobbrow @WardenGnaw Is there any change of this getting fixed? It seems this issue is specific to Apple Silicon. What about Windows on arm?

@mlangiu
Copy link

mlangiu commented Sep 28, 2023

It seems this issue is specific to Apple Silicon.

Nope this happens also on older Intel macs

@ni4
Copy link

ni4 commented Sep 28, 2023

+1, I'm on Intel Mac and have the same issue. It hurts so bad during debug that in a new code I always initialize all local vars :)

@kkHAIKE
Copy link

kkHAIKE commented Oct 19, 2023

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

Is it working normally?

@sunshaoce
Copy link

sunshaoce commented Nov 29, 2023

I have recompiled my lldb-mi with this patch:

diff --git a/src/MICmdCmdVar.cpp b/src/MICmdCmdVar.cpp
index 6712a48..e9ceab7 100644
--- a/src/MICmdCmdVar.cpp
+++ b/src/MICmdCmdVar.cpp
@@ -304,7 +304,8 @@ void CMICmdCmdVarCreate::CompleteSBValue(lldb::SBValue &vrwValue) {
   // And update its children
   lldb::SBType valueType = vrwValue.GetType();
   if (!valueType.IsPointerType() && !valueType.IsReferenceType()) {
-    const MIuint nChildren = vrwValue.GetNumChildren();
+    const auto temp = vrwValue.GetNumChildren();
+    const MIuint nChildren = temp > 64 ? 64 : temp;
     for (MIuint i = 0; i < nChildren; ++i) {
       lldb::SBValue member = vrwValue.GetChildAtIndex(i);
       if (member.IsValid())

@shiretu Would you consider improving it and then submit it to lldb-mi?

@robUx4
Copy link

robUx4 commented Feb 17, 2024

This is still happening with cpptools v1.18.5. I disabled the "Variables" view of the debugger so I can get some work done. Mouse hover on a variable still shows the variables when needed.

@wbf22
Copy link

wbf22 commented May 16, 2024

I actually had it freeze again after removing the file @jamesjuett

cd ~/.vscode/extensions/vadimcn.vscode-lldb-1.10.0/lldb/bin
rm debugserver

But hiding the variables section in the debug window does work (rerunning after hiding)

@ddavlet
Copy link

ddavlet commented Jul 23, 2024

  • OS and Version: MacOS 14.5 (Sonoma)
  • VS Code Version: 1.91.1
  • C/C++ Extension Version: 1.21.2
  • Mac Air M2 chip

This issue still persists

@clayborg
Copy link

clayborg commented Aug 6, 2024

This is an issue with synthetic child providers. For STL classes you often don't want to see the actual children of a std::list<T> or std::vector<T>, you want to see the children as an array. When we have uninitialized variables, the synthetic child provider has to be very careful to check for invalid values and not do too much work when calculating the number of children. So this can most likely be fixed by checking out the typename of the type that is causing this delay, and then finding out which synthetic child provider it is using and adding a fix to this formatter if upstream LLDB doesn't already have it fixed. To see which synthetic formatters are enabled, type this command at the prompt:

(lldb) type synthetic list 

And then with the variable that causes the problem run:

(lldb) frame variable --raw <name>

The --raw option will disable the synthetic child plug-in. These synthetic plugins can either be natively compiled into lldb, or they can be python modules that add the support. If they are python then it is usually easy to fix right away. If the synthetic child provider is compiled into LLDB, then a new version of LLDB will need to be distributed. This formatter does seem to have issues. I just compiled your example and ran this in the command line:

Process 10801 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
    frame #0: 0x0000000100003f20 a.out`freeze_up() at main.cpp:5:17
   2   	
   3   	void freeze_up()
   4   	{
-> 5   		auto bad_var = {1, 2, 3};		// A breakpoint here will trigger the issue.
   6   	}						// A breakpoint here is fine (bad_var is initialised).
   7   	
   8   	int main() {
(lldb) fr var
(std::initializer_list<int>) bad_var = {
  [0] = 72715400
  [1] = 2
  [2] = 0
  [3] = 0
  [4] = 0
  [5] = 0
  [6] = 0
  [7] = 0
  [8] = 0
  [9] = 0
  [10] = 0
  [11] = 0
  [12] = 16
  [13] = 0
  [14] = 5
  [15] = 0
  [16] = 1876948112
  [17] = 1
  [18] = 49200
  [19] = 1
  [20] = 0
  [21] = 0
  [22] = 0
  [23] = 0
  [24] = 0
  [25] = 0
  [26] = 0
  [27] = 0
  [28] = 671744
  [29] = 1
  [30] = 16384
  [31] = 0
  [32] = 688128
  [33] = 1
  [34] = 327680
  [35] = 0
  [36] = 0
  [37] = 0
  [38] = 0
  [39] = 0
  [40] = 0
  [41] = 0
  [42] = 0
  [43] = 0
  [44] = 0
  [45] = 0
  [46] = 0
  [47] = 0
  [48] = 0
  [49] = 0
  [50] = 0
  [51] = 0
  [52] = 0
  [53] = 0
  [54] = 0
  [55] = 0
  [56] = 0
  [57] = 0
  [58] = 0
  [59] = 0
  [60] = 0
  [61] = 0
  [62] = 0
  [63] = 0
  [64] = 0
  [65] = 0
  [66] = 0
  [67] = 0
  [68] = 0
  [69] = 0
  [70] = 0
  [71] = 0
  [72] = 0
  [73] = 0
  [74] = 0
  [75] = 0
  [76] = 0
  [77] = 0
  [78] = 0
  [79] = 0
  [80] = 0
  [81] = 0
  [82] = 0
  [83] = 0
  [84] = 0
  [85] = 0
  [86] = 0
  [87] = 0
  [88] = 1876948240
  [89] = 1
  [90] = -1753497600
  [91] = 1
  [92] = 393232384
  [93] = 0
  [94] = -1754251264
  [95] = 1
  [96] = 49152
  [97] = 1
  [98] = 1876948664
  [99] = 1
  [100] = 0
  [101] = 0
  [102] = 0
  [103] = 1325694976
  [104] = 0
  [105] = 0
  [106] = 0
  [107] = 1
  [108] = 1
  [109] = 0
  [110] = 1876949096
  [111] = 1
  [112] = 0
  [113] = 0
  [114] = 1876949117
  [115] = 1
  [116] = 1876949151
  [117] = 1
  [118] = 1876949171
  [119] = 1
  [120] = 1876949186
  [121] = 1
  [122] = 1876949217
  [123] = 1
  [124] = 1876949231
  [125] = 1
  [126] = 1876949250
  [127] = 1
  [128] = 1876949272
  [129] = 1
  [130] = 1876949286
  [131] = 1
  [132] = 1876949303
  [133] = 1
  [134] = 1876949360
  [135] = 1
  [136] = 1876949392
  [137] = 1
  [138] = 1876949432
  [139] = 1
  [140] = 1876949451
  [141] = 1
  [142] = 1876949468
  [143] = 1
  [144] = 1876949480
  [145] = 1
  [146] = 1876949491
  [147] = 1
  [148] = 1876949525
  [149] = 1
  [150] = 1876949553
  [151] = 1
  [152] = 1876950259
  [153] = 1
  [154] = 1876950270
  [155] = 1
  [156] = 1876950278
  [157] = 1
  [158] = 1876950286
  [159] = 1
  [160] = 1876950323
  [161] = 1
  [162] = 1876950358
  [163] = 1
  [164] = 1876950379
  [165] = 1
  [166] = 1876950432
  [167] = 1
  [168] = 1876950462
  [169] = 1
  [170] = 1876950489
  [171] = 1
  [172] = 1876950555
  [173] = 1
  [174] = 1876950580
  [175] = 1
  [176] = 0
  [177] = 0
  [178] = 1876949056
  [179] = 1
  [180] = 1876950624
  [181] = 1
  [182] = 1876950640
  [183] = 1
  [184] = 1876950671
  [185] = 1
  [186] = 1876950724
  [187] = 1
  [188] = 1876950753
  [189] = 1
  [190] = 1876950807
  [191] = 1
  [192] = 1876950846
  [193] = 1
  [194] = 1876950887
  [195] = 1
  [196] = 1876950946
  [197] = 1
  [198] = 1876951007
  [199] = 1
  [200] = 1876951021
  [201] = 1
  [202] = 0
  [203] = 0
  [204] = 1667594341
  [205] = 1650553973
  [206] = 1885300076
  [207] = 1030255713
  [208] = 1886221359
  [209] = 1768843567
  [210] = 1768697204
  [211] = 1630499955
  [212] = 1953853230
  [213] = 0
  [214] = 1886221359
  [215] = 1768843567
  [216] = 1768697204
  [217] = 1630499955
  [218] = 1953853230
  [219] = 1297041408
  [220] = 1163018821
  [221] = 1163026263
  [222] = 1230196560
  [223] = 1498566484
  [224] = 1886334781
  [225] = 1869098868
  [226] = 1919051117
  [227] = 1409316709
  [228] = 1028477509
  [229] = 1919251576
  [230] = 892480877
  [231] = 1819239222
  [232] = 1392538223
  [233] = 1280066888
  [234] = 1768042301
  [235] = 1937387374
  [236] = 1398407272
  [237] = 1429159240
  [238] = 1936876915
  [239] = 1818453807
  [240] = 1869904225
  [241] = 1865297774
  [242] = 2037198184
  [243] = 1752398381
  [244] = 1129338880
  [245] = 1095517791
  [246] = 809325383
  [247] = 1476407416
  [248] = 1398752080
  [249] = 1230393925
  [250] = 1314866499
  [251] = 1027951937
  [252] = 1029636144
  [253] = 1920169263
  [254] = 1668246575
  [255] = 1647275105
  ...
}
*** Some of the displayed variables have more members than the debugger will show by default. To show all of them, you can either use the --show-all-children option to frame variable or raise the limit by changing the target.max-children-count setting.

As you can see we stopped asking for children when we reached the target.max-children-count setting. The MI plug-in will want to do the same thing. We can see the raw variable values here:

(lldb) frame variable --show-types --raw
(std::initializer_list<int>) bad_var = {
  (const int *) __begin_ = 0x000000016fdff510
  (size_t) __size_ = 4299233552
}

So we think there are 4299233552 children to show. This value is way to easy to mess up because there isn't much we can do to know when the values are bad. We can check that __begin_ is aligned on a int boundary, but in this case, it works.

@clayborg
Copy link

clayborg commented Aug 6, 2024

In this case the provider is built in:

(lldb) type synthetic info bad_var
synthetic applied to (std::initializer_list<int>) bad_var is:  libc++ std::initializer_list synthetic children

@clayborg
Copy link

clayborg commented Aug 6, 2024

So the question is: is the MI plug-in asking and producing that many children for this variable and that is what is taking the time and blowing up memory? If so, it should stop creating children for a variable when it hits the target.max-children-count lldb setting value.

@sunshaoce
Copy link

lldb-mi has attempted to fix this problem in lldb-tools/lldb-mi#115, but it seems that cpptools is not using the latest version of lldb-mi.

@KevinEady
Copy link

Thanks for the info @sunshaoce . I was able to work around the issue by building lldb-mi from scratch (from commit a6c8c66), and replacing the executable inside ~/.vscode/extensions/ms-vscode.cpptools-1.21.6-darwin-arm64/debugAdapters/lldb-mi/bin/lldb-mi

Obviously any update to the extension would cause the file to be overwritten. Not sure how to update it inside the cpptools extension directly. This whole "downloading" section seems outdated https://github.com/microsoft/vscode-cpptools/blob/main/CONTRIBUTING.md#about-the-code -- there is no packageManager.ts

@jjaine
Copy link

jjaine commented Oct 9, 2024

OS and Version: MacOS 14.6.1 (Sonoma)
VS Code Version: 1.93.1
C/C++ Extension Version: 1.21.6
LLDB Version: lldb-1600.0.36.3
Apple M2 Max Chip

Similar issue, but for me the debugger crashes

ERROR: LLDB exited unexpectedly with exit code 139 (0x8B). Debugging will now abort.

Tried building lldb-mi with the instructions from lldb-tools/lldb-mi#100 using the commit a6c8c66, which made no difference. Also tried the fix from the lldb-tools/lldb-mi#118 for lldb-mi, also no difference.

The workaround of hiding the Variables view from #7240 (comment) works.

Downgrading Xcode to 15.4 (and thus to lldb-1500.0.404.7) also returns the issue to the current state (which for me was still working in most cases).

@icppWorld
Copy link

But hiding the variables section in the debug window does work

Thank you for this tip. Is a great workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests