SpawnDebugger inherits parent's file descriptors on MacOS/linux

[analog-cbarber]

The SpawnDebugger implementation for MacOS and Linux does a fork/exec but simply inherits
the parent file descriptors, as a result the debugger stdout/stderr will just go to the current process
output descriptors. When you turn on tracing in the debugger, the spawning process will then display
raw XML trace messages.

[garfieldnate]

Thanks for reporting!

Has this been a major issue for you? Or is it more of a paper cut?

The way to fix this for *nix is apparently to re-open stdout/stderr and direct them to /dev/null (or a log file or something) instead:

        freopen("/dev/null", "w", stdout);
        freopen("/dev/null", "w", stderr);

For Windows we have to construct and pass an appropriate STARTUPINFO structure:

    // Open NUL device for stdout and stderr
    HANDLE hNull = CreateFile(
        "NUL",
        GENERIC_WRITE,
        FILE_SHARE_READ | FILE_SHARE_WRITE,
        NULL,
        OPEN_EXISTING,
        FILE_ATTRIBUTE_NORMAL,
        NULL
    );

    if (hNull == INVALID_HANDLE_VALUE) {
        perror("Failed to open NUL device");
        return false;
    }

    // Set up the STARTUPINFO structure
    STARTUPINFO si = { sizeof(STARTUPINFO) };
    si.dwFlags = STARTF_USESTDHANDLES;
    si.hStdOutput = hNull;
    si.hStdError = hNull;


    BOOL ret = CreateProcess(
                   0,
                   const_cast< LPSTR >(commandLine.str().c_str()),      // Command line
                   0,                              // Process handle not inheritable
                   0,                              // Thread handle not inheritable
                   true,                           // Set handle inheritance to true
                   0,                              // No creation flags
                   0,                              // Use parent's environment block
                   0,                              // Use parent's starting directory
                   &si,                            // Pointer to STARTUPINFO structure
                   &m_pDPI->debuggerProcessInformation);   // Pointer to PROCESS_INFORMATION structure

    CloseHandle(hNull);
    CloseHandle(pipes[ChildRead]);
    CloseHandle(pipes[ChildWrite]);
    CloseHandle(pipes[ParentWrite]);

[analog-cbarber]

It was a major annoyance but I just worked around it by spawning writing my own function to spawn it directly from python