Make rpc callback exception more explicit and wind up to show more infos

[unkcpz]

Combine with aiidateam/aiida-core#6671, it improves the debug workflow of daemon errors with RMQ.

[agoscinski]

Looks good to me moslty. I am not sure if the formatting of the error message as I find these raise ... from exc plus traceback hard to read because of the repetition but I also remember that I had to do this in some code because the original traceback wasn't always displayed. Was this here also the case or why do you print the traceback another time?

Here an example of the repetition I mean:

Traceback (most recent call last):
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 10, in <module>
    callback()
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 6, in foo
    faa()
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 2, in faa
    raise ValueError("Something")
ValueError: Something

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 27, in <module>
    raise RuntimeError(
RuntimeError: Error invoking callback 'foo' at /Users/alexgo/code/aiida-core/format_errmsg.py:5.
Exception: ValueError: Something

Full Traceback:
Traceback (most recent call last):
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 10, in <module>
    callback()
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 6, in foo
    faa()
  File "/Users/alexgo/code/aiida-core/format_errmsg.py", line 2, in faa
    raise ValueError("Something")
ValueError: Something

For the code

def faa():
    raise ValueError("Something")


def foo():
    faa()

callback = foo
try:
    callback()
except Exception as exc:
    import traceback
    import inspect

    # Get traceback as a string
    tb_str = ''.join(traceback.format_exception(type(exc), exc, exc.__traceback__))

    try:
        source_file = inspect.getfile(callback)
        # getsourcelines returns a tuple (list_of_source_lines, starting_line_number)
        _, start_line = inspect.getsourcelines(callback)
        callback_location = f'{source_file}:{start_line}'
    except Exception:
        callback_location = '<unknown location>'
    
    # Include the callback name, file/line info, and the full traceback in the message
    raise RuntimeError(
        f"Error invoking callback '{callback.__name__}' at {callback_location}.\n"
        f'Exception: {type(exc).__name__}: {exc}\n\n'
        f'Full Traceback:\n{tb_str}'
    ) from exc

[agoscinski]

tests pass now and rebased

[unkcpz]

Was this here also the case or why do you print the traceback another time?

I can not remind myself when this traceback was swallowed, but yes it is a problem that the traceback may printed twice if the exception is came from the same thread where the runner runs. I think it is fine to get the merged since it helps for other PR where exceptions swallowed and make it hard to debug.