Added remote debug facilities (remote pdb server) (#1404)

* Added remote debug facilities

* Added options to the test suite to enable the remote debug server

* Improved documentation on debugging

* Debugging informations -> Advanced documentation

* Back to py 3.5

CONTRIBUTING.md

@@ -19,6 +19,15 @@ There are several ways of doing this:
 
 Should there be agreement that your feature idea adds enough value to offset the maintenance burden, you can go ahead and implement it, including tests and documentation.
 
+## Debugging
+
+UltiSnips embeds some remote debugging facilities in the `UltiSnips.remote_pdb` module.
+When enabled (by setting `let g:UltiSnipsDebugServerEnable=1`), whenever an exception is raised, vim will pause
+and you will be able to connect to the debug server with netcat or telnet.
+By default, the server listens on 'localhost:8080' (it can be changed).
+
+See `:help UltiSnips-Advanced-Debugging` for more informations
+
 ## Testing
 
 UltiSnips has a rigorous test suite and every new feature or bug fix is expected to come with a new test.
@@ -80,6 +89,13 @@ In this shell we can then trigger the test execution:
     ... now inside container
     # ./test_all.py
 
+### Enable the remote debug server
+
+The test suite provides `--remote-pdb*` options equivalent to the config variables to enable the debug server during the test suite.
+Note that some tests may fail because the post-mortem will catch an expected exceptions and that these options are mainly useful for single test case debugging. 
+
+Check `./test_all.py --help` for more informations.
+
 ## Documenting
 
 User documentation goes into [`doc/UltiSnips.txt`](https://github.com/SirVer/ultisnips/blob/00_contributing/doc/UltiSnips.txt).

doc/UltiSnips-advanced.txt

@@ -0,0 +1,92 @@
+*UltiSnips-advanced.txt*    Advanced topics for UltiSnips
+
+
+1. Debugging                                    |UltiSnips-Advanced-Debugging|
+   1.1 Setting breakpoints                      |UltiSnips-Advanced-breakpoints|
+   1.2 Accessing Pdb                            |UltiSnips-Advanced-Pdb|
+
+=============================================================================
+1. Debugging                                   *UltiSnips-Advanced-Debugging*
+                                               *g:UltiSnipsDebugServerEnable*
+UltiSnips comes with a remote debugger disabled        *g:UltiSnipsDebugHost*
+by default. When authoring a complex snippet           *g:UltiSnipsDebugPort*
+with python code, you may want to be able to     *g:UltiSnipsPMDebugBlocking*
+set breakpoints to inspect variables.
+It is also useful when debugging UltiSnips itself.
+
+Note: Due to some technical limitations, it is not possible for pdb to print
+the code of the snippet with the `l`/`ll` commands.
+
+You can enable it and configure it with the folowing variables: >
+
+    let g:UltiSnipsDebugServerEnable=0
+        (bool) Set to 1 to Enable the debug server. If an exception occurs or
+        a breakpoint (see below) is set, a Pdb server is launched, and you can
+        connect to it through telnet.
+
+    let g:UltiSnipsDebugHost='localhost'
+        (string) The host the server listens on
+
+    let g:UltiSnipsDebugPort=8080
+        (int) The port the server listens to
+
+    let g:UltiSnipsPMDebugBlocking=0
+        (bool) Set whether the post mortem debugger should freeze vim.
+        If set to 0, vim will continue to run if an exception
+        arises while expanding a snippet and the error message describing the
+        error will be printed with the directives to connect to the remote
+        debug server. Internally, Pdb will run in another thread and the session
+        will use the python trace back object stored at the moment the error
+        was caught. The variable values and the application state may not reflect
+        the exact state at the moment of the error.
+        If set to 1, vim will simply freeze on the error and will resume
+        only after quiting the debugging session (you must connect via telnet
+        to type the Pdb's `quit` command to resume vim). However, the
+        execution is paused right after caughting the exception, reflecting
+        the exact state when the error occured.
+
+NOTE: Do not run vim as root with `g:UltiSnipsDebugServerEnable=1` since anything
+can connect to it and do anything with root privileges.
+Try to use these features only for...  debugging... and turn it off when you
+are done.
+
+These variables can be set at any moment. The debug server will be active
+only when an exception arises (or a breakpoint set as below is reached),
+and only if `g:UltiSnipsDebugServerEnable` is set at the moment of the
+error. It will be innactive as soon as the `quit` command is issued
+from telnet.
+
+1.1 Setting breakpoints                      *UltiSnips-Advanced-breakpoints*
+-----------------------
+
+The easiest way of setting a breakpoint inside a snippet or UltiSnips
+internal code is the following: >
+
+    from UltiSnips.remote_pdb import RemotePDB
+    RemotePDB.breakpoint()
+
+...You can also raise an exception since it will be caught, and then will
+launch the post-mortem session. However, using the breakpoint method allows
+to continue the execution once the debugger quit.
+
+1.2 Accessing Pdb                                    *UltiSnips-Advanced-Pdb*
+-----------------
+
+Even though it's possible to use the builtin Pdb, (or any other compatible
+debugger), the best experience is achived with Pdb++.
+You can install it this way: >
+
+    pip install pdbpp
+
+It is a no-configuration replacement of the built-in pdb.
+
+To connect to the pdb server, simply use a telnet-like client.
+To have readline support (arrow keys working and history), you can use socat: >
+
+    socat READLINE,history=$HOME/.ultisnips-dbg-history TCP:localhost:8080
+
+(Change `localhost` and `8080` to match your configuration)
+To leave the server and continue the execution, run Pdb's `quit` command
+
+Known issue: Tab completion is not supported yet.
+

doc/UltiSnips.txt

@@ -55,9 +55,10 @@ UltiSnips                                      *snippet* *snippets* *UltiSnips*
 5. UltiSnips and Other Plugins                  |UltiSnips-other-plugins|
    5.1 Existing Integrations                    |UltiSnips-integrations|
    5.2 Extending UltiSnips                      |UltiSnips-extending|
-6. FAQ                                          |UltiSnips-FAQ|
-7. Helping Out                                  |UltiSnips-helping|
-8. Contributors                                 |UltiSnips-contributors|
+6. Debugging                                    |UltiSnips-Debugging|
+7. FAQ                                          |UltiSnips-FAQ|
+8. Helping Out                                  |UltiSnips-helping|
+9. Contributors                                 |UltiSnips-contributors|
 
 This plugin only works if 'compatible' is not set.
 {Vi does not have any of these features}
@@ -1811,7 +1812,18 @@ AddNewSnippetSource. Please contact us on github if you integrate UltiSnips
 with your plugin so it can be listed in the docs.
 
 =============================================================================
-6. FAQ                                                        *UltiSnips-FAQ*
+6. Debugging                                            *UltiSnips-Debugging*
+
+UltiSnips comes with a remote debugger disabled
+by default. When authoring a complex snippet
+with python code, you may want to be able to
+set breakpoints to inspect variables.
+It is also useful when debugging UltiSnips itself.
+
+See |UltiSnips-Advanced-Debugging| for more informations
+
+=============================================================================
+7. FAQ                                                        *UltiSnips-FAQ*
 
 Q: Do I have to call UltiSnips#ExpandSnippet() to check if a snippet is
    expandable? Is there instead an analog of neosnippet#expandable?
@@ -1835,7 +1847,7 @@ A: Yes there is, try
   endfunction
 
 =============================================================================
-7. Helping Out                                            *UltiSnips-helping*
+8. Helping Out                                            *UltiSnips-helping*
 
 UltiSnips needs the help of the Vim community to keep improving. Please
 consider joining this effort by providing new features or bug reports.
@@ -1849,7 +1861,7 @@ You can contribute by fixing or reporting bugs in our issue tracker:
 https://github.com/sirver/ultisnips/issues
 
 =============================================================================
-8. Contributors                                      *UltiSnips-contributors*
+9. Contributors                                      *UltiSnips-contributors*
 
 UltiSnips has been started and maintained from Jun 2009 - Dec 2015 by Holger
 Rapp (@SirVer, [email protected]). Up to April 2018 it was maintained by Stanislav

plugin/UltiSnips.vim

@@ -10,6 +10,24 @@ if version < 704
    finish
 endif
 
+" Enable Post debug server config
+if !exists("g:UltiSnipsDebugServerEnable")
+   let g:UltiSnipsDebugServerEnable = 0
+endif
+
+if !exists("g:UltiSnipsDebugHost")
+   let g:UltiSnipsDebugHost = 'localhost'
+endif
+
+if !exists("g:UltiSnipsDebugPort")
+   let g:UltiSnipsDebugPort = 8080
+endif
+
+if !exists("g:UltiSnipsPMDebugBlocking")
+   let g:UltiSnipsPMDebugBlocking = 0
+endif
+
+
 " The Commands we define.
 command! -bang -nargs=? -complete=customlist,UltiSnips#FileTypeComplete UltiSnipsEdit
     \ :call UltiSnips#Edit(<q-bang>, <q-args>)

pythonx/UltiSnips/err_to_scratch_buffer.py

@@ -4,9 +4,12 @@
 import traceback
 import re
 import sys
+import time
+from bdb import BdbQuit
 
 from UltiSnips import vim_helper
 from UltiSnips.error import PebkacError
+from UltiSnips.remote_pdb import RemotePDB
 
 
 def _report_exception(self, msg, e):
@@ -42,11 +45,20 @@ def wrap(func):
     def wrapper(self, *args, **kwds):
         try:
             return func(self, *args, **kwds)
+        except BdbQuit :
+            pass # A debugger stopped, but it's not really an error
         except PebkacError as e:
+            if RemotePDB.is_enable() :
+                RemotePDB.pm()
             msg = "UltiSnips Error:\n\n"
             msg += str(e).strip()
+            if RemotePDB.is_enable() :
+                host, port = RemotePDB.get_host_port()
+                msg += '\nUltisnips\' post mortem debug server caught the error. Run `telnet {}:{}` to inspect it with pdb\n'.format(host, port)
             _report_exception(self, msg, e)
         except Exception as e:  # pylint: disable=bare-except
+            if RemotePDB.is_enable() :
+                RemotePDB.pm()
             msg = """An error occured. This is either a bug in UltiSnips or a bug in a
 snippet definition. If you think this is a bug, please report it to
 https://github.com/SirVer/ultisnips/issues/new
@@ -56,6 +68,10 @@ def wrapper(self, *args, **kwds):
 Following is the full stack trace:
 """
             msg += traceback.format_exc()
+            if RemotePDB.is_enable() :
+                host, port = RemotePDB.get_host_port()
+                msg += '\nUltisnips\' post mortem debug server caught the error. Run `telnet {}:{}` to inspect it with pdb\n'.format(host, port)
+
             _report_exception(self, msg, e)
 
     return wrapper

pythonx/UltiSnips/remote_pdb.py

@@ -0,0 +1,111 @@
+import sys
+import threading
+from bdb import BdbQuit
+
+from UltiSnips import vim_helper
+
+class RemotePDB(object):
+    """
+    Launch a pdb instance listening on (host, port).
+    Used to provide debug facilities you can access with netcat or telnet.
+    """
+
+    singleton = None
+
+    def __init__(self, host, port):
+        self.host = host
+        self.port = port
+        self._pdb = None
+
+    def start_server(self):
+        """
+        Create an instance of Pdb bound to a socket
+        """
+        if self._pdb is not None :
+          return
+        import pdb 
+        import socket
+
+        self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        self.server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
+        self.server.bind((self.host, self.port))
+        self.server.listen(1)
+        self.connection, address = self.server.accept()
+        io = self.connection.makefile("rw")
+        parent = self
+
+        class Pdb(pdb.Pdb):
+            """Patch quit to close the connection"""
+            def set_quit(self):
+                parent._shutdown()
+                super().set_quit()
+
+        self._pdb = Pdb(stdin=io, stdout=io)
+
+    def _pm(self, tb):
+        """
+        Launch the server as post mortem on the currently handled exception
+        """
+        try:
+            self._pdb.interaction(None, tb) 
+        except: # Ignore all exceptions part of debugger shutdown (and bugs... https://bugs.python.org/issue44461 )
+            pass
+
+    def set_trace(self, frame):
+        self._pdb.set_trace(frame)
+
+    def _shutdown(self):
+        if self._pdb is not None :
+            import socket
+            self.connection.shutdown(socket.SHUT_RDWR)
+            self.connection.close()
+            self.server.close()
+            self._pdb = None
+
+    @staticmethod
+    def get_host_port(host=None, port=None):
+        if host is None :
+            host = vim_helper.eval('g:UltiSnipsDebugHost')
+        if port is None :
+            port = int(vim_helper.eval('g:UltiSnipsDebugPort'))
+        return host, port
+
+    @staticmethod
+    def is_enable():
+        return bool(int(vim_helper.eval('g:UltiSnipsDebugServerEnable')))
+
+    @staticmethod
+    def is_blocking():
+        return bool(int(vim_helper.eval('g:UltiSnipsPMDebugBlocking')))
+
+    @classmethod
+    def _create(cls):
+        if cls.singleton is None:
+            cls.singleton = cls(*cls.get_host_port())
+
+    @classmethod
+    def breakpoint(cls, host=None, port=None):
+        if cls.singleton is None and not cls.is_enable() :
+            return
+        cls._create()
+        cls.singleton.start_server()
+        cls.singleton.set_trace(sys._getframe().f_back)
+
+    @classmethod
+    def pm(cls):
+        """
+        Launch the server as post mortem on the currently handled exception
+        """
+        if cls.singleton is None and not cls.is_enable() :
+            return
+        cls._create()
+        t, val, tb = sys.exc_info()
+        def _thread_run():
+            cls.singleton.start_server()
+            cls.singleton._pm(tb)
+        if cls.is_blocking() :
+          _thread_run()
+        else :
+          thread = threading.Thread(target=_thread_run)
+          thread.start()
+

test/vim_test_case.py

@@ -135,6 +135,12 @@ def setUp(self):
         vim_config.append('let g:UltiSnipsJumpForwardTrigger="?"')
         vim_config.append('let g:UltiSnipsJumpBackwardTrigger="+"')
         vim_config.append('let g:UltiSnipsListSnippets="@"')
+
+        vim_config.append('let g:UltiSnipsDebugServerEnable={}'.format(1 if self.pdb_enable else 0))
+        vim_config.append('let g:UltiSnipsDebugHost="{}"'.format(self.pdb_host))
+        vim_config.append('let g:UltiSnipsDebugPort={}'.format(self.pdb_port))
+        vim_config.append('let g:UltiSnipsPMDebugBlocking={}'.format(1 if self.pdb_block else 0))
+
 
         # Work around https://github.com/vim/vim/issues/3117 for testing >
         # py3.7 on Vim 8.1. Actually also reported against UltiSnips