pykeepass_cache library

[Evidlo]

I've been working on a library that transparently wraps pykeepass and temporarily holds a database open in a background process. I thought it might be useful for keepmenu.

Also any feedback would be appreciated.

[firecat53]

Thanks for letting me know! I'll definitely try it out when I get a chance. I've always been a little frustrated with the opening delay.

[firecat53]

I'm having a chicken/egg problem. pykeepass_cache requires the password, but I typically prompt for the password only if the database isn't already open. So how can I supply the password to pykeepass_cache if the server is already running without prompting for it? The goal is to open the database on first run, prompting for the password if necessary but then being able to re-open keepmenu without the password as long as the timeout hasn't been reached.

[Evidlo]

Use the cached_databases function to check if the database is already open on the server.

…

On Aug 28, 2019 12:35 PM, "Scott Hansen" ***@***.***> wrote: I'm having a chicken/egg problem. pykeepass_cache requires the password, but I typically prompt for the password only if the database isn't already open. So how can I supply the password to pykeepass_cache if the server is already running without prompting for it? The goal is to open the database on first run, prompting for the password if necessary but then being able to re-open keepmenu without the password as long as the timeout hasn't been reached. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#40?email_source=notifications&email_token=ABJT7YK6XLFTA2XEUHPDK5TQG2ZMLA5CNFSM4IP744PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5L4LOA#issuecomment-525845944>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJT7YJ2UL2EILUPSDZFD6LQG2ZMLANCNFSM4IP744PA> .

[firecat53]

Sigh. Sorry. Dumb question.

A request: would it be possible to add a method/call to shut down the server (close the database)?

[firecat53]

Few more things after testing for awhile:

• It actually adds a significantly noticeable delay to retrieving entries even after the db was open. Even just retrieving entries from my not too big keepass database in ipython %time print([i for i in kpo.entries]) took 3.71s (wall time). Regular pykeepass was 157ms.
• Tends to crash on EOFError: stream has been closed after just a few calls.
• Always crashes on first run with no db open yet on a FileNotFoundError (looks like for the socket)

[Evidlo]

It actually adds a significantly noticeable delay to retrieving entries even after the db was open.
Even just retrieving entries from my not too big keepass database in ipython %time print([i for i in kpo.entries]) took 3.71s (wall time). Regular pykeepass was 157ms.

How many entries are in that database? My database with 100 entries takes about 100ms.

Tends to crash on EOFError: stream has been closed after just a few calls.

That happens when you access kpo after the server has shut down. You need to delete the object before opening again. I might be able to work around this, or at the very least raise a custom exception when the server is shut down.

del kpo
kpo = PyKeePass('test.kdbx', 'password', 'test.key')

Always crashes on first run with no db open yet on a FileNotFoundError (looks like for the socket)

I haven't seen this. Can you post some example code?

[firecat53]

How many entries are in that database? My database with 100 entries takes about 100ms.

725

In ipython:

In [1]: %time print([i for i in kpo.entries])

It actually appears that the processing and printing are very fast, but there is a significant delay after the printing completes before the entire command finishes.

That happens when you access kpo after the server has shut down. You need to delete the object before opening again. I might be able to work around this, or at the very least raise a custom exception when the server is shut down.

Always crashes on first run with no db open yet on a FileNotFoundError (looks like for the socket)

I haven't seen this. Can you post some example code?

To reproduce both these in ipython (ensure no existing db instances are open):

In [1]: from pykeepass_cache import PyKeePass 
In [2]: kpo=PyKeePass("/home/user/passwords.kdbx", password="password", keyfile="/home/user/keyfile")
---------------------------------------------------------------------------                                                    [26/4720]
ConnectionRefusedError                    Traceback (most recent call last)                                                     
~/docs/family/scott/src/projects/pykeepass_cache/pykeepass_cache/pykeepass_cache.py in _fork_and_run(func, timeout, socket_path)
     93     try:                 
---> 94         conn = unix_connect(socket_path)    
     95         return func(conn)                                   
                                  
~/.local/lib/python3.7/site-packages/rpyc/utils/factory.py in unix_connect(path, service, config)
    111     """                                                                                                                         
--> 112     s = SocketStream.unix_connect(path)
    113     return connect_stream(s, service, config)
                                                                    
~/.local/lib/python3.7/site-packages/rpyc/core/stream.py in unix_connect(cls, path, timeout)
    184             s.settimeout(timeout)
--> 185             s.connect(path)
    186             return cls(s)                                                                                                       
                                  
ConnectionRefusedError: [Errno 111] Connection refused
                                                                    
During handling of the above exception, another exception occurred:
                                                                    
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-2-3863a71dd4ec> in <module>
----> 1 kpo=PyKeePass("/home/firecat53/passwords.kdbx", password="password", keyfile="/home/firecat53/keyfile")

~/docs/family/scott/src/projects/pykeepass_cache/pykeepass_cache/pykeepass_cache.py in PyKeePass(filename, password, keyfile, transforme
d_key, timeout, socket_path)
    174 
    175     func = lambda conn: conn.root.PyKeePass(filename, password, keyfile, transformed_key)
--> 176     return _fork_and_run(func, timeout=timeout, socket_path=socket_path)

~/docs/family/scott/src/projects/pykeepass_cache/pykeepass_cache/pykeepass_cache.py in _fork_and_run(func, timeout, socket_path)
    129             # maybe wait for existence of socket_path
    130             time.sleep(1)
--> 131             conn = unix_connect(socket_path)
    132             return func(conn)
    133 

~/.local/lib/python3.7/site-packages/rpyc/utils/factory.py in unix_connect(path, service, config)
    110     :returns: an RPyC connection
    111     """
--> 112     s = SocketStream.unix_connect(path)
    113     return connect_stream(s, service, config)
    114 

~/.local/lib/python3.7/site-packages/rpyc/core/stream.py in unix_connect(cls, path, timeout)
    183         try:
    184             s.settimeout(timeout)
--> 185             s.connect(path)
    186             return cls(s)
    187         except BaseException:

FileNotFoundError: [Errno 2] No such file or directory

In [3]: kpo=PyKeePass("/home/user/passwords.kdbx", password="password", keyfile="/home/user/keyfile")
In [4]: %time print([i for i in kpo.entries]) 
...... Lots of output
CPU times: user 354 ms, sys: 43.2 ms, total: 397 ms Wall time: 3.73 s

Running it a second time in a row:

In [5]: %time print([i for i in kpo.entries])                                                                                  [36/4578]
---------------------------------------------------------------------------
EOFError                                  Traceback (most recent call last)                                
<timed eval> in <module>                                            
                                  
~/.local/lib/python3.7/site-packages/rpyc/core/netref.py in __getattribute__(self, name)
    159             return object.__getattribute__(self, "__array__")
    160         else:
--> 161             return syncreq(self, consts.HANDLE_GETATTR, name)
    162 
    163     def __getattr__(self, name):

~/.local/lib/python3.7/site-packages/rpyc/core/netref.py in syncreq(proxy, handler, *args)
     74     """
     75     conn = object.__getattribute__(proxy, "____conn__")
---> 76     return conn.sync_request(handler, proxy, *args)
     77 
     78 

~/.local/lib/python3.7/site-packages/rpyc/core/protocol.py in sync_request(self, handler, *args)
    456         """
    457         timeout = self._config["sync_request_timeout"]
--> 458         return self.async_request(handler, *args, timeout=timeout).value
    459 
    460     def _async_request(self, handler, args=(), callback=(lambda a, b: None)):  # serving

~/.local/lib/python3.7/site-packages/rpyc/core/protocol.py in async_request(self, handler, *args, **kwargs)
    477             raise TypeError("got unexpected keyword argument(s) %s" % (list(kwargs.keys()),))
    478         res = AsyncResult(self)
--> 479         self._async_request(handler, args, res)
    480         if timeout is not None:
    481             res.set_expiry(timeout)
~/.local/lib/python3.7/site-packages/rpyc/core/protocol.py in _async_request(self, handler, args, callback)                     [3/4578]
    462         self._request_callbacks[seq] = callback
    463         try:
--> 464             self._send(consts.MSG_REQUEST, seq, (handler, self._box(args)))
    465         except Exception:
    466             self._request_callbacks.pop(seq, None)

~/.local/lib/python3.7/site-packages/rpyc/core/protocol.py in _send(self, msg, seq, args)
    257                     continue
    258                 data = self._send_queue.pop(0)
--> 259                 self._channel.send(data)
    260             finally:
    261                 self._sendlock.release()

~/.local/lib/python3.7/site-packages/rpyc/core/channel.py in send(self, data)
     73         header = self.FRAME_HEADER.pack(len(data), compressed)
     74         buf = header + data + self.FLUSHER
---> 75         self.stream.write(buf)

~/.local/lib/python3.7/site-packages/rpyc/core/stream.py in write(self, data)
    265         try:
    266             while data:
--> 267                 count = self.sock.send(data[:self.MAX_IO_CHUNK])
    268                 data = data[count:]
    269         except socket.error:

~/.local/lib/python3.7/site-packages/rpyc/core/stream.py in __getattr__(self, name)
     92         if name.startswith("__"):  # issue 71
     93             raise AttributeError("stream has been closed")
---> 94         raise EOFError("stream has been closed")
     95 
     96     def close(self):

EOFError: stream has been closed

It also appears that the server instance is staying running in the background
even when the timeout is expired.

[Evidlo]

The first issue with FileNotFoundError should be fixed now: libkeepass/pykeepass_cache@31012f5

It also appears that the server instance is staying running in the background even when the timeout is expired.

It seems to work for me

import subprocess                                                                                                   
from pykeepass_cache import PyKeePass                                                                               
import time                                                                                                         
                                                                                                                    
print('-------------- server running, socket should exist ------------------')                                      
kp = PyKeePass('test3.kdbx', 'password', 'test3.key', timeout=5)                                                    
print(subprocess.run('lsof /tmp/pykeepass.sock', shell=True))                                                       
                                                                                                                    
time.sleep(6)                                                                                                       
print('-------------- server not running, socket should not exist ------------------')                              
print(subprocess.run('lsof /tmp/pykeepass.sock', shell=True))                                                       

[evan@blackbox tmp] python test.py
-------------- server running, socket should exist ------------------
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
python  12062 evan    3u  unix 0x00000000cd53dc26      0t0 1187577 /tmp/pykeepass.sock type=STREAM
python  12062 evan    4u  unix 0x000000007b662acf      0t0 1187579 /tmp/pykeepass.sock type=STREAM
CompletedProcess(args='lsof /tmp/pykeepass.sock', returncode=0)
-------------- server not running, socket should not exist ------------------
lsof: status error on /tmp/pykeepass.sock: No such file or directory
lsof 4.91
 latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
 latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
 latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
 usage: [-?abhKlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-E] [+|-e s] [+|-f[gG]]
 [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
 [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Use the ``-h'' option to get more help information.
CompletedProcess(args='lsof /tmp/pykeepass.sock', returncode=1)