Deleting cache entries when expired

[livia-b]

I'd like to delete some cache entries when they are not valid anymore (for example, my result is a filename, and I want to discard it if the file does not exist). Is there a direct way to do it? Is it an enhancement you could be interested in?

Here is my workaround:
https://gist.github.com/livia-b/63ffd88a4da5dcb4b357

[mmckerns]

klepto doesn't have a notion of a no longer "valid" entry. What kind of things would make an entry no longer valid…? Cache entry expires after a certain amount of time…?

Your function has three parts:

1. lookup: handled by function.lookup(*args,**kwds)
2. delete: handled by cache.pop(function.key(*args, **kwds)) and similarly with cache.archive.pop
3. validation: not currently handled

With regard to (2), it's potentially a good idea to have some flavor of a pop-thru function, which would pop from both cache and archive. (1) is simple as is. For (3), I'm not sure what the compelling use cases are yet, so I'm not sure what's best to do in that regard.

If you are worried about there being a key corresponding to a file (as is in the case of a dir_archive) where the file no longer exists… that shouldn't ever be the case as the archive is always built from inspecting the list of files on disk. Then, if you have an entry in cache that is not in archive, because the file was deleted… you can use dump, load, or sync to update either the cache or archive as desired. Maybe a more compelling case is if the file becomes corrupt for some reason… and needs to be dropped, but then you'd just del or pop the file from the archive.

So maybe my question is why should valid be a cache method, or some free-standing function?

[livia-b]

Actually I was not thinking of a file belonging to a dir_archive, but to a file which is the output of a memoized function. So in my case I have something like:

@inf_cache()
def compute(*args){
     uniqueFilename = somefunction(*args)
     return uniqueFilename
 }

filename = checkCacheEntryValidity(compute, os.path.isfile,  *args)

Another use case could be (but I don't have concrete examples):

@inf_cache()
def compute(*args){
     obj = foo(*args)
     obj.expirationDate = bar(*args)
    return obj
 }
def valid(obj):
    return obj.expirationDate > datetime.datetime.now()
validObj= checkCacheEntryValidity(compute, valid,  *args)

My current approach of wrapping a memoized function seems to work (I haven't checked in the long run the additional burden of checking every call), but I wanted to see if it can be interesting having the cache dealing with it.

[mmckerns]

So, you are thinking that the filename would be an output of a memoized function…? Then basically, checking if the file is "valid" has nothing to do with it being cached. So, I'm not sure I understand why this should be built-in to klepto. In that case, it's not a valid cache entry you are looking for, but a valid cache output… which klepto doesn't ever check. Maybe I still don't get the use case you have in mind, or maybe I'm misunderstanding you.

You might want to look at what is available from the dir_archive class methods _fname, _mkdir, _getdir, _rmdir, _isdir. These use the pox module and some basic python standard library stuff.

[petebachant]

This is a very old issue, but a related question: Is there currently any functionality to set an elapsed time for an object to expire and be removed from the cache and/or archive?

[IvanaGyro]

@petebachant AFAIK, no. However, you can inherit the class in _archives.py and implement __getitem__ and __setitem__ to make the function of purging the expired entries.

[mmckerns]

entries that expire (after a pre-determined time, or after some other condition is met) are "coming soon"