[DISCUSSION] Feature idea for plugin development: standardize paths for cache and persistent data

[Yusyuriv]

The problem

Right now every plugin does its own thing when it needs to cache data or to save persistent data.

My proposal

Standardize paths for cache and persistent data and provide APIs both for C#/F# and for JS/Python plugins. The difference between persistent data directory and cache directory is that the cache directory should always be safe to delete. The plugin might need to regenerate the data on startup or on query, but it should still work. I had to do delete cache data myself a few times for the built-in Program plugin. Persistent data, as the name suggests, should not be deleted manually.

Cache

• The path to the cache directory would be FlowLauncher/Cache/{PLUGIN_NAME}
• IPublicApi gets several additional methods, most of which are not necessary, but are there for convenience. Method names are just rough ideas and are subject to change:
  • string GetCacheRootDirectoryPath() — returns the absolute path to the cache directory. Creates it if it does not exist yet.
  • void SaveCacheJsonFile<T>(string path, T data) — serializes data to JSON and saves it in a file. SaveCacheJsonFile("my-data.json", myData)
  • T? ReadCacheJsonFile<T>(string path) — reads the JSON file from a file in the current plugin's cache directory and deserializes it, returning the deserialized object. ReadCacheJsonFile<MyData>("my-data.json")
  • void SaveCacheYamlFile<T>(string path, T data) — serializes data to YAMLand saves it in a file.
  • T? ReadCacheYamlFile<T>(string path) — reads the YAML file from a file in the current plugin's cache directory and deserializes it, returning the deserialized object.
  • void SaveCacheTextFile(string path, string data) — same as the JSON one, but without serialization, for saving simple strings. SaveCacheTextFile("my-data.txt", myString)
  • string ReadCacheTextFile(string path) — same as the JSON one, but without deserialization, for reading simple strings. ReadCacheTextFile("my-data.txt")
  • string GetCacheFilePath(string path) — returns the absolute path to the file in the current plugin's cache directory, creating directories recursively if they don't exist. This if for when the data the plugin needs to save is not a simple text or JSON file. For example, if the plugin needs to cache images or if it needs to create an SQLite database
  • void ClearCache() — remove all files from the current plugin's cache directory

Persistent data

• The path to the persistent data directory would be FlowLauncher/Settings/{PLUGIN_NAME}/PersistentData (the current settings directory + /PersistentData) or FlowLauncher/PersistentData/{PLUGIN_ID}, whichever one looks better. I mentioned the Settings directory first because some users might be backing up their settings already, so they would also begin backing up plugins' persistent data this way without the need to manually add another directory to their backup software.
• IPublicApi gets additional methods, similar to the cache ones:
  • string GetPersistentDataRootDirectory()
  • void SavePersistentJsonFile<T>(string pathT data)
  • T? ReadPersistentJsonFile<T>(string pathTo
  • void SavePersistentYamlFile<T>(string pathT data)
  • T? ReadPersistentYamlFile<T>(string path)
  • void SavePersistentTextFile(string path, string data)
  • void ReadPersistentTextFile(string path)
  • string GetPersistentFilePath(string path)
  • void ClearPersistentData()

Built-in files

• Plugins can ship with their own data included (for example, Search-MDI plugin). This data would be in FlowLauncher/Plugins/{PLUGIN_DIR}. Unlike other types of data, this one doesn't need file saving/deleting. If a plugin includes data in its build, this data is supposed to be immutable.
• IPublicAPI gets additional methods, similar to the cache ones:
  • T? ReadIncludedJsonFile<T>(string path)
  • T? ReadIncludedYamlFile<T>(string path)
  • string ReadIncludedTextFile(string path)
  • string GetIncludedFilePath(string path)

Notes

I included methods for YAML reading/writing because some people might prefer YAML to JSON and Flow already includes a library for working with YAML, this could let plugin developers avoid a dependency.

Conclusion

I think this would simplify developing plugins that need to cache or save data. I'd like to hear your opinions.

[cibere]

For the cache, specifically with non jsonrpc-v1 plugins, what's the point in storing the data in files vs just in memory? or is the addition just for jsonrpc-v1 plugins?

Also, what's wrong with just using one of the system's temp dirs?

[Yusyuriv]

For the cache, specifically with non jsonrpc-v1 plugins, what's the point in storing the data in files vs just in memory? or is the addition just for jsonrpc-v1 plugins?

It's not just for v1. For example, it could be data that you update once a day by downloading updates from an API. You store it in a cache file so if the user restarts Flow Launcher, you don't need to re-download this data, it's already downloaded. But if the file gets deleted, it's not an issue, the plugin simply downloads the data from the API again and caches it again until the next day. This could lower the amount of requests you make to the API and could be helpful if the API limits the amount of requests you can make hourly, daily, weekly, or monthly.

Another example would be the Color plugin that needs to display images for the colors. Instead of re-rendering them on each keystroke, it renders the image for each color once and saves it in a cache directory. If the user types the same color again, it doesn't get re-rendered; instead, the already existing image file is displayed.

I think I've also seen a plugin that caches data in an SQLite database.

Also, what's wrong with just using one of the system's temp dirs?

Nothing wrong with that, I'm just thinking of ways to make plugin development more convenient and standardized.

[cibere]

Huh, I hadn't thought about it that way. I can see a couple ways of implementing that into my existing plugins 🤔

[jjw24]

I think this is a great proposal. The more we can standardise and help plugin devs reduce the amount of effort to create their plugins, the better it will be in the long run.

One thing I would note though, is whether we should put cache in FlowLauncher/Settings/{PLUGIN_NAME} along with persistent data? This perhaps will help when users want to back up plugin only data, but it could also add more to the size of the backup if persistent data is large. Or on the other hand move both out so user backup is smaller with settings only files and cache and persistent data can be re-saved.

BTW, there is no need to append the plugin id (FlowLauncher/Settings/{PLUGIN_NAME}-{PLUGIN_ID}), we don't expect duplicate plugins saved.

[Yusyuriv]

I think persistent data should, well, be persistent, so it should be easy to backup. For example, a plugin for quickly saving simple notes could store those notes in files. Users definitely would want those to still exist once they, for example, copy their settings to another PC.

Cache, on the other hand, by definition is something that should be okay to lose and should not need backups.

BTW, there is no need to append the plugin id (FlowLauncher/Settings/{PLUGIN_NAME}-{PLUGIN_ID}), we don't expect duplicate plugins saved.

Sorry, I was a little sleep-deprived when I wrote this so I mixed up plugin installations, settings, plugin IDs and versions. What I had in mind was that installed plugins are installed in Plugins/{NAME}-{VERSION}, which is not related to settings (Settings/{NAME}). I'll edit my original message.

[jjw24]

Just going to convert this into discussion :)