Implement "import/export from/to JSON/CSV" functionality

[jamesmunns]

We don't have a great way to actually dump history, including:

• logs
• schemas
• endpoints
• topics-in
• topics-out

We should have an easy way to import and export these. Potentially always including schemas to allow for data decoding.

It might also be nice to have a compact postcard file format version of this as well.

[CBJamo]

It would be nice to have just a little filtering. Two obvious ones jump out to me.

First is path filtering, something along the lines of export topics --path /some/thing and some simple globbing, like export topics --path /group/*

And second, device filtering, so export logs --device <id or name>

[thaodt]

hi @jamesmunns I'd like to tackle this issue if you still didn't plan to do it yet.

My initial approach is introducing new sub-command, lets say ExportCommand.
I will leverage the existing poststation-sdk functions, use SquadClient which already has methods such as get_device_schemas() and get_device_logs(). Also Im gonna use the device lookup functionality from poststation-cli that supports both hex serial numbers and name matching.

The new sub-command could look like this:

// New subcommands for poststation-cli
enum ExportCommand {
    Logs {
        device: String,
        path_filter: Option<String>,
        format: ExportFormat,
        output: PathBuf,
    },
    Schemas {
        device: String,
        path_filter: Option<String>,
        format: ExportFormat,
        output: PathBuf,
    },
    Topics {
        device: String,
        direction: TopicDirection,
        path_filter: Option<String>,
        format: ExportFormat,
        output: PathBuf,
    }
}

enum ExportFormat {
    Json,
    Csv,
    Postcard,
}

enum TopicDirection {
    In,
    Out,
    Both,
}

The Export flow will be:

1. Parse device identifier using existing device lookup fns
2. Fetch data using poststation-sdk methods
3. Apply path filters if specified
4. Serialize to chosen format
5. Write to output file

but I cannot find the details of SquadClient on poststation-sdk rust docs, but i see you use it directly in some examples. IIUC, SquadClient is PoststationClient ?

What do you think about my approach?

[jamesmunns]

Hey @thaodt, you're welcome to take a try at this, I think I'll need to make changes on the Poststation side to fully support exporting historical topic and endpoint data, as there is no endpoint for that. Schemas and logs could be exported currently.

Yes, SquadClient was renamed to PoststationClient in the last major release.

Feel free to implement partially now, or I'll add a comment here when I add Poststation endpoints for exporting.