File validation

[jl-wynen]

Should be simple and fast.
E.g.

• is valid HDF5
• is valid NeXus (complicated and there are non-standard files)
• scan for secrets
• more?

However, is this even a responsibility of Scitacean or should the server take care of it?

[nitrosx]

Given how variable is the nexus structure, I think that the only thing we can validate is if we can open the file as a hdf5 and nexus.
I'm not sure if that is needed or not and if it provides any value.

Maybe we should provide a basic validation and user can provide a customize validation function that is suited for their use-case?

Another validation that we can do is checking the hash if it is provided.

[jl-wynen]

Maybe we should provide a basic validation and user can provide a customize validation function that is suited for their use-case?

I agree. There are ways to validate NeXus files according to application definitions. But this is very specific to a given application and can't really be done by Scitacean.

Another validation that we can do is checking the hash if it is provided.

This is already implemented. But it is a check for the upload/download, i.e. it checks if the file after upload/download has the same hash as before.

[nitrosx]

If you are OK with it, let's go with a basic validation, aka if we can open the hdf/nexus file and maybe see the first level of keys.

[jl-wynen]

I am now more inclined to see this as the responsibility of the user, not Scitacean. Checking that a HDF5 file can be opened provides little utility. Especially given the extra complications in Scitacean for supporting this check.

So I would say we close this issue as "won't fix". Users can use tools like https://github.com/scipp/chexus if they want to validate their files. Any objections @nitrosx?