xy data loader #1287
Merged
xy data loader #1287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR can be squashed. |
|
This PR requires archesproject/arches#10003 |
|
The previously mentioned PR needs to merged into dev/7.5.x for this to work. |
chiatt
approved these changes
Sep 28, 2023
| @@ -139,11 +138,10 @@ def post(self, request): | |||
| file_data["renderer"] = next( | |||
| (renderer["id"] for renderer in settings.RENDERERS if renderer["name"] == "pdfreader"), None | |||
| ) | |||
| elif dataset_default_format is not None: # instrument was given by zip file name | |||
| elif file_data["name"].split(".")[-1] == "dx" or file_data["name"].split(".")[-1] == 'txt': # instrument was given by zip file name | |||
There was a problem hiding this comment.
It seems like afs should have a setting with a list of possible 'txt' extensions.
| } | ||
| const delimiterCharacter = config?.delimiterCharacter ?? ','; | ||
|
|
||
| const valueRegex = (delimiterCharacter.length < 2) ? new RegExp(`[${delimiterCharacter}\\s]+`) : new RegExp(`${delimiterCharacter}`); |
There was a problem hiding this comment.
This regex silently failed when I used ** as a delimiter. This is a pretty unlikely delimiter, but it would be nice to have a message on the front end stating that the delimiter cannot be parsed - just in case.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds features relevant for #1218 . Allows for dynamically formatted files to be read. Deprecates other afs formatters (raman reader, xrf reader, fors reader) which are retained for backward compatibility. Adds the concept of an "importer" - an attribute stored on the file tile that points to a database record that specifies a configuration that gets passed to the new xy reader. That configuration tells the XY reader how to parse the file. The reader itself uses the XY Parser utility to do the dirty work of parsing the file. Supports the concept of transformations - provided multiple Y values, transforming those Y values via known methods and functions. As part of this PR, the "mean" or "average" transformation is included (as well as a "basic" transformation which selects the first y value if multiples are given.
Resolves #1218
I suspect we will need to include some kind of import functionality in Arches for Science to transfer these importers between installations and perhaps provide a well-known set, but that is outside the scope of this PR.