Issue223 #260
Issue223 #260
Conversation
There was a problem hiding this comment.
Got a couple of errors I encountered but looks good and an interesting approach to JSON encoding I wasn't aware of personally.
Apologies for the delay in getting to review this. If you're out of spare time now then please just let me know and I can look into investigating the errors if you're okay with that.
| ], | ||
| "python.testing.unittestEnabled": false, | ||
| "python.testing.pytestEnabled": true, | ||
| "python.defaultInterpreterPath": "F:\\Program Files\\python3.9\\python" |
There was a problem hiding this comment.
Would remove this as it's dependent on the machine config of whoever's running the scripts
|
|
||
| processor.write_json(fname, prepped) | ||
|
|
||
| processor = ProcessorUSMART() | ||
|
|
There was a problem hiding this comment.
processor.process() below needs updated to pass the file_type param value of json I think. When I ran this I was still getting .csv file names instead of .json files
| @@ -110,7 +110,7 @@ def main(): | |||
| source_usmart = pd.concat( | |||
| [ | |||
| source_usmart, | |||
| pd.read_csv( | |||
| pd.read_json( | |||
| folder + r"/" + filename, | |||
| parse_dates=["DateCreated", "DateUpdated"], | |||
There was a problem hiding this comment.
This line caused an error for me. I think it needs to be convert_dates instead
Description
Previously, the usmart api script (usmart.py) outputs to a .csv which is then ingested into merge_data.py. This should be converted to a .json output to keep consistent with the rest of the project.
Changes done in a way which allows a mix and match approach while other providers are switched from csv to json
Motivation and Context
See original ticket
How Has This Been Tested?
Unit tests (pytest) written for JSON objects and testing reading/writing of the same
Screenshots (if appropriate):
Types of changes
Checklist: