Consider use of UUIDs to allow importing multiple independent sets of HSDS CSV files

[pmackay]

How are the ID columns used when importing CSVs? Are they just for cross referencing within the CSV content? Presumably on import Rails allocates IDs to the entries?

Has the use of a UUID field been considered for replication of content, so an item (org or location) could be uniquely identified? I notice OR has a "Resource ID" field, might that be a future feature on Ohana?

[monfresh]

The import script uses the ID in the CSV file to set the ID in the Rails DB, so the ID in the CSV file acts as a unique identifier.

[pmackay]

Would the Resource ID field in the OR spec be implemented by adding a separate UUID field to each applicable model?

And how does current import process handle entries with the same ID as one in the database? Presumably it cannot be a UUID because it has to be an integer/key column.

[monfresh]

Why is a Resource ID field needed? The id field already serves the purpose of a unique identifier.

Is there a specific issue with the import process you are running into? If so, please state the bug and the steps to reproduce.

[pmackay]

It comes back to importing records from other systems. Its not possible to define IDs just as simple database primary keys that guarantee uniqueness. What if I have an Ohana db with 100 orgs, and then try to import another org for a different system that has an ID that collides with one in Ohana?

[monfresh]

Thanks! That's more helpful. Starting with the actual issue is always a good idea 😄 I hadn't thought about this scenario, and in this case, yes, a uuid field would make sense.

[md5]

Having a uuid or resource_id field on each table would make it possible to implement the metadata table in HSDS as well, although the spec seems to say that the id field should contain the UUID.

[md5]

I supposed it's possible to do so now since the id fields are all the same type.

[cderenburger]

Could this potentially allow importing (and potentially resetting/dropping) from each separate source db as independent tasks? I'm currently attempting to import our whole state db and am having issues completing an import. Breaking this into smaller tasks might help with larger data sets as well.

[monfresh]

This particular issue only deals with adding a new field to the DB.

If you're having problems importing a large data set, please open a new issue. Thanks!