Skip to content
This repository has been archived by the owner on Apr 22, 2024. It is now read-only.

Properly handle DB transactions that might fail #135

Open
ssciolla opened this issue May 4, 2020 · 0 comments
Open

Properly handle DB transactions that might fail #135

ssciolla opened this issue May 4, 2020 · 0 comments

Comments

@ssciolla
Copy link
Contributor

ssciolla commented May 4, 2020

While reviewing PR #129, @pushyamig added the following comment:

Looking into the sqlalchemy library, the execute function will raise an exception ( in the unknown situation). This should be wrapped in try-catch. Similarly, in the other parts of code that are doing a DB transaction. This issue need not be solved here but can be taken as a separate issue

Originally posted by @pushyamig in #129

While I will admit some ignorance on this topic and can certainly be persuaded to change my views, I don't really feel like wrapping our current code using database transactions -- conn.execute or df.to_sql -- with try/except alone is going to do very much. Without the try/except, the code will fail if a transaction fails, leaving the database in an inconsistent state (if other transactions were already executed and committed). With the try/except, the only difference I can imagine would be that we're raising the error ourselves and exiting explicitly.

That said, we can definitely do more to monitor transactions and rollback proposed changes if they would cause an error. This would keep the database from entering an inconsistent or partially populated state. I don't think (but don't know for sure) this can be done with pandas.

I think one way we might handle this is to use the SQLAlchemy ORM with a try/except. Something like what is shown in the first example here. I already have a rough implementation of the ORM, so this might not be that much work.

@ssciolla ssciolla self-assigned this May 4, 2020
@ssciolla ssciolla changed the title Handle DB transactions more gracefully Properly handle DB transactions that might fail May 4, 2020
@ssciolla ssciolla removed their assignment Feb 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant