Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated picks created by lag_calc are not written to file in the develop version #598

Closed
zhong-yy opened this issue Dec 14, 2024 · 1 comment

Comments

@zhong-yy
Copy link

zhong-yy commented Dec 14, 2024

Describe the bug
The modified Party object after running (client_)lag_calc cannot be written to file correctly in the develop version. The write function just write the original catalog to file, without containing the modified picks and there is no comments in the picks. However, the master version works as expected.

To Reproduce

  • Switch to the develop version
  • Running (client_)lag_calc on a Party object.
  • Save Party with party.write("x.tgz") in the default tar format.
  • Read the party from file party_read=Party().read("x.tgz")
  • Compare the object before writing to file (party[0][0].event.picks) and the one recovered from file party_read[0][0].event.picks
  • Switch to the master version and do the above steps

Expected behavior
The modified picks should be written to file.

Desktop (please complete the following information):

  • Operating System: Ubuntu 24.04.1
  • Python version: 3.10
  • EQcorrscan version: 0.5.0.dev

Additional context
A related question: reading a party larger than 100M is extremely slow, how can I accelerate the reading process? Currently I first convert my catalog to a pandas dataframe and then save it to a csv file, but it cannot be used to directly recover the original state of the party.

@zhong-yy zhong-yy changed the title Updated picks created by lag_calc are be written to file in the develop version Updated picks created by lag_calc are written to file in the develop version Dec 14, 2024
@zhong-yy zhong-yy changed the title Updated picks created by lag_calc are written to file in the develop version Updated picks created by lag_calc are not written to file in the develop version Dec 14, 2024
@calum-chamberlain
Copy link
Member

Reading and writing large catalog objects is very slow, which is the main pinch point in reading and writing Party objects. Your best bet is to pickle your parties:

import pickle

with open("party.pkl", "wb") as f:
    pickle.dump(party, f)

I usually write one party per data chunk to make it easier to read lots in.

The issue with different picks in develop relates to an update to the Family.catalog property in develop, which did not properly check for changes to detections. This should be patched in #601

calum-chamberlain added a commit that referenced this issue Dec 15, 2024
Always get new catalog from detections #598
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants