How to set up custom embedding nets for CUDA training #1202
janfb
announced in
Announcements
Replies: 1 comment 5 replies
-
|
the PR under which you commented is merged into the Essentially, you have to pass the device only once, i.e., when you create the inference class. Importantly, your custom embedding net should not be on the CUDA device yet, it will be moved internally. net = YourCustomEmbeddingNet(...) # lives on the CPU, output layers returns 10 units
neural_posterior = posterior_nn(model="maf", embedding_net=net, hidden_features=10, num_transforms=2)
inference = SNPE(prior=prior, device="cuda", density_estimator=neural_posterior)Then, The same applies to the data: The remaining code you posted will probably not work, I think there is a misunderstanding. You do not need to code an explicit training loop. All you need to do is appending the data to inference.append_simulations(your_sampled_theta, your_simulated_data)
inference.train()
posterior = inference.build_posterior()(If you really have to use your custom dataloader (e.g., because your data does not fit into the RAM at once), then things become a bit tricky, but there has been a similar issue, see #1193) |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Thank you very much for the reply :D I still tried with my custom dataloader using append_simulation like this: The data seemed to be passed successfully. Here I used "cuda" for the prior, and make sure the data are saved in cpu. However there still some errors in the next training step. And I wonder if it's a pytorch issue. I'm using python 3.8, pytorch 2.3.0, and py3.8_cuda11.8_cudnn8.7.0_0. Do you think there is something wrong with the package versions? I really appreciate you help! Here are all the warnings and errors I got, in case I missed telling something important: |
Beta Was this translation helpful? Give feedback.
-
|
In your code, it doesn't really make sene to append the data to Things you can try to debug this:
I do not think that it is a PyTorch issue. It looks like a device mismatch coming from the |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thank you very much for the explaination:) I changed my code to a simple and clear structure as you showed before: I didn't use the assert ... because the embedding net is not iterabel. In the script I'm using now:
However the errors didn't really change a lot... Could you notice something I've missed here? Thank you so so much! |
Beta Was this translation helpful? Give feedback.
-
|
Given your code snipped, I don't see why this would not work. Maybe your data or your embedding net is still on the GPU? You can check the device of the net like this: assert next(net.parameters()).device == "cpu"
assert x.device == "cpu"
assert theta.device == "cpu"
neural_posterior = posterior_nn(model="maf", embedding_net=net, hidden_features=10, num_transforms=2)
inference = SNPE(device="cuda", density_estimator=neural_posterior)
inference.append_simulations(theta, x)
inference.train()
posterior = inference.build_posterior() |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! That was an issue from the previous data simulation. The code fore SNPE worked well:) |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Moving the comment by @ningyuxin1999 under #1186 here for discussion:
Here are the errors I got from it:
I tried with cpu device as well, but got:
I'm not sure if I understood it correctly, could you maybe help? I really appriciate that.
Originally posted by @ningyuxin1999 in #1186 (comment)
Beta Was this translation helpful? Give feedback.
All reactions