Gradient from hybrid (integer/double) input arrays

[anessler]

Hello all,
I am attemtping to utilize DJL to implement the ANI nueral network that predicts a potential energy from a system of atoms. The inputs are an integer array ("species") to specify atom types (e.g., 1=hydrogen, 6=carbon, etc.) and the "coordinates" for atoms in the system (N x 3, where N is the number of atoms and each atom has an X-, Y-, Z-coordinate). The forward method returns the energy of the system (double). In Python, the torch.autograd.grad function is used with the energy and the coordinates as the input to obtain the gradient for each of the atoms (Nx3). However, in my DJL implementation, the returned energy object has the requires gradient value set to false.

I am under the impression that since "species" is an input to the forward method and an integer array (i.e., non-differentiable, requires grad=false) the energy is also being returned with the requires grad flag set to false for the DJL implementation. I am certinaly open to other interpretations!

How can I alter my DJL implementation to utilize the output energy to compute the gradient? I have attached a Jupyter Notebook (utilizing the IJava kernel) that I am currently using. Unfortunately, the PyTorch script of my model is ~2x too large to include as an attachement...

Any help would be greatly appreciated!
test_ipynb.zip

EDIT:
I also wanted to include the Python version that is able to print out the gradient. Additionally, the Python script creates the PyTorch script of the model assuming you have TorchANI set up.

jit2.zip

[KingWaq]

Hello, I'm a student from Tsinghua University, and I'm sorry that I have no ability to answer your problem. But now I have a problem which you can handle, I think. So I want to ask for your help and I will be deeply grateful for any help from you.
The problem is that I have a pytorch model which has two inputs(one is float[][][] input1, another is int[][][] input2), but when I create my translator, I find that the translator just has one input(Translator<I, O>),So I really have no idea how to put the one-input-translator into creating the Criteria that used to get model. So could you tell me how to solve this problem?

[anessler]

Hello! I appreciate the response and I am happy to provide what little information I can. I moved away from the Translator implementation pretty quickly, because it was evident I was not going to get the results I wanted so I am certainly not an expert in any sense of the word...

I would recommend trying to put the information into an NDList object and use that as your input to the Translator. I am not sure what you have setup as an output from your model, but I am going to assume it is multiple arrays as well (NDList).

Translator<NDList, NDList> translator = new Translator<NDList, NDList>(){

     @Override
     public NDList processInput(translatorContext cox, NDList input){
          // Any modifications you wish to do with the input data
          return input;
     }

     @Override
     public NDList processOutput(TranslatorContext ctx, NDList list){
          // Any modifications you wish to do with the output data
          return list;
     }
}

float[][][] input1 = new float[][][] // Create your input float array
int[][][] input2 = new int[][][]  // Create your input int array
try(NDManager manager = NDManager.newBaseManager()){
     Model model = Model.newInstance("TORCHSCRIPT.pt");
     model.load(Paths.get("/PATH/TO/TORCHSCRIPT.pt");

     NDList list = new NDList();
     NDArray array1 = manager.create(input1);
     NDArray array2 = manager.create(input2);
     list.add(array1);
     list.add(array2);

     try(Predictor<NDList, NDList> predictor = model.newPredictor(translator)){
          NDList output = predictor.predict(list);
     }
}

I highly doubt this will work... But this might be a start to the correct setup? You can edit the output/inputs of the translator and processInput/processOutput methods to try and match your model requirements. Best of luck!

[anessler]

I ended up implementing the model using the JavaCPP Presets for PyTorch to get a gradient from the hybrid input.

[KingWaq]

Hello！Dear Aaron, I feel so inconceivable when I receive this reply. Your idea have been a great help to me. Yeah, I try to writing code like your example and it works!But now, I’m meeting a new problem,which is when I put the Ndlist into my model, there is a error that ‘ Expected Tuple but got String ’,just like the figure.And after a lot of attempts, I find when my input’s shape does not match the requirements, the error will occur. I have a model needing two inputs—float[1][1][10] input1,float[1][1][10] input2 that is uesed to test,and if I provide inputs like float[1][1][10] input1,float[1][1][10], there is no error,however, if I provide inputs that does not match the shape, it doesnot work and gives me the same error . But I ‘m not sure if the reasons for the former and the later are the same, because the person who gives me the former model told me that the inputs I provided was right.So now, I have no idea that what I can do for next But I’m really grateful for you help, and you can tell me any trouble you have met,I will try my best to provide my advice and wish to help you.Thank you very much 从 Windows 版邮件发送 发件人: Aaron Nessler 发送时间: 2023年5月22日 23:35 收件人: deepjavalibrary/djl 抄送: Because_of_Your_Beauty; Comment 主题: Re: [deepjavalibrary/djl] Gradient from hybrid (integer/double) inputarrays (Discussion #2475) Hello! I appreciate the response and I am happy to provide what little information I can. I moved away from the Translator implementation pretty quickly, because it was evident I was not going to get the results I wanted so I am certainly not an expert in any sense of the word... I would recommend trying to put the information into an NDList object and use that as your input to the Translator. I am not sure what you have setup as an output from your model, but I am going to assume it is multiple arrays as well (NDList). Translator<NDList, NDList> translator = new Translator<NDList, NDList>(){ @OverRide public NDList processInput(translatorContext cox, NDList input){ // Any modifications you wish to do with the input data return input; } @OverRide public NDList processOutput(TranslatorContext ctx, NDList list){ // Any modifications you wish to do with the output data return list; } } float[][][] input1 = new float[][][] // Create your input float array int[][][] input2 = new int[][][] // Create your input int array try(NDManager manager = NDManager.newBaseManager()){ Model model = Model.newInstance("TORCHSCRIPT.pt"); model.load(Paths.get("/PATH/TO/TORCHSCRIPT.pt"); NDList list = new NDList(); NDArray array1 = manager.create(input1); NDArray array2 = manager.create(input2); list.add(array1); list.add(array2); try(Predictor<NDList, NDList> predictor = model.newPredictor(translator)){ NDList output = predictor.predict(inputI); } } I highly doubt this will work... But this might be a start to the correct setup? You can edit the output/inputs of the translator and processInput/processOutput methods to try and match your model requirements. Best of luck! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[frankfliu]

You model only accept Tuple of non-tensor input, you have to use IValue directly. see: https://docs.djl.ai/master/docs/faq.html#how-can-i-pass-arbitrary-input-data-type-to-a-pytorch-model