Use scalar instead of Tensor in inference

[Baerlie]

Hi,

I try to use a model that expects a Tensor and a scalar (integer) as input. I tried to create a custom Translator to provide the attributes needed for the model inference, but I can add only NDArray to NDList. Here is my code and the error message, I think that explains my problem better than words :)

// loading the model
Criteria<NDList, NDList> criteria = Criteria.builder()
                .setTypes(NDList.class, NDList.class)
                .optModelPath(Paths.get(MODEL_PATH))
                //.optTranslator(new AudioTranslator())
                .optEngine("PyTorch")
                .optProgress(new ProgressBar())
                .build();
ZooModel<NDList, NDList> model = criteria.loadModel();

// creating predictor
Predictor<NDList, NDList> predictor = model.newPredictor(new AudioTranslator());
PtNDManager manager = (PtNDManager)NDManager.newBaseManager();

// run inference
float[] data = new float[3];
data[0] = 0; data[1] = 1; data[2] = 2;
NDList values = new NDList(manager.create(data, new Shape(1, data.length)));
predictor.predict(values);

// processInput function of custom Translator
@Override
public NDList processInput(TranslatorContext ctx, NDList input) {
    // convert to 32 bit Float Tensor, values between -1 and 1
    float[] inputData = input.get(0).toFloatArray();
    float[] data = new float[inputData.length];

    for(int i=0; i<inputData.length; i++) {
        data[i] = inputData[i] * (1/32768);
    }

    NDArray array = ctx.getNDManager().create(data, new Shape(1, input.size()));
    NDArray sr = ctx.getNDManager().create(16000)/*.toType(DataType.INT32, false)*/;
    return new NDList(array, sr);
}

The runtime error is the following:
Exception in thread "main" ai.djl.translate.TranslateException: ai.djl.engine.EngineException: forward() Expected a value of type 'int' for argument 'sr' but instead found type 'Tensor'.
Position: 2
Declaration: forward(torch.vad.model.vad_annotator.VADRNNJITMerge self, Tensor x, int sr) -> Tensor

Is it possible to somehow force the predictor to call the forward method of the model with something else than Tensor attributes?

Thank you for any help in advance!
Br
Beatrice

[frankfliu]

@Baerlie

PyTorch use IValue in its c++ API, we can automatically map NDList to IValue for Tensors (List, Map, tuple of Tensors). For non-tensor data or other complicate data type, you have to use IValue directly: https://github.com/deepjavalibrary/djl/blob/master/engines/pytorch/pytorch-engine/src/test/java/ai/djl/pytorch/integration/IValueTest.java#L201

[Baerlie]

Thanks, I will read and try!

[Baerlie]

@frankfliu
It works, thank you for the fast and simple solution!

// example code for inference with Tensor and int, in case someone else needs it
Criteria<NDList, NDList> criteria = Criteria.builder()
        .setTypes(NDList.class, NDList.class)
        .optModelPath(Paths.get(MODEL_PATH))
        .optEngine("PyTorch")
        .optProgress(new ProgressBar())
        .optDevice(Device.cpu())
        .build();
ZooModel<NDList, NDList> model = criteria.loadModel();

PtNDManager manager = (PtNDManager)NDManager.newBaseManager(Device.cpu());
PtSymbolBlock block = (PtSymbolBlock) model.getBlock();
IValue audioData = IValue.from((PtNDArray)manager.zeros(new Shape(1, SAMPLE_WINDOW_SIZE)));
IValue sr = IValue.from(16000);
IValue ret = block.forward(audioData, sr);

audioData.close();
sr.close();
ret.close();