Fill-mask language model

[agistaputri20]

Does anyone has an example on how to run inference a fill-mask language modeling on Scala/Java using DJL?

[frankfliu]

We don't have an end2end example, but you can find sample code in our unit test: https://github.com/deepjavalibrary/djl/blob/master/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/FillMaskTranslatorTest.java#L90-L108

It's very simple if you use DJL model zoo:

        Criteria<String, Classifications> criteria =
                Criteria.builder()
                        .setTypes(String.class, Classifications.class)
                        .optModelUrls("djl://ai.djl.huggingface.pytorch/bert-base-uncased")
                        .optEngine("PyTorch")
                        .build();

[agistaputri20]

Alright thank you! But how to make that ModelUrl from huggingface model? Because I have my own model on huggingface hub.

[agistaputri20]

And when I use my own model, do I have to make a translator that contain a preprocess and postprocess function?

[frankfliu]

@agistaputri20

1. For Huggingface fill-mask model, we have built-in translators that you can use, they handle the pre-processing and porst-processing for you. You don't need create your own translator. You have want to some special handling, you can use our FillMaskTranslator as base and modify on top of it.
2. DJL support batch inference, but you need set padding/truncation for tokenizer.
3. You use this script to import your hugingface model: https://github.com/deepjavalibrary/djl/blob/master/extensions/tokenizers/src/main/python/model_zoo_importer.py, the output .zip file is ready to use in DJL, you only need specify TranslatorFactory:

        Criteria<String, Classifications> criteria =
                Criteria.builder()
                        .setTypes(String.class, Classifications.class)
                        .optModelPath(Paths.get("YOUR_ZIP_FILE.zip"))
                        .optEngine("PyTorch")
                        .optTranslatorFactory(new FillMaskTranslatorFactory())
                        .build();

[frankfliu]

Yes, you can use Criteria and load the model directly in Scala. What you need is just include huggingface tokenizer in your project:

https://docs.djl.ai/master/extensions/tokenizers/index.html#installation

[agistaputri20]

In this section should I download the master DJL folder on my local first, so I can get the path then try to run it?

[frankfliu]

Yes, that's the easiest way to convert a huggingface FillMask model to DJL.

[agistaputri20]

Great! I already have my huggingface zip model

Then, I've tried to run my code but why this is happen? Even though I have imported the ai.djl.huggingface.translator.FillMaskTranslator and huggingface dependency on my build.sbt file. So, what should I add on my program?

[frankfliu]

You need remove .optTranslator(new FillMaskTranslator()) in your code and replace with .optTranslatorFactory(new FillMaskTranslatorFactory())

FillMaskTranslator's constructor is not public, you can not use new FillMaskTranslator(), you have to use FillMaskTranslator.builter() ... to create a new FillMaskTranslator instance. Or use preferred FillMaskTranslatorFactory way.

[agistaputri20]

I have to use this model from huggingface: https://huggingface.co/jibi2906/my-finetuned-distilbert-pt. But it makes error like this, do you know how to solve it?

[frankfliu]

I updated my earlier reply. You need use .optTranslatorFactory(new FillMaskTranslatorFactory())

[agistaputri20]

Nice, it works! Thank you :)

[agistaputri20]

Wait, sorry is it possible to detect and fill more than one [MASK]?

[frankfliu]

No, that's the model's limitation.