Can't get example working in "NLP support with Huggingface tokenizers"

[nezda]

I am just starting my journey with djl. I want to use custom huggingface QA models so I followed the directions in the Usage section of http://djl.ai/extensions/tokenizers/ . I converted deepset/bert-base-cased-squad2 which resulted in model/nlp/question_answer/ai/djl/huggingface/pytorch/deepset/bert-base-cased-squad2/0.0.1/bert-base-cased-squad2.zip as described. However, when I try to load the model using

Criteria<QAInput, String> criteria = Criteria.builder()
    .setTypes(QAInput.class, String.class)
    .optModelPath(Paths.get("model/nlp/question_answer/ai/djl/huggingface/pytorch/deepset/bert-base-cased-squad2/0.0.1/bert-base-cased-squad2.zip"))
    .optProgress(new ProgressBar()).build();

a call to criteria.loadModel() raises exceptions. Here is my model debug logging including those exceptions

[16:51:31 Engine DEBUG] No engine found from EngineProvider
zz
[16:51:31 DefaultModelZoo DEBUG] Scanning models in repo: class ai.djl.repository.SimpleRepository, file:/Users/noone/code/djltest/model/nlp/question_answer/ai/djl/huggingface/pytorch/deepset/bert-base-cased-squad2/0.0.1/bert-base-cased-squad2.zip
[16:51:31 ModelZoo DEBUG] Loading model with Criteria:
	Application: UNDEFINED
	Input: class ai.djl.modality.nlp.qa.QAInput
	Output: class java.lang.String
	ModelZoo: ai.djl.localmodelzoo
	No translator supplied

[16:51:31 ModelZoo DEBUG] Searching model in specified model zoo: ai.djl.localmodelzoo
[16:51:31 ModelZoo DEBUG] Checking ModelLoader: ai.djl.localmodelzoo:bert-base-cased-squad2 UNDEFINED [
	ai.djl.localmodelzoo/bert-base-cased-squad2/bert-base-cased-squad2 {}
]
[16:51:31 ModelZoo DEBUG] No matching default translator found. The valid input and output classes are: 
	(ai.djl.ndarray.NDList, ai.djl.ndarray.NDList)
 for ModelLoader: ai.djl.localmodelzoo:bert-base-cased-squad2
[16:51:31 Config ERROR] Yikes! Caught exception java.lang.RuntimeException: ai.djl.repository.zoo.ModelNotFoundException: No matching model with specified Input/Output type found. in Thread[main,5,main]
java.lang.RuntimeException: ai.djl.repository.zoo.ModelNotFoundException: No matching model with specified Input/Output type found.
	at com.noone.Extractor.main(Extractor.java:84)
Caused by: ai.djl.repository.zoo.ModelNotFoundException: No matching model with specified Input/Output type found.
	at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:184)
	at com.noone.Extractor.main(Extractor.java:76)
Caused by: ai.djl.repository.zoo.ModelNotFoundException: No matching default translator found. The valid input and output classes are: 
	(ai.djl.ndarray.NDList, ai.djl.ndarray.NDList)

	at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:91)
	at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:172)
	... 1 common frames omitted

The file path in SimpleRepository correctly points to the model zip file. I am not clear on many things.

• Will the Criteria look inside bert-base-cased-squad2.zip to find
  • the model bert-base-cased-squad2.pt (because they both have the same base name, bert-base-cased-squad2
  • does it read serving.properties and configure itself with translatorFactory=ai.djl.huggingface.translator.QuestionAnsweringTranslatorFactory
    • does it "know" to find the model's tokenizer.json in there?

I'm happy to provide any further details and would be grateful for any tips.

Thanks in advance,

• Luke

[zachgk]

The DJL modules are fairly à la carte and it seems like you are missing some dependencies. I think you are missing the PyTorch engine. You also probably want the PyTorch model zoo where you can find the translator. You can also try using the HuggingFace model zoo which has models that can be downloaded more easily.

[nezda]

I think I have all of those dependencies

        <dependency>
          <groupId>ai.djl</groupId>
          <artifactId>api</artifactId>
        </dependency>
        <dependency>
          <groupId>ai.djl.huggingface</groupId>
          <artifactId>tokenizers</artifactId>
        </dependency>
        <dependency>
          <groupId>ai.djl.pytorch</groupId>
          <artifactId>pytorch-model-zoo</artifactId>
        </dependency>
        <dependency>
          <groupId>ai.djl.pytorch</groupId>
          <artifactId>pytorch-jni</artifactId>
          <scope>runtime</scope>
        </dependency>
        <dependency>
          <groupId>ai.djl.pytorch</groupId>
          <artifactId>pytorch-native-cpu</artifactId>
          <classifier>osx-x86_64</classifier>
          <scope>runtime</scope>
        </dependency>

and I believe I was able to load and use prepackaged models but I don't understand how to load a model in a zip file as created here. Is it supposed to look inside the zip files I provided and interpret serving.properties to configure itself?

Please advise,

• Luke

[frankfliu]

You do need specify a TranslatorFactory if your model is in a zip file. You can either:

1. use ai.djl.huggingface.translator.QuestionAnsweringTranslatorFactory directory,
2. use DeferredTranslatorFactory, it will read the configurations from serving.properties.

Criteria<QAInput, String> criteria = Criteria.builder()
    .setTypes(QAInput.class, String.class)
    .optModelPath(Paths.get("model/nlp/question_answer/ai/djl/huggingface/pytorch/deepset/bert-base-cased-squad2/0.0.1/bert-base-cased-squad2.zip"))
    .optTranslatorFactory(new DeferredTranslatorFactory())
    .optProgress(new ProgressBar()).build();

[nezda]

Thank you so much @frankfliu - both of those options worked! I made a PR #2511 so the docs reflect this advice.

I was poking around trying to see how the logging might help user know more about what was working than it currently showed. The only thing different when it works is it executes

djl/api/src/main/java/ai/djl/repository/zoo/BaseModelLoader.java

Line 95 in ce57306

mrl.prepare(artifact, progress);

just a few lines after the other threw the exception which logs

[09:46:46 SimpleRepository DEBUG] Extracting artifact: /Users/noone/code/model/nlp/question_answer/ai/djl/huggingface/pytorch/deepset/bert-base-cased-squad2/0.0.1/bert-base-cased-squad2.zip ...
Downloading: 100% |████████████████████████████████████████|

It might be cool to log what it found in that zip file to increase user confidence they're close? I see just a couple lines later it gets the model path and parses the serving properties

djl/api/src/main/java/ai/djl/repository/zoo/BaseModelLoader.java

Lines 101 to 108 in ce57306

	Path modelPath = mrl.getRepository().getResourceDirectory(artifact);
	Path modelDir = Files.isRegularFile(modelPath) ? modelPath.getParent() : modelPath;
	if (modelDir == null) {
	throw new AssertionError("Directory should not be null.");
	}
	modelDir = Utils.getNestedModelDir(modelDir);
	loadServingProperties(modelDir, arguments, options);

Would it make sense to do any of this stuff even in the event of failure on

djl/api/src/main/java/ai/djl/repository/zoo/BaseModelLoader.java

Line 85 in ce57306

TranslatorFactory factory = getTranslatorFactory(criteria, arguments);

just to improve the error reporting?