DJL Community Meeting 5/4/2022

[zachgk]

Time: 9:00AM-10:00AM PT

This is part of a series of biweekly online community meetings to discuss DJL development, events, and projects. The agenda and conference information are located below. If you have a project to share, are working on part of DJL, would like to raise a discussion about DJL development, or want to discuss anything else, leave a comment on this discussion to add it to the agenda.

Instructions on joining the meeting

==============Conference Bridge Information==============
You have been invited to an online meeting, powered by Amazon Chime.
Chime meeting ID: 9238192792
Join via Chime clients (manually): Select "Meetings > Join a Meeting", and enter 9238192792
Join via Chime clients (auto-call): If you invite auto-call as attendee, Chime will call you when the meeting starts, select "Answer"
Join via browser screen share: https://chime.aws/9238192792
Join via phone (US): [+1-929-432-4463,,,9238192792#](tel:+1-929-432-4463,,,9238192792)
Join via phone (US toll-free): [+1-855-552-4463,,,9238192792#](tel:+1-855-552-4463,,,9238192792)
International dial-in: https://chime.aws/dialinnumbers/
In-room video system: Ext: 62000, Meeting PIN: 9238192792#
=================================================

Agenda

Introductions and Updates

Please say: name, company/organization/school, what are you working on with DJL

Scheduled Items

What improvements are you most hoping for in DJL?

[nguyenvietyen]

I'm Viet Yen Nguyen, CTO of @hypefactors.

Typical use cases for us:

• NLP inference workloads integrated in streaming data pipelines
• Computer vision

Wishlist:

• Support for running on Apple M1 chips (seems to be along the way)
• Collaboration with Huggingface; make their model hub part accessible like a model zoo
• Big dream: share pre/post processor code with Python pipelines using a deeper integration than process invokation. Or alternatively: bring NDArray functionality up to a much higher level so that postprocessing in JVM-space is incentivized
• Out-of-the box benchmarking framework a la Triton Server's performance analyzer

I have a general desire regarding back pressuring GPU loads when under heavy load (e.g. memory, computational). So that OOMs or other resource contention problems are avoided. But I think it's out of scope of DJL, and is probably more related to hardware/software/engine interaction.

[saudet]

This is currently just out of curiosity, but have you considered using TorchScript for your pre/post processing? It's possible to export a lot of Python code as is to TorchScript, and since we can run that using the C++ runtime of PyTorch without CPython, it sounds like the perfect compromise. And if you've already tried to do that, what have you found lacking from TorchScript?

[lanking520]

This is currently just out of curiosity, but have you considered using TorchScript for your pre/post processing? It's possible to export a lot of Python code as is to TorchScript, and since we can run that using the C++ runtime of PyTorch without CPython, it sounds like the perfect compromise. And if you've already tried to do that, what have you found lacking from TorchScript?

You can, if you are willing to throw your logic to a Module and make it happened. We can treat it just another model loading. But, you lose the flexibility to tweak around in Java and build all kinds of operators, and support for training.

Torch introduced FX where it can allow this from happening... I think this will be helpful for some Python-unique logic. But mostly we found the difficult part is something with pandas, opencv, and other 3rd party library Being used and impossible to convert.

[saudet]

Yes, it does restrict what one can do in Python and Java, but I still think it's a good compromise the scientists and engineers can agree to follow. The rest is mainly about extending the runtime (such as adding operators that can call OpenCV, Arrow, etc) and the development tools. It's kind of ironic that the best IDE-like tool that I've found that can deal with C++, Java, and Python in the same project to be Visual Studio Code https://github.com/microsoft/vscode, but hey that's the not-so-great world we live in. :)

BTW, it is possible to train these modules, so that's not a problem, see issue pytorch/pytorch#17614

[saudet]

FYI for @nguyenvietyen

• Out-of-the box benchmarking framework a la Triton Server's performance analyzer

Triton has a low-level C API, which allows us to use it as part of the JVM process, making it very easy to do zero-copy data sharing with other libraries like CPython, PyTorch, OpenCV, etc, see the JavaCPP Presets for Triton Inference Server for an example of that. That's also the kind of thing that could be integrated into DJL, so make sure to let them know if this is something you would like to use. Also please let @jackyh know about this if it look interesting to you since he can provide support from inside NVIDIA.

[nguyenvietyen]

Thank you for pointing to this work @saudet.

I now realised I put out a "need" of which the finer requirements were not thought out. Now by reflecting deeper on it, we tend to use benchmark-like code for two purposes: 1) to detect bottlenecks, and 2) to establish baseline performance. When it comes to 2), I don't think the effort of creating presets wouldn't be worthwhile, for we already have quite effective means to establish baseline performance (e.g. simple Python script). I think for 1), Triton's performance analyzer as it is would not fit our own picture, because it would by its very nature omit overhead that is normal in our execution environment (e.g. fiber management, tokenization, data pipeline I/O). This is only a statement of our needs - perhaps someone else has a more justifiable use case.

[frankfliu]

We do have customer doing exactly as you described. The issue with this solution is that user has to use IValue directly, which is tied to PyTorch engine. We didn't have time to abstract this type of usage for all the engines.

[saudet]

Right, I think the larger issue is that we don't have more general standards than ONNX. It is possible to have TVM compile, for example, both a TorchScript module and an ONNX model, exported from TF or wherever, and have them run on the same TVM runtime, so it's not like it's impossible to come up with something like this, but unfortunately it seems like we're stuck with the limited capabilities of ONNX when it comes to a common format.

[nguyenvietyen]

This is currently just out of curiosity, but have you considered using TorchScript for your pre/post processing?

Not yet. But I'll include it in the options when we tackle the next model.

For (processing) code that will run 24/7, we kinda prefer doing that in the JVM space due to static typing and benefiting from the existing code quality assurance infrastructure we have there. So today, that means we maintain two versions of the same processing code, in Python and Scala. This workaround isn't ideal, but is the best known trade-off for us at the time.

[saudet]

I like the suggestion. Have you tried this in practice?

If you're talking about the "0% Python" approach, I did that with DL4J in the past and it works fine, but the main issue is that there's always going to be something from Python you'd wish you had. The C++ API for PyTorch isn't as complete as the Python API, but it is a lot more complete than the C++ or Java APIs of any other framework (DL4J, TensorFlow, etc), so I'm sure there are projects where it makes sense to just do everything in C++ (or Java in this case) with PyTorch, avoiding the whole Python ecosystem, especially if the engineers working on those projects don't especially like Python. :)

I've used JavaCPP presets before, and find that memory management related effort tends to bite back. In that regard, DJL tends to abstracts nicely into the garbage-collected Java mindset.

Memory management with DJL is actually very similar to JavaCPP. The way DJL does it comes in fact from DL4J and JavaCPP:
http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/
https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management
Sure, there were bugs, many were fixed, there might still be some, so please do report them, and they will be fixed!
Conversely, it means that it shouldn't be too hard to integrate the JavaCPP Presets for PyTorch to DJL...

[saudet]

That reminds me though, I'm aware of at least a few happy users of the JavaCPP Presets for PyTorch:
@HGuillemet @hal-314 @hulk-us @guangster @jbaron @jxtps @mehmetcalikus @napps2017 @scott01272001 @tgg9527 @carlosuc3m @doofin @wumo @zaleslaw
Please comment about your experience here! Good or bad, especially the bad, I want to get those bugs fixed. Thank you!

[hal-314]

Hi

I'm a happy user of JavaPresets. I used for deployment. In this case, two torchscript models in spark using scala. One of them was an object detector that D4J doesn't support out of the box.

The good part is when you know how it works, their is 100% integration of Pytorch with your other code. Use exactly the Pytorch version used to develop yor model. Also, you get the whole c++ pytorch api that it's almost equivalent to Python. For example, new torch.jit.optimize or torch.inference_mode.

The bad parts are:

• Sometimes, you catch a bug that doesn't happen on pytorch. For example, I had problems with using only one core for model inference. I suspect is how is being compiled. There are some flags that are different (pytorch uses mkl, ... ) in JavaCpp. In future, I would want to experiment statically compiling torch along mkl, etc to match with pytorch.
• Few complete docs. Torchscript github issue was essential too. For example, for automatic memory management or reducing binary size with Pytorch.
• Be ready for trial and error at the beginning. Pass data & get model prediction from the model was hard. For example, couldn't make it work the new api to create a tensor :/ .

At the end, it was worthy. JavaPresets for Pytorch has edges (is one developer) and is quite untested but it gives you sometime that nobody could, the whole c++ pytorch api without intermediate layers -> if you want to exploit the full pytorch power, it is the only library in Java.

[jxtps]

I'm using JavaCPP PyTorch for inference in Java, still doing the training in Python. We're still in development so have yet to actually deploy in production, but so far it works great!

While the "close to the C++ metal" experience has a bit of a learning curve, it provides fantastic control of precisely what happens to which tensor when. This makes it possible to leverage the GPU for various pre/post-processing tasks "around" the network, allowing me to not have to go back to python when I need to make minor interface changes.

Specific example: at first I was creating wrappers models in python before exporting the actual model. Now I do that glue work in Java, which makes for a much more flexible development experience. There's always a bit of glue code necessary, and this enables a nicer separation between the ML training & inference.

[saudet]

@hal-314

• Sometimes, you catch a bug that doesn't happen on pytorch. For example, I had problems with using only one core for model inference. I suspect is how is being compiled. There are some flags that are different (pytorch uses mkl, ... ) in JavaCpp. In future, I would want to experiment statically compiling torch along mkl, etc to match with pytorch.

It is possible to link the presets with binaries from LibTorch, which are linked statically with MKL, etc. Instructions for that are here:
https://github.com/bytedeco/javacpp-presets/tree/master/pytorch#documentation

• Few complete docs. Torchscript github issue was essential too. For example, for automatic memory management or reducing binary size with Pytorch.

The C++ API in general isn't well documented, indeed.

• Be ready for trial and error at the beginning. Pass data & get model prediction from the model was hard. For example, couldn't make it work the new api to create a tensor :/ .

If you have a specific example, please let me know! It's probably not something complicated that needs to be fixed.

Thanks for the feedback, everyone!

[jbaron]

my 2 cents:

• plus: Nice consistent C++ API that translates well to Java. I was surprised for example how easy it was to to define and train a LSTM network.

• minus: there is no way after defining the network to make it into a static network (like using tracing). This is all done at Python side and has no C++ API equivalent. This results that every training steps involves a lot of Java/Native calls, making it slow (especially as the network complexity grows).

[saudet]

• minus: there is no way after defining the network to make it into a static network (like using tracing). This is all done at Python side and has no C++ API equivalent. This results that every training steps involves a lot of Java/Native calls, making it slow (especially as the network complexity grows).

Yes, this is annoying. It looks like it would require a lot of engineering effort to pull off, see pytorch/pytorch#26086 (comment), so maybe we'll always need Python somewhere in the pipeline after all.

@frankfliu @lanking520 @zachgk Have you guys ever received demand to have CPython integrated into Java, so that we can run embedded Python scripts like this: https://github.com/bytedeco/javacpp-presets/tree/master/tvm#the-howtodeployjava-source-file
without having to do Docker or manually install Anaconda, and figure out how to pass data around between processes and what not?

[saudet]

I think @HannesWell is using the JavaCPP Presets for CPython in production with OSGi.
@HannesWell Any comments about how that's working out?

[HannesWell]

I think @HannesWell is using the JavaCPP Presets for CPython in production with OSGi. @HannesWell Any comments about how that's working out?

That's right, we are using it in Eclipse (which uses its Equinox OSGi implementation). Actually that's working well.
The main issue we had was extracting the native files. But once I have completed bytedeco/javacpp#511 (Sorry again for the delay, I regularly think about completing this but then other for me more important things come up), this should be working out of the box.
At the moment we have embedded CPython presets of JavaCPP embedded in a Eclipse-Plugin and perform the extraction by ourself. This is the main reason it is not urgent for me, but I would definitely want to simplify this on our side and want to avoid custom handling.

[guangster]

Hi, I'm very happy with JavaCPP and the PyTorch presets. We use them in research projects involving tabular data where data pipelines are involved and have to come from Java. A few notes: Plus - It's the only library that allows Java fine-tuning of pre-trained networks, whereas most other libraries can only load existing weights for inference, or train without loading pre-trained weights. Very crucial for a lot of deep learning research use-cases. - Integration with OpenCV presets works flawlessly thanks to JavaCPP's Pointer APIs. This is huge for efficiency and convenience in applying transforms in Java. - One of the closest PyTorch-like API in Java I have seen. Minus - Long-term maintenance. Since it is mostly maintained by one person, there's a risk for us to use it in larger projects if this is ever dropped. Related to this, the release versions are also tied to the latest torch (newest updates in javacpp are designed for newest torch). - Torchscript is a bit of a pain because it doesn't work for things like CLIP which has a ton of control flows, but that is more of an issue with torch and c++ torchlib I guess.

…

On Wed, May 11, 2022 at 2:33 PM jxtps ***@***.***> wrote: I'm using JavaCPP PyTorch for inference in Java, still doing the training in Python. We're still in development so have yet to actually deploy in production, but so far it works great! While the "close to the C++ metal" experience has a bit of a learning curve, it provides fantastic control of precisely what happens to which tensor when. This makes it possible to leverage the GPU for various pre/post-processing tasks "around" the network, allowing me to not have to go back to python when I need to make minor interface changes. Specific example: at first I was creating wrappers models in python before exporting the actual model. Now I do that glue work in Java, which makes for a much more flexible development experience. There's always a bit of glue code necessary, and this enables a nicer separation between the ML training & inference. — Reply to this email directly, view it on GitHub <#1593 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRFTEYOF52FZSOIYK7L353VJP4RBANCNFSM5UAR33QA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[saudet]

• Long-term maintenance. Since it is mostly maintained by one person, there's a risk for us to use it in larger projects if this is ever dropped. Related to this, the release versions are also tied to the latest torch (newest updates in javacpp are designed for newest torch).

Yes, I'm trying really hard to make this into more than a single person project. I think we'll be one step closer to making this happen if we can make the case that DJL should be rebased on the JavaCPP Presets for PyTorch, CPython, OpenCV, etc, similarly to how it's already using TensorFlow for Java. Having projects used by big names like Amazon, Google, NVIDIA (/cc @jackyh), ServiceNow, etc do help with confidence, and that creates a positive feedback loop. If you know of any other contacts that might help, please do let me know about them!

[lanking520]

Thanks for everyone's input, we do have a community meeting next Wednesday. Please feel free to bring these topic there for more discussion.

[saudet]

The meetings for DJL, and TF for Java for that matter (/cc @karllessard), are around 1~2 AM for me, so it's not very convenient, but when there's discussion related to JavaCPP that is on the agenda, I'm happy to attend. Either for this time or any other future meetings, just let me know! Thanks