Unable to load the pb file from Custom training model

[Kabster17]

Hi Szaza,

Thanks a lot for your contribution. I am using Windows 10 and I have incorporated training of my custom model using tiny-yolo to identify 2 classes.
Now, I would like to load it into an android app however I am unable to use the app on my phone, it keeps crashing.
Is it because I did my training of my custom model using tensorflow 1.14.0?
As i am new to this area of Android Studio and applications. I would really appreciate some guidance. Thanks.
Here is the error that i am receiving when i start the app on my mobile device.

2019-12-29 18:41:56.072 28172-28172/org.tensorflow.yolo E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.tensorflow.yolo, PID: 28172
java.lang.RuntimeException: Failed to load model from 'file:///android_asset/tiny-yolo-voc-graph.pb'
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.(TensorFlowInferenceInterface.java:113)
at org.tensorflow.yolo.TensorFlowImageRecognizer.create(TensorFlowImageRecognizer.java:42)
at org.tensorflow.yolo.view.ClassifierActivity.onPreviewSizeChosen(ClassifierActivity.java:55)
at org.tensorflow.yolo.view.CameraActivity.lambda$setFragment$0$CameraActivity(CameraActivity.java:116)
at org.tensorflow.yolo.view.-$$Lambda$CameraActivity$SAOsaS8oaXCIIH441aIwhMnb5HU.onPreviewSizeChosen(Unknown Source:2)
at org.tensorflow.yolo.view.CameraConnectionFragment.setUpCameraOutputs(CameraConnectionFragment.java:291)
at org.tensorflow.yolo.view.CameraConnectionFragment.openCamera(CameraConnectionFragment.java:298)
at org.tensorflow.yolo.view.CameraConnectionFragment.access$000(CameraConnectionFragment.java:52)
at org.tensorflow.yolo.view.CameraConnectionFragment$1.onSurfaceTextureAvailable(CameraConnectionFragment.java:163)
at android.view.TextureView.getTextureLayer(TextureView.java:390)
at android.view.TextureView.draw(TextureView.java:339)
at android.view.View.updateDisplayListIfDirty(View.java:20761)
at android.view.View.draw(View.java:21614)
at android.view.ViewGroup.drawChild(ViewGroup.java:4558)
at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4333)
at android.view.View.updateDisplayListIfDirty(View.java:20747)
at android.view.View.draw(View.java:21614)
at android.view.ViewGroup.drawChild(ViewGroup.java:4558)
at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4333)
at android.view.View.draw(View.java:21891)
at android.view.View.updateDisplayListIfDirty(View.java:20761)
at android.view.View.draw(View.java:21614)
at android.view.ViewGroup.drawChild(ViewGroup.java:4558)
at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4333)
at android.view.View.updateDisplayListIfDirty(View.java:20747)
at android.view.View.draw(View.java:21614)
at android.view.ViewGroup.drawChild(ViewGroup.java:4558)
at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4333)
at android.view.View.updateDisplayListIfDirty(View.java:20747)
at android.view.View.draw(View.java:21614)
at android.view.ViewGroup.drawChild(ViewGroup.java:4558)
at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4333)
at android.view.View.draw(View.java:21891)
at com.android.internal.policy.DecorView.draw(DecorView.java:1082)
at android.view.View.updateDisplayListIfDirty(View.java:20761)
at android.view.ThreadedRenderer.updateViewTreeDisplayList(ThreadedRenderer.java:725)
at android.view.ThreadedRenderer.updateRootDisplayList(ThreadedRenderer.java:731)
at android.view.ThreadedRenderer.draw(ThreadedRenderer.java:840)
at android.view.ViewRootImpl.draw(ViewRootImpl.java:3935)
at android.view.ViewRootImpl.performDraw(ViewRootImpl.java:3709)
at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:3017)
at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1876)
at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:8499)
at android.view.Choreographer$CallbackRecord.run(Choreographer.java:949)
at android.view.Choreographer.doCallbacks(Choreographer.java:761)
at android.view.Choreographer.doFrame(Choreographer.java:696)
at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:935)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at android.app.ActivityThread.main(ActivityThread.java:7037)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:494)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:965)
2019-12-29 18:41:56.072 28172-28172/org.tensorflow.yolo E/AndroidRuntime: Caused by: java.io.IOException: Not a valid TensorFlow Graph serialization: NodeDef mentions attr 'explicit_paddings' not in Op<name=Conv2D; signature=input:T, filter:T -> output:T; attr=T:type,allowed=[DT_HALF, DT_BFLOAT16, DT_FLOAT, DT_DOUBLE]; attr=strides:list(int); attr=use_cudnn_on_gpu:bool,default=true; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW"]; attr=dilations:list(int),default=[1, 1, 1, 1]>; NodeDef: {{node 0-convolutional}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.loadGraph(TensorFlowInferenceInterface.java:561)
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.(TensorFlowInferenceInterface.java:105)
... 53 more

[ZZANZU]

did you actually uploaded .pb file into your asset folder?

[Kabster17]

Hi ZZANZU,
Yes, i did upload the .pb file into the asset folder but i dont get it why its not loading the model. I used tensorflow 1.14 gpu to train, not sure if there is a possible mismatch between the tensorflow dependencies and with my trained model. Could you share your experience if you uploaded your own custom model?

[szaza]

Hi @Kabster17,
It can be because of a mismatch between the tensorflow versions, so if you use tensorflow 1.14.0 for training and 1.6.0 on Android side, than you can have problems. Did you try to increase TF version in the mobile app from 'org.tensorflow:tensorflow-android:1.6.0'? As I see the highest version is 1.13.1, however it is still not 1.14.0, so I'm not sure if it will work for you. https://mvnrepository.com/artifact/org.tensorflow/tensorflow-android

As an alternative solution you can try to downgrade the TF version used for training to 1.6.0, or just to use the same version for training and on the mobile app.
I hope using the same version on both sides will solve your problem. If you still have problems please let me know an we will figure out something.

Best regards,
Zoltan

[Kabster17]

Hi Zoltan,

Thanks a lot for your reply man. Really appreciate it. Yeah i found out that i needed to downgrade to tf to 1.6.0 for the training and did some training with some dummy data for about 50 epochs after which I did manage to install/run the app without it crashing and no errors. However, it is unable to detect anything yet hopefully more training and lower loss rate would solve that.
I have a few questions regarding your application:

1. Is it possible to set a threshold value for the apk/app with regards to the detection of yolo model?
2. Have you ventured into the incorporating on image capturing and video recording as possible enhancements to your app?

[szaza]

Hi @Kabster17,
yes, usually 50 epochs are not enough, more training is needed.

1. yes it is possible in the classifier class itself: https://github.com/szaza/android-yolo-v2/blob/master/src/org/tensorflow/yolo/YOLOClassifier.java; You have to change the value of the THRESHOLD constant.
2. I did not considered video recording and image capturing yet, however it would be a great feature. Would you like to work on it? I would appreciate it.

[Kabster17]

Hi Zoltan @szaza ,

Thank you for your reply. Yes, i am looking into ways to enhance your application to perform image capturing and video recording with the detection bounding boxes being present. I think it would make the app more complete.
But I am really new to android studio and JAVA. Do you have any resources to recommend/suggest to have a better understanding of your app and help me kick start feature enhancements? Thanks.

[szaza]

Hi,
thanks for further development, I'm really glad when somebody adds values to the application.
I can only recommend you the official Android development guides + stack overflow: https://developer.android.com/guide. In my opinion that is the best combination.
Best regards,
Zoltan

[Kabster17]

Hi @szaza ,

Yeah sure. Thanks for your help.
I have an issue now, after training my model consisting of dataset of 2 images and 3 classes. I have brought the loss values to about 0.15.
Yet, i am unable to detect using the app after uploading my custom model.pb file into the assets folder.
I have changed the Threshold to a constant as shown:
private final static double THRESHOLD = 0.001;
from
private final static float THRESHOLD = 0.3f;

Is there something wrong with my edit or method of training? I am using the same images i used to train to test the app's detection with.

[szaza]

Hi,
I have no idea what could be wrong, however I've created a guideline for training YOLO. Could you please try to follow step-by-step this guideline, just to make sure that your installation works properly: https://sites.google.com/view/tensorflow-example-java-api/complete-guide-to-train-yolo?
Have you tested your generated pb file on desktop side? Does it recognize your images on the desktop environment?

[Kabster17]

Hi @szaza,

I have tested my generated pb file on desktop side, and it gives me no detection as well. It does not recognize the images. Like i have mentioned before, I have trained using Tiny-Yolo using tensorflow 1.14.0 version. It is a set of custom data but i dont understand using the pb files and meta files to perform detection is not occuring. Have you experienced this?

I trained for 5 classes using about 100 images per class to train for about 25000 steps but after generating the pb files using this command:
->python flow --model cfg/tiny-yolo-voc-5c.cfg --load -1 --savepb
and i used this command to test the generated pb file:
->python flow --pbLoad built_graph/tiny-yolo-voc-5c.pb --metaLoad built_graph/tiny-yolo-voc-5c.meta --imgdir sample_img/ --threshold 0.01 --gpu 1.0

Using the pb files, i am not able to get any detection but loading the images thru the checkpoint model, I am able to achieve good detection.
I am not sure why the pb files generated cannot give me any detection. Do you have any idea?

[szaza]

Hi,
I had similar behavior when I trained the model by using DarkNet and I tried to export the .pb file using DarkFlow. In my opinion the .pb generation is wrong. As I remember there was a darkflow bug.
I wrote a topic about it: https://sites.google.com/view/tensorflow-example-java-api/complete-guide-to-train-yolo/convert-darknet-weights-to-pb-file.
I'm not sure if it helps you, but it worth a try.

[Kabster17]

Hi @szaza,

I just started to read up on the darkflow bug with regards to the generation of the .pb files. Let me just check with you the following:

1. I trained my custom model with yolov2 weights instead of tiny-yolo-voc weights, is there a difference between them? I mean I am able to detect using the normal python flow method as mentioned earlier.
2. What is the issue with darkflow being unable to generate .pb files after 14000 steps in training? Is it true and whats the work around it?
   I have mentioned the link below that directs you to link:
   How to get a WORKING protobuf file (.pb) from a ckpt file? thtrieu/darkflow#698

[szaza]

Hi Kabster17,

"I trained my custom model with yolov2 weights instead of tiny-yolo-voc weights, is there a difference between them? I mean I am able to detect using the normal python flow method as mentioned earlier." - yes, that can be a problem. YoloV2 is a larger model than tiny-yolo, so easily can happen that you don't have enough memory in your mobile phone to execute it.

Is your installation able to work with the sample weights, that can be found here: https://github.com/szaza/android-yolo-v2/tree/master/assets?

[Kabster17]

Hi @szaza,

Yes i could detect and work with your sample .pb and labels.txt files.
Actually, i have gotten the app working and all but it is kinda detecting rubbish with the threshold being set is at 0.001 or no detection at all.
I trained a model to detect 2 classes with about 100 images in total and the 2 classes are mostly present together in an image. The loss rate went down to 1.03 at 14000 steps and the tiny-yolo model could detect using both the .pb file @Threshold=0.01 and with the checkpoint file with the same threshold value of 0.01.
But when i upload it to the assets folder to install the app, i get very poor detection results.
Can i know at what loss rate and steps did you train your model until and does the frame dimensions affect the detection results?
I tested the app to detect the exact images/dataset that i used to train the model but yet i am not unable to get any detection/bounding boxes.

[Kabster17]

Hi @szaza,

Yes i could detect and work with your sample .pb and labels.txt files.
Actually, i have gotten the app working and all but it is kinda detecting rubbish with the threshold being set is at 0.001 or no detection at all.
I trained a model to detect 2 classes with about 100 images in total and the 2 classes are mostly present together in an image. The loss rate went down to 1.03 at 14000 steps and the tiny-yolo model could detect using both the .pb file @Threshold=0.01 and with the checkpoint file with the same threshold value of 0.01.
But when i upload it to the assets folder to install the app, i get very poor detection results.
Can i know at what loss rate and steps did you train your model until and does the frame dimensions affect the detection results?
I tested the app to detect the exact images/dataset that i used to train the model but yet i am not unable to get any detection/bounding boxes.

Can you suggest to me what can i do to improve the detection results with regards to the training and what not. As I would like to get some sort of decent detection results to look further into improving the model or evaluating the model. Thanks.

[szaza]

Sorry for the late reply, I was very busy.
Did you manage to detect objects on images in the meantime?