Custom Trained YOLOv3 Tiny Model not detecting anything

[DaphiFluffi]

Hi again,

I wanted to open up a new issue about this. I trained a custom model using Darknet. It is trained to detect only one class, a pillbox. And when tested in Darknet, it did. I trained the model for 4000 iterations.
Then I converted it to Tensorflow, froze the graph and converted it to ONNX successfully. Then I was able to import it into Unity.
I adjusted the anchor values to my own and filled out the Detector in the Inspector. But, unfortunately, nothing is detected when I build the app on Android. But there are also no errors.
What could the mistake be? Could it have something to do with the input shape?

Any help is very apprechiated :)

[Ademord]

Same for me I dont detect anything when I imported the yolov3 model linked in this repo.

[DaphiFluffi]

the default one works for me, but not my custom one

[Ademord]

did you integrate the detector into a new code? if you could help me understand how do i reuse it in another class? (inputs and format, outputs and format?)

[DaphiFluffi]

no not really. I am just using the ARAnchorManager's Prefab to instantiate my own objects on top of the detected object. I only reverse engineered that part.

[Ademord]

ah so you are doing some variant of the face-filter scenario... i see. i am trying to understand how to use any kind of yolo (onnx) and pass it the camera input (render texture), and then output the class and bbox. i got the code from https://github.com/keijiro/TinyYOLOv2Barracuda working but now I need to upgrade it to yolov3 and idk how to (yet).

[DaphiFluffi]

I don't do anything with face decetion. And if everything you want is to detect an object from your phone's current camera feed, this is exactly what this repository does. And it works with Yolo v3 for me as well. Do you have Errors? Did you export it to Android successfully? Which Objects did you test with?

[Ademord]

i am trying to integrate yolov3 using barracuda into a game's camera feed, not using a phone or a webcam. most repos out there using barracuda (not many in general), are using phone or PC cameras...

[DaphiFluffi]

Ok, i don't know much about that. But you should read into how the AR Camera in the AR Session Origin captures the Phone Camera's Feed. I think it would be possible to change it out quite easily.

[Kanyade]

Hi @DaphiFluffi @derenlei @ROBYER1 @artoonie @BeiieB
Is there any news on this? Have you solved it? I am in the exact same boat. The model works elsewhere, just not within Unity.

The main issue I think is the fact that the original yolo v3 model in the repo has output_l(conv16) and output_m(conv23) values and our custom model has one output.

This is the error I could extract from adb debugger.

Any tips would be appreciated.

[Kanyade]

I managed to have two outputs by changing the tensorflow freeze graph function output node names to yolov3-tiny/convolutional10/BiasAdd,yolov3-tiny/convolutional13/BiasAdd and setting this in tf2onnx as well. I did it because at the end of the DW2TF conversion script it shows the output layers:

I did the conversion by your guide @DaphiFluffi https://github.com/DaphiFluffi/Yolov3-To-Onnx-Unity that's why we have really similar results.

And also a weird thing to note with the onnx formats is that the outputs' shape changes to -1 -1 -1 -1 but in Netron it shows like this:

Could it be because of the unknown value?(First in outputs) Any idea how to solve it?

[Ademord]

I managed to have two outputs by changing the tensorflow freeze graph function output node names to yolov3-tiny/convolutional10/BiasAdd,yolov3-tiny/convolutional13/BiasAdd and setting this in tf2onnx as well. I did it because at the end of the DW2TF conversion script it shows the output layers:

I did the conversion by your guide @DaphiFluffi https://github.com/DaphiFluffi/Yolov3-To-Onnx-Unity that's why we have really similar results.

And also a weird thing to note with the onnx formats is that the outputs' shape changes to -1 -1 -1 -1 but in Netron it shows like this:

Could it be because of the unknown value?(First in outputs) Any idea how to solve it?

Hey, just to clarify which yolov3 model did you use and what command from the source you mentioned? If you could share the command you used to replicate? And in Unity to load the model did you change the parsing of the outputs? Would you have a very small project to make replicate it at best?

Hi @michalek-marcin ,

Did you freeze the tf graph to convert model parameters into constants? You may need tf weights (.ckpt, you can get it here: DW2TF) and the tf graph (.pb). Somthing like this:
python3 -m tensorflow.python.tools.freeze_graph --input_graph=./checkpoint_dir/yolov2-tiny.pb --input_binary=true --output_node_names=yolov2-tiny/convolutional9/BiasAdd --input_checkpoint=./checkpoint_dir/yolov2-tiny.ckpt --output_graph=./checkpoint_dir/yolov2-tiny-freeze.pb

I then convert the frozen graph to onnx through:
python3 -m tf2onnx.convert --graphdef ./yolov2-tiny-freeze.pb --output yolov2-tiny.onnx --inputs yolov2-tiny/net1:0 --outputs yolov2-tiny/convolutional9/BiasAdd:0

Make sure you use the correct graph inputs and outputs.

[Kanyade]

@Ademord
it was a custom model, not the one in the repo. I followed @DaphiFluffi 's guide which I referenced with these modifications to have two outputs:

But I still don't know how to get it to work inside Unity, maybe I'll try to test the onnx model itself outside Unity to see if the problem is with the mode itself

[Ademord]

@Kanyade
thank you so much! and to integrate this into the render did you change any parts of the code? im struggling with this integration for a while now

[Kanyade]

@Ademord

I did not change anything, if you download the original repo then it should work fine with the correct Asset versions with the original onnx model.

I completely aborted the custom model I did while writing the previous comments as the output shape was invalid and the conversions must have gone wrong.

Here I solved the custom model problem, I'll post it so others can benefit from it as well:

https://gist.github.com/Kanyade/c8ee3b7eca6f6e470e418b018ea02e83

Special thanks to @jkjung-avt for this repo: https://github.com/jkjung-avt/tensorrt_demos#yolov3
and to @prratadiya for this repo: https://github.com/prratadiya/tensorrt-installation-colab

It works in the project now, it is the same logic behind the conversion as how the default model was converted -- with TensorRT --.

The only problem is that now the last shape of the output values is reduced compared to the default onnx model in the demo and that the detection boxes are bigger than my mobile screen. I think there is a correlation but I did not dive deep into it yet.

The yolov3-tiny-416 is the original from the repo and the yolov3-tiny is mine

[Ademord]

I am using the yolo model and on linux my build is giving me an error I have document on this issue https://forum.unity.com/threads/urgent-shader-loading-problem.1194259/
Could anyone help me here?