CNN Triplet Tranining

[wangyida]

• 3D object recognition and face recognition samples
• 2 types of triplet loss for loss
• free number of sets for triplet loss

[wangyida]

@bhack @shelhamer @mtamburrano This is a newly rebased PR for triplet loss, I just added a face recognition sample in the examples for general using.
This is related to PR #2603

[shelhamer]

@wangyida when making changes please update your existing PR instead of opening a new PR. You can update a PR at any time by force pushing git push -f to your fork's branch. This makes it easier to track what is going on.

[bhack]

And I think you are still incorporating changes that you don't want to propose here. Review file changes of this PR. Do you really want this?

[bhack]

Probably you want to razionalize the two commits.

[wangyida]

Now two more changes in file mode is restored, I accidentally reset a commit version earlier than my first commit in the other PR, so I reopened here, it might also be able to reopen the disabled PR?

[bhack]

Ok changes seems isolated now. I think you can continue here. Next time close and reopen a new one only in particular cases.

[bhack]

Is this ready for a review pass now?

[wangyida]

@bhack Yes, sure~

[bhack]

@shelhamer Can you make a first quick pass?

[IMG-PRCSNG]

On using your triplet loss layer, I encountered an error which raised a SIGSEGV
src/caffe/layers/triplet_loss_layer.cu - Line number 36
loss += dist_sq_pos.gpu_data()[i];

The problem seems to be arising on using gpu_data. Was wondering if you have faced this problem before.
I will poke around a bit more and get back to you.
@wangyida

[wangyida]

@IMG-PRCSNG I have run the GPU codes on K40 and there seems no problem in the latest codes, I used to encounter some problems in the gpu_data of some metric operation functions, so I changed it to a cpu_data as the same data are stored also which could be used.

[IMG-PRCSNG]

Currently I am using only the CPU part of the code which doesn't throw any error. I find no reason why the GPU code should fail. I will try it out with cpu_data as you suggested. Thanks!

[swamiviv]

@wangyida Great job on the triplet loss. I am slightly confused though about how the filelists for the Image Data layer should be created. Can you provide an example ? Are the triplets generated online per batch ?

[wangyida]

@swamiviv As for the object pose related issue, you can have a look in the code in the training data generation code using leveldb, and for other free style using manner of training data, you can just have a reference on this one, it is the same algorithm on training samples arrangement style using filelist #2982

[swamiviv]

Thanks a lot!

On Thu, Oct 15, 2015 at 10:43 PM, Yida Wang notifications@github.com
wrote:

@swamiviv https://github.com/swamiviv As for the object pose related
issue, you can have a look in the code in the training data generation code
using leveldb, and for other free style using manner of training data, you
can just have a reference on this one, it is the same algorithm on training
samples arrangement style using filelist #2982
#2982

—
Reply to this email directly or view it on GitHub
#3123 (comment).

--Swami

[cysin]

HI Yida, I tried your triplet loss implementation but got errors when using GPU mode:

I1026 10:25:52.054105  3953 blocking_queue.cpp:50] Data layer prefetch queue empty
*** Aborted at 1445826352 (unix time) try "date -d @1445826352" if you are using GNU date ***
PC: @     0x7f1b9ca65237 caffe::TripletLossLayer<>::Forward_gpu()
*** SIGSEGV (@0x7011c5000) received by PID 3953 (TID 0x7f1b9cfe4a40) from PID 18632704; stack trace: ***
    @     0x7f1b9c1c5130 (unknown)
    @     0x7f1b9ca65237 caffe::TripletLossLayer<>::Forward_gpu()
    @     0x7f1b9c9db501 caffe::Net<>::ForwardFromTo()
    @     0x7f1b9c9db887 caffe::Net<>::ForwardPrefilled()
    @     0x7f1b9ca04fff caffe::Solver<>::Test()
    @     0x7f1b9ca058ae caffe::Solver<>::TestAll()
    @     0x7f1b9ca059e9 caffe::Solver<>::Step()
    @     0x7f1b9ca06335 caffe::Solver<>::Solve()
    @           0x40bbbb train()
    @           0x409481 main
    @     0x7f1b8fd4daf5 __libc_start_main
    @           0x409c1d (unknown)
Segmentation fault (core dumped)

[swamiviv]

I get the same error too. Was it resolved ?

On Sun, Oct 25, 2015 at 10:26 PM, cysin notifications@github.com wrote:

HI Yida, I tried your triplet loss implementation but got errors when
using GPU mode:

I1026 10:25:52.054105 3953 blocking_queue.cpp:50] Data layer prefetch queue empty
*** Aborted at 1445826352 (unix time) try "date -d @1445826352" if you are using GNU date ***
PC: @ 0x7f1b9ca65237 caffe::TripletLossLayer<>::Forward_gpu()
*** SIGSEGV (@0x7011c5000) received by PID 3953 (TID 0x7f1b9cfe4a40) from PID 18632704; stack trace: ***
@ 0x7f1b9c1c5130 (unknown)
@ 0x7f1b9ca65237 caffe::TripletLossLayer<>::Forward_gpu()
@ 0x7f1b9c9db501 caffe::Net<>::ForwardFromTo()
@ 0x7f1b9c9db887 caffe::Net<>::ForwardPrefilled()
@ 0x7f1b9ca04fff caffe::Solver<>::Test()
@ 0x7f1b9ca058ae caffe::Solver<>::TestAll()
@ 0x7f1b9ca059e9 caffe::Solver<>::Step()
@ 0x7f1b9ca06335 caffe::Solver<>::Solve()
@ 0x40bbbb train()
@ 0x409481 main
@ 0x7f1b8fd4daf5 __libc_start_main
@ 0x409c1d (unknown)
Segmentation fault (core dumped)

—
Reply to this email directly or view it on GitHub
#3123 (comment).

--Swami

[wangyida]

@cysin OK, Let me have a check on the GPU codes.

[bhack]

/cc @vpisarev for opencv/opencv_contrib#276

[mtamburrano]

I see that you try to access on cpu_data when you make the dot operation to calculate the loss.
I think that all the pointers should refer to gpu, I don't think that the underlying cblas function can handle mixed cpu and gpu pointers in that way

[wangyida]

@mtamburrano I encountered some GPU memory issues that time when I using the matrix dot function defined in CUDA, so I change to use cpu_data instead cause this function seems able to have a cpu_data backup at the same time when it is used.

[bhack]

I see that caffe_gpu_dot call cublasSdot or cublasDdot. How that could handle host pointers?

[wangyida]

@bhack OK, I will have a deeper check on it, I used to succeeded on a K80 server with this code and I find that caffe_gpu_dot is seldom used in all GPU codes of layers.

[bhack]

@wangyida Yes it is used but I cannot find any layer that use it with CPU pointers cause it is a wrapper call on cublas.

[igorbb]

Thanks @wangyida .

Reading previous comments and some of the code I think things are following into place...

So in theory one could just use the examples/imagenet/create_imagenet.sh to convert an image list to a LMDB, and if the list is like the following ( assuming 3 negative samples), it will be okay.

root_path_folder/anchor_A.jpg CLASS_A
root_path_folder/positve_A.jpg CLASS_A
root_path_folder/negative_A1.jpg CLASS_NOT_A
root_path_folder/negative_A2.jpg CLASS_NOT_A
root_path_folder/negative_A3.jpg CLASS_NOT_A
...
root_path_folder/anchor_B.jpg CLASS_B
root_path_folder/positve_B.jpg CLASS_B
root_path_folder/negative_B1.jpg CLASS_NOT_B
root_path_folder/negative_B2.jpg CLASS_NOT_B
root_path_folder/negative_B3.jpg CLASS_NOT_B
....

And I do need to be carefurell to set batch as a multiple of 5.
The class is not really important as Triplet Losst wont be used as label,

Are my intuitions correct ?

---

So it seems to be working with the solver_type = CPU, but whenever I use the GPU I also get a error in the forward pass.

*** Aborted at 1456679782 (unix time) try "date -d @1456679782" if you are using GNU date ***
PC: @     0x7fd9ff8d8517 caffe::TripletLossLayer<>::Forward_gpu()
*** SIGSEGV (@0x706cad000) received by PID 14782 (TID 0x7fd9ffe19a40) from PID 113954816; stack trace: ***
    @     0x7fd9fdf5bd40 (unknown)
    @     0x7fd9ff8d8517 caffe::TripletLossLayer<>::Forward_gpu()
    @     0x7fd9ff846ce1 caffe::Net<>::ForwardFromTo()
    @     0x7fd9ff847067 caffe::Net<>::ForwardPrefilled()
    @     0x7fd9ff8b3319 caffe::Solver<>::Step()
    @     0x7fd9ff8b3b65 caffe::Solver<>::Solve()
    @           0x40baeb train()
    @           0x409461 main
    @     0x7fd9fdf46ec5 (unknown)
    @           0x409bfb (unknown)
    @                0x0 (unknown)
Segmentation fault (core dumped)

[wangyida]

@igorbb I have tested my latest codes again on triplet loss type 0 and 1 for K40 GPU. There isn't such problem as you have described. Maybe you should pull the latest codes because I rebased all commits into a single commit.

[igorbb]

@wangyida I managed to get the previous commit running (with the labs Titan X) after some small modifications.
I will test the new commit Friday and let you know by the weekend.

[igorbb]

Hey @wangyida
Just tested https://github.com/Wangyida/caffe/tree/58dc67922be2052fa399ffe1a482ef43babce5fa . Now it is working fine. Maybe I had a pre-rebase version.

[wangyida]

@igorbb Good news, there are still some CPU operation in my GPU codes. Welcome anyone boosting the efficiency of it.

[wusongbeckham]

@wangyida
Dear Yida,
I defined a new loss function and when I see the source code of backward_cpu, you just use the caffe_cpu_axpby(). Could you help me to explain the details of this function?
Thanks in advance.
Song Wu

[wangyida]

@wusongbeckham This is a linear operation function dependent on cblas, you could just have a reference on cblas headers.

[wusongbeckham]

Hi, dear Yida,
Do you have the code which is about how to generate the
convert_minst_triplet_data.bin. I can not find the bin file.
Thanks.
Song Wu

On Wed, Mar 9, 2016 at 1:33 AM, Yida Wang notifications@github.com wrote:

@wusongbeckham https://github.com/wusongbeckham This is a linear
operation function dependent on cblas, you could just have a reference on
cblas headers.

—
Reply to this email directly or view it on GitHub
#3123 (comment).

[wusongbeckham]

@wangyida
Hi, dear Yida,
Do you have the code which is about how to generate the convert_minst_triplet_data.bin. I can not find the bin file.
Thanks.
Song Wu

[wangyida]

@wusongbeckham The previous codes are written for early type of triplet and you could have a reference on examples/triplet/convert_triplet_data.cpp

[wusongbeckham]

@wangyida
Hi, Yida,
During the training of triplet network on the MNIST in your code, the test loss is becoming bigger and bigger. Why? I think it should become smaller and smaller.
Thanks.
Song Wu

[wangyida]

@wusongbeckham Problem might be the fact that you are using a small batch size or the. The loss should be smaller in overall process. It also goes wrong if the samples are not properly arranged while you have selected too many negative samples. You can have a test on Losstype 0 while Losstype 1 and 2 are more efficient.

[benjaminplanche]

@wangyida
First of all, great job here! May I ask what is the status of this PR? There doesn't seem to be any activity around it now...

I just started digging into your implementation and example, but as @raingo and @igorbb suggested, it could be nice to have an end-to-end example, covering also the data preparation (a .pynb file is a good idea). Just to get started faster...

[d4nst]

@wangyida
Thanks for your contribution.

I am wondering if you or other people are doing online generation of triplets and how. It seems that this is a crucial step in the triplet learning process and I can't find any way to do this in Caffe.

[seanbell]

I think the code would be much more readable if this were a string or enum.

[bjutdy]

@wusongbeckham
I user convert_triplet_data.bin convert Minist to levedb , but it is fail ,can you give me the create_minist_triple.sh ? thank you

[bjutdy]

hi,Hi, Yida:
can you give us the url of the train dataset?

[wangyida]

@bjutdy OK, I will upload my rendered training data from ShapeNet for recognition in PASCAL3D+ in dropbox, you could download it.

[bjutdy]

@wangyida
thanks ! I want to test the triplet, I have minist only,but i can not user the caffe/build/examples/triple/convert_triplet_data.bin convert minist to leveldb , can you give us some suggestions ?

[vtantia]

Hi, @wangyida are you working on this? Thanks for your efforts. There should be a small change. In src/caffe/proto/caffe.proto the triplet_loss_param cannot be 143 as it has already been used for input_param. I have assigned it 146 and it is compiling fine.

[vtantia]

Also, @wangyida could you please share the dropbox link for the data?

[aizawak]

Amazing work @wangyida. I'd like to implement triplet loss on a data set of my own. How should I label my data? (I'm trying to convert the data myself and am wondering what format to use to identify positive, negative, anchor, etc.)

[aizawak]

Is anyone else returning the following failed test:

[----------] 2 tests from TripletLossLayerTest/0, where TypeParam = struct caffe::CPUDevice<float>
[ RUN      ] TripletLossLayerTest/0.TestForward
..\..\src\caffe\test\test_triplet_loss_layer.cpp(112): error: The difference between this->blob_top_loss_->cpu_data()[0] and loss is 1.0810106992721558, which exceeds 1e-6, where
this->blob_top_loss_->cpu_data()[0] evaluates to 0.98075640201568604,
loss evaluates to 2.0617671012878418, and
1e-6 evaluates to 9.9999999999999995e-007.
[  FAILED  ] TripletLossLayerTest/0.TestForward, where TypeParam = struct caffe::CPUDevice<float> (0 ms)
[ RUN      ] TripletLossLayerTest/0.TestGradient
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.077058762311935425, which exceeds threshold_ * scale, where
computed_gradient evaluates to -0.43571779131889343,
estimated_gradient evaluates to -0.35865902900695801, and
threshold_ * scale evaluates to 0.0099999997764825821.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,35; feat = 0.52477920055389404; objective+ = 1.8039973974227905; objective- = 1.8111705780029297
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.038115590810775757, which exceeds threshold_ * scale, where
computed_gradient evaluates to 0.17970642447471619,
estimated_gradient evaluates to 0.14159083366394043, and
threshold_ * scale evaluates to 0.0099999997764825821.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,37; feat = -0.57654058933258057; objective+ = 1.8086471557617187; objective- = 1.8058153390884399
[  FAILED  ] TripletLossLayerTest/0.TestGradient, where TypeParam = struct caffe::CPUDevice<float> (1 ms)
[----------] 2 tests from TripletLossLayerTest/0 (2 ms total)

[----------] 2 tests from TripletLossLayerTest/1, where TypeParam = struct caffe::CPUDevice<double>
[ RUN      ] TripletLossLayerTest/1.TestForward
..\..\src\caffe\test\test_triplet_loss_layer.cpp(112): error: The difference between this->blob_top_loss_->cpu_data()[0] and loss is 1.0810105808020773, which exceeds 1e-6, where
this->blob_top_loss_->cpu_data()[0] evaluates to 0.98075638929016384,
loss evaluates to 2.0617669700922412, and
1e-6 evaluates to 9.9999999999999995e-007.
[  FAILED  ] TripletLossLayerTest/1.TestForward, where TypeParam = struct caffe::CPUDevice<double> (1 ms)
[ RUN      ] TripletLossLayerTest/1.TestGradient
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.077061012824835573, which exceeds threshold_ * scale, where
computed_gradient evaluates to -0.43571781438032586,
estimated_gradient evaluates to -0.35865680155549029, and
threshold_ * scale evaluates to 0.01.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,35; feat = 0.52477923920378089; objective+ = 1.8039973966033025; objective- = 1.8111705326344123
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.038117944273945215, which exceeds threshold_ * scale, where
computed_gradient evaluates to 0.17970644072077691,
estimated_gradient evaluates to 0.14158849644683169, and
threshold_ * scale evaluates to 0.01.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,37; feat = -0.57654061773791909; objective+ = 1.808647101821613; objective- = 1.8058153318926764
[  FAILED  ] TripletLossLayerTest/1.TestGradient, where TypeParam = struct caffe::CPUDevice<double> (1 ms)
[----------] 2 tests from TripletLossLayerTest/1 (3 ms total)

[----------] 2 tests from TripletLossLayerTest/2, where TypeParam = struct caffe::GPUDevice<float>
[ RUN      ] TripletLossLayerTest/2.TestForward
..\..\src\caffe\test\test_triplet_loss_layer.cpp(112): error: The difference between this->blob_top_loss_->cpu_data()[0] and loss is 1.0810106992721558, which exceeds 1e-6, where
this->blob_top_loss_->cpu_data()[0] evaluates to 0.98075640201568604,
loss evaluates to 2.0617671012878418, and
1e-6 evaluates to 9.9999999999999995e-007.
[  FAILED  ] TripletLossLayerTest/2.TestForward, where TypeParam = struct caffe::GPUDevice<float> (14 ms)
[ RUN      ] TripletLossLayerTest/2.TestGradient
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.077058762311935425, which exceeds threshold_ * scale, where
computed_gradient evaluates to -0.43571779131889343,
estimated_gradient evaluates to -0.35865902900695801, and
threshold_ * scale evaluates to 0.0099999997764825821.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,35; feat = 0.52477920055389404; objective+ = 1.8039973974227905; objective- = 1.8111705780029297
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.038115590810775757, which exceeds threshold_ * scale, where
computed_gradient evaluates to 0.17970642447471619,
estimated_gradient evaluates to 0.14159083366394043, and
threshold_ * scale evaluates to 0.0099999997764825821.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,37; feat = -0.57654058933258057; objective+ = 1.8086471557617187; objective- = 1.8058153390884399
[  FAILED  ] TripletLossLayerTest/2.TestGradient, where TypeParam = struct caffe::GPUDevice<float> (593 ms)
[----------] 2 tests from TripletLossLayerTest/2 (608 ms total)

[----------] 2 tests from TripletLossLayerTest/3, where TypeParam = struct caffe::GPUDevice<double>
[ RUN      ] TripletLossLayerTest/3.TestForward
..\..\src\caffe\test\test_triplet_loss_layer.cpp(112): error: The difference between this->blob_top_loss_->cpu_data()[0] and loss is 1.0810105808020773, which exceeds 1e-6, where
this->blob_top_loss_->cpu_data()[0] evaluates to 0.98075638929016384,
loss evaluates to 2.0617669700922412, and
1e-6 evaluates to 9.9999999999999995e-007.
[  FAILED  ] TripletLossLayerTest/3.TestForward, where TypeParam = struct caffe::GPUDevice<double> (7 ms)
[ RUN      ] TripletLossLayerTest/3.TestGradient
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.077061012824835573, which exceeds threshold_ * scale, where
computed_gradient evaluates to -0.43571781438032586,
estimated_gradient evaluates to -0.35865680155549029, and
threshold_ * scale evaluates to 0.01.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,35; feat = 0.52477923920378089; objective+ = 1.8039973966033025; objective- = 1.8111705326344123
C:\caffe\include\caffe/test/test_gradient_check_util.hpp(175): error: The difference between computed_gradient and estimated_gradient is 0.038117944273945215, which exceeds threshold_ * scale, where
computed_gradient evaluates to 0.17970644072077691,
estimated_gradient evaluates to 0.14158849644683169, and
threshold_ * scale evaluates to 0.01.
debug: (top_id, top_data_id, blob_id, feat_id)=0,0,0,37; feat = -0.57654061773791909; objective+ = 1.808647101821613; objective- = 1.8058153318926764
[  FAILED  ] TripletLossLayerTest/3.TestGradient, where TypeParam = struct caffe::GPUDevice<double> (592 ms)
[----------] 2 tests from TripletLossLayerTest/3 (600 ms total)

[aizawak]

1. Missing .cpp file extension
2. Is '#include "caffe/vision_layers.hpp"' needed?
3. Missing '#include "caffe/layers/triplet_loss_layer.hpp"'

[kwin-wang]

@wangyida I don't really understand this problem. Could you describe this problem in more detail?

@Strateus triplet loss could surely do this, but there might be a problem: a single pair wise term is joint together with several triplet sets, so your reference sample and positive samples should not differs too much. You could just remove the pair wise term in the triplet loss if the loss don't decrease as expected.