Contrastive Learning

Katharina Breininger

March 18, 2023

Materials and further reading

• Lecture Slides
• Chapter 15: Representation Learning (Goodfellow, Bengio, Courville: Deep Learning, MIT Press, 2016)
• Blog post on contrastive losses (nice visualizations)
• 15-minute summary of contrastive learning on YouTube
• Programming tutorial on SimCLR (Google Colab)

Selection of influential papers (* has been referenced in the lecture):

• Contrastive Loss: Chopra, S., Hadsell, R., & LeCun, Y. (2005). Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2005 (pp. 539-546). [pdf]
• Siamese Networks: Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., & Shah, R. (1994). Signature verification using a `siamese' time delay neural network. In Advances in Neural Information Processing Systems 6 (NIPS 1993) (pp. 737-744). [pdf]
• (*) Triplet Loss: Schroff, F., Kalenichenko, D., & Philbin, J. (2015). FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 (pp. 815-823).
• (*) SimCLR: Chen T., Kornblith S., Norouzi M., Hinton G., "A Simple Framework for Contrastive Learning of Visual Representations," In Proceedings of the 37th International Conference on Machine Learning (ICML 2020). [abs]
• MoCo: He K., Fan H., Wu Y., Xie S., Girshick R.; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9729-9738 [abs]
• Barlow Twins: Zbontar J., Jing L., Misra I., LeCun Y., Deny S., Barlow Twins: Self-Supervised Learning via Redundancy Reduction. Proceedings of the International Conference on Machine Learning (ICML), 2021.
• SimSiam: Chen X., He K., Exploring Simple Siamese Representation Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
• BYOL: Grill J.-B., et al., Bootstrap your own latent: A new approach to self-supervised Learning. Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
• Benchmarking 2022: Gwilliam M., Shrivastava A.: Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. [abs]