Abstract:
In this talk, I will mainly discuss two things we explored recently in the space of unsupervised visual representation learning. First is an approach called SimSiam, where we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. We find collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapse. Second, I will briefly talk about our recent experience on exploring recent self-supervised learning methods with a vision transformer backbone, mainly on the instability challenges we faced when we train self-supervised ViT. Feedbacks are welcome!
Short Bio:
Xinlei Chen is a research scientist working at Facebook AI Research since 2018. He obtained a Ph.D. from the school of computer science at Carnegie Mellon University, and before that, he obtained a Bachelor's degree from Zhejiang University, China. He is mainly interested in computer vision, natural language processing, and in general, machine learning.