Abstract
With the large-scale video-text datasets being collected, learning general visual-textual representation has gained increasing attention. While recent methods are designed with the assumption that the alt-text description naturally conveys the meaning and context of the video in semantics (i.e. well aligned with each other), it is unlikely to be satisfied for the Internet data, which potentially harms the quality of the learned visual-textual representation. To address this challenge, we first revisit three mainstream approaches: correspondence modeling, contrastive learning and predictive coding, demonstrating that a simple co-training strategy with these methods leads to a clear improvement in performance. To further explore the complementary nature of different training strategies, we propose a simple yet effective joint training framework that factorizes the total objective into conditional ones, termed as Cali-NCE 1. Our method first estimates confidence scores for measuring the correspondence between video and text descriptions, and the scores are later used to calibrate the sample weightings during contrastive training. Through extensive experiments, we show that the proposed approach achieves state-of-the-art performance on multiple downstream tasks: text-to-video retrieval, video action recognition, and video retrieval.
Original language | English |
---|---|
Title of host publication | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) |
Publisher | IEEE |
Pages | 6317-6327 |
Number of pages | 11 |
ISBN (Electronic) | 9798350302493 |
ISBN (Print) | 9798350302509 |
DOIs | |
Publication status | Published - 14 Aug 2023 |
Event | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition - Vancouver, Canada Duration: 18 Jun 2023 → 22 Jun 2023 |
Publication series
Name | IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops |
---|---|
Publisher | IEEE |
ISSN (Print) | 2160-7508 |
ISSN (Electronic) | 2160-7516 |
Conference
Conference | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition |
---|---|
Abbreviated title | CVPR 2023 |
Country/Territory | Canada |
City | Vancouver |
Period | 18/06/23 → 22/06/23 |
Bibliographical note
Publisher Copyright:© 2023 IEEE.
Keywords
- Training
- Weight measurement
- Representation learning
- Semantics
- Training data
- Predictive models
- Predictive coding
ASJC Scopus subject areas
- Computer Vision and Pattern Recognition
- Electrical and Electronic Engineering