Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics

• Type of Material: Article
• Authors: Xiaoyuan Guo, Emory University; Jiali Duan, University of Southern California; C-C Jay Kuo, University of Southern California; Judy Gichoya, Emory University; Imon Banerjee, Arizona State University
• Language: English
• Date: 2022-11-29
• Publisher: IEEE
• Publication Version: Preprint (Prior to Peer Review)
• Copyright Statement: © 2022, IEEE
• Final Published Version (URL): https://ieeexplore.ieee.org/document/9956616
• Title of Journal or Parent Work: IEEE Xplore
• Keywords: visualization; vision modality fusion; masked image modeling; language modality; discretized visual semantics; semantics; unsupervised learning; image representation; language vocabulary; vision language pretraining augmentation; neural nets; joint learning; visual representation; self-supervised ground-truths; natural language processing; bridges; image fusion; benchmark testing; vocabulary; supervised learning; codebook learning; VQ-VAE; Representation learning; buildings; computer vision; language modality fusion; masked language modeling; semantic meaning
• Research Categories: Psychology, Developmental

Abstract

Language modality within the vision language pretraining framework is innately discretized, endowing each word in the language vocabulary a semantic meaning. In contrast, visual modality is inherently continuous and high-dimensional, which potentially prohibits the alignment as well as fusion between vision and language modalities. We therefore propose to "discretize" the visual representation by joint learning a codebook that imbues each visual token a semantic. We then utilize these discretized visual semantics as self-supervised ground-truths for building our Masked Image Modeling objective, a counterpart of Masked Language Modeling which proves successful for language models. To optimize the codebook, we extend the formulation of VQ-VAE which gives a theoretic guarantee. Experiments validate the effectiveness of our approach across common vision-language benchmarks.