12 accepted papers
While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to dis…
We present VideoPoet, a language model capable of synthesizing high-quality video from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that…
We introduce the MAsked Generative VIdeo Transformer, MAGVIT, to tackle various video synthesis tasks with a single model. We introduce a 3D tokenizer to quantize a video into spatial-temporal visual tokens and propose an embedding method for masked video token modeling to facilitate multi-task lear…
We present Mu$^2$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AST) and Machine Translation (MT), in over 100 languages. By leveraging a quantized…
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM…
Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing.…
Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels correspondi…
Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, F2-XEnDec, to combine self-supervised and…
Experimental Advanced Superconducting Tokamak (EAST) is the world's first fully superconducting tokamak fusion device with non-circular cross-section which was built in China The EAST articulated maintenance arm (EAMA) system is developed for real-time detection and rapid repair operations to damage…
Fast inspection and light maintenance capability is already a clear demand to control the tokamak condition and improve the efficiency of the experimental campaigns. EAST remote maintenance system has been developed to implement inspection and grasping tasks during plasma. The paper presents design…
We consider the problem of designing hybrid analog-digital beamformers in a downlink multi-user large-scale MIMO system. The objective is to minimize the total transmit power, while fulfilling SINR targets of all users. A dual virtual uplink problem is formulated for the original downlink problem ba…
In this paper, we investigate user association and resource block (RB) allocation in downlink heterogeneous cellular networks. Our goal is to jointly optimize user association and RB allocation to maximize network throughput while taking into account fairness among users. To effectively control inte…