13 accepted papers
High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-based aggregation face challenges of computational efficiency. We present LiteLong, a resource-efficient method for synth
Low-Rank Adaptation (LoRA) is the leading parameter-efficient fine-tuning method for Large Language Models (LLMs), but it still suffers from catastrophic forgetting. Recent work has shown that specialized LoRA initialization can alleviate catastrophic forgetting. There are currently two approaches t
The adaptation research of SAM in the field of medical image mainly adopts two types of methods: parameter fine-tuning and prompt engineering, but these methods face two main issues: (1)Parameter fine-tuning methods have limitations in focusing the model encoder’s attention on the foreground of medi…
Large Language Model-based Dense Retrieval (LLM-DR) optimizes over numerous heterogeneous fine-tuning collections from different domains. However, the discussion about its training data distribution is still minimal. Previous studies rely on empirically assigned dataset choices or sampling ratios, w…
The spatial and spectral resolution trade-off in the hyperspectral imaging is a fundamental and essential issue, and automatically generating high-resolution images in both spatial and spectral domains (HR-HS) by merging a low spatial resolution hyperspectral (LR-HS) image and a high spatial resolut…
There have been researches revealing obvious associations between facial phenotypes and non-communicable diseases (NCDs), which enables effective health assessment with the integration of model-based learning methods. However, the paucity and poor quality of available datasets hinder the development…
Bone marrow smear cytology involves observing and analyzing the morphological features of bone marrow cells, and identifying regions of interest (ROI) where the cells are morphologically clear and evenly distributed is a crucial part of this process. However, existing deep learning methods for selec…
Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent studies have explored improving pre-trained language models to boost dense retrieval performance. This paper proposes CoT…
Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as a minimal data augmentation method, and passes the same input sentence to a…
Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the origina…
Video language pre-training methods have mainly adopted sparse sampling techniques to alleviate the temporal redundancy of videos. Though effective, sparse sampling still suffers inter-modal redundancy: visual redundancy and textual redundancy. Compared with highly generalized text, sparsely sampled…
Unsupervised contrastive sentence embedding models, e.g., unsupervised SimCSE, use the InfoNCE loss function in training. Theoretically, we expect to use larger batches to get more adequate comparisons among samples and avoid overfitting. However, increasing batch size leads to performance degradati…
Before entering the neural network, a token needs to be converted to its one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained from the pre-trained masked language model, which can be seen as a more informa…