Multi-source Data Lossless Compression via Parallel Expansion Mapping and xLSTM

Abstract

Explosive growth of multi-source data (MSD) poses challenges in data transmitting and storing. Neural Network (NN)-based lossless compressors are an important type of compression approaches to alleviate these problems. However, existing NN-based lossless compressors suffer from poor compression ratio and high time cost at the same time. To address these issues, we propose a novel MSD Lossless Compressor (MSDLC) with two compression stages: 1) We propose a Parallel Expansion Mapper (PEM) to map redundant pieces in MSD into unused alphabet values, which not only compresses MSD but also saves time for the next stage’s NN-based lossless compression. 2) With the mapped MSD as input, we design a NN-based lossless compressor to further improve compression ratio, where we introduce the state-of-the-art xLSTM model and design a Deep Spatial Gating Module (DSGM) as the backbone of NN. We compare MSDLC with 11 baselines on 6 real-world datasets and the results validate that MSDLC obtains the best average compression ratio and time cost. Compared with baselines, compression ratios are improved by 1.103%~113.897%, and the time costs are improved by 41.367%~73.891%. The codes can be available at https://github.com/mhuidong/MSDLC.