Light-weight reference-based compression of FASTQ data

Research output: Contribution to journalArticlepeer-review

Authors

  • Yongpeng Zhang
  • Linsen Li
  • Yanli Yang
  • Xiao Yang
  • Zexuan Zhu

Colleges, School and Institutes

External organisations

  • College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China.
  • Broad Institute

Abstract

Background: The exponential growth of next generation sequencing (NGS) data has posed big challenges to data storage, management and archive. Data compression is one of the effective solutions, where reference-based compression strategies can typically achieve superior compression ratios compared to the ones not relying on any reference. Results: This paper presents a lossless light-weight reference-based compression algorithm namely LW-FQZip to compress FASTQ data. The three components of any given input, i.e., metadata, short reads and quality score strings, are first parsed into three data streams in which the redundancy information are identified and eliminated independently. Particularly, well-designed incremental and run-length-limited encoding schemes are utilized to compress the metadata and quality score streams, respectively. To handle the short reads, LW-FQZip uses a novel light-weight mapping model to fast map them against external reference sequence(s) and produce concise alignment results for storage. The three processed data streams are then packed together with some general purpose compression algorithms like LZMA. LW-FQZip was evaluated on eight real-world NGS data sets and achieved compression ratios in the range of 0.111-0.201. This is comparable or superior to other state-of-the-art lossless NGS data compression algorithms. Conclusions: LW-FQZip is a program that enables efficient lossless FASTQ data compression. It contributes to the state of art applications for NGS data storage and transmission. LW-FQZip is freely available online at: http://csse.szu.edu.cn/staff/zhuzx/LWFQZip.

Details

Original languageEnglish
Article number188
JournalBMC Bioinformatics
Volume16
Issue number1
Publication statusPublished - 9 Jun 2015