Czech Eric, Millar Timothy R, Tyler Will, White Tom, Elsworth Benjamin, Guez Jérémy, Hancox Jonny, Jeffery Ben, Karczewski Konrad J, Miles Alistair, Tallman Sam, Unneberg Per, Wojdyla Rafal, Zabad Shadi, Hammerbacher Jeff, Kelleher Jerome
Open Athena AI Foundation, Lincoln, New Zealand.
Related Sciences, Lincoln, New Zealand.
bioRxiv. 2025 Feb 6:2024.06.11.598241. doi: 10.1101/2024.06.11.598241.
Variant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasises efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. Biobank scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed.
Zarr is a format for storing multi-dimensional data that is widely used across the sciences, and is ideally suited to massively parallel processing. We present the VCF Zarr specification, an encoding of the VCF data model using Zarr, along with fundamental software infrastructure for efficient and reliable conversion at scale. We show how this format is far more efficient than standard VCF based approaches, and competitive with specialised methods for storing genotype data in terms of compression ratios and single-threaded calculation performance. We present case studies on subsets of three large human datasets (Genomics England: =78,195; Our Future Health: =651,050; All of Us: =245,394) along with whole genome datasets for Norway Spruce (=1,063) and SARS-CoV-2 (=4,484,157). We demonstrate the potential for VCF Zarr to enable a new generation of high-performance and cost-effective applications via illustrative examples using cloud computing and GPUs.
Large row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely-used, open-source technologies has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores, while maintaining compatibility with existing file-oriented workflows.
变异调用格式（VCF）是用于交换遗传变异数据及相关质量控制指标的标准文件格式。VCF数据模型通常的逐行编码（无论是文本形式还是打包二进制形式）强调了对给定变异的所有数据进行高效检索，但按字段或样本进行数据访问效率低下。目前可用的生物样本库规模的数据集包含数十万份全基因组数据以及数百太字节的压缩VCF文件。逐行数据存储从根本上说是不合适的，需要一种更具扩展性的方法。
Zarr是一种用于存储多维数据的格式，在各学科中广泛使用，非常适合大规模并行处理。我们提出了VCF Zarr规范，即使用Zarr对VCF数据模型进行的一种编码，以及用于大规模高效可靠转换的基础软件基础设施。我们展示了这种格式如何比基于标准VCF的方法高效得多，并且在压缩率和单线程计算性能方面与存储基因型数据的专门方法具有竞争力。我们给出了关于三个大型人类数据集（英国基因组学：=78,195；我们的未来健康：=651,050；全民健康研究：=245,394）的子集以及挪威云杉（=1,063）和新冠病毒（=4,484,157）全基因组数据集的案例研究。我们通过使用云计算和GPU的示例展示了VCF Zarr实现新一代高性能且经济高效应用的潜力。
大型逐行编码的VCF文件是当前研究的主要瓶颈，存储和处理这些文件会产生巨大成本。基于广泛使用的开源技术的VCF Zarr规范有潜力大幅降低这些成本，并可能催生一个多样化的下一代工具生态系统，可直接从基于云的对象存储中分析遗传变异数据，同时保持与现有面向文件的工作流程的兼容性。
