Klöwer Milan, Razinger Miha, Dominguez Juan J, Düben Peter D, Palmer Tim N
Atmospheric, Oceanic and Planetary Physics, University of Oxford, Oxford, UK.
European Centre for Medium-Range Weather Forecasts, Reading, UK.
Nat Comput Sci. 2021 Nov;1(11):713-724. doi: 10.1038/s43588-021-00156-2. Epub 2021 Nov 25.
Hundreds of petabytes are produced annually at weather and climate forecast centers worldwide. Compression is essential to reduce storage and to facilitate data sharing. Current techniques do not distinguish the real from the false information in data, leaving the level of meaningful precision unassessed. Here we define the bitwise real information content from information theory for the Copernicus Atmospheric Monitoring Service (CAMS). Most variables contain fewer than 7 bits of real information per value and are highly compressible due to spatio-temporal correlation. Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself. All CAMS data are 17× compressed relative to 64-bit floats, while preserving 99% of real information. Combined with four-dimensional compression, factors beyond 60× are achieved. A data compression Turing test is proposed to optimize compressibility while minimizing information loss for the end use of weather and climate forecast data.
全球气象和气候预测中心每年会产生数百拍字节的数据。压缩对于减少存储量和促进数据共享至关重要。当前的技术无法区分数据中的真实信息和虚假信息，导致有意义的精度水平无法评估。在此，我们从信息论的角度为哥白尼大气监测服务（CAMS）定义了按位真实信息内容。由于时空相关性，大多数变量每个值所包含的真实信息少于7位，并且具有高度的可压缩性。将没有真实信息的位舍入为零有助于无损压缩算法，并对数据本身的不确定性进行编码。相对于64位浮点数，所有CAMS数据都实现了17倍的压缩，同时保留了99%的真实信息。结合四维压缩，可实现超过60倍的压缩率。我们提出了一种数据压缩图灵测试，以在优化可压缩性的同时，为气象和气候预测数据的最终使用将信息损失降至最低。
