Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files

X Delaunay, A Courtois… - Geoscientific Model …, 2019 - gmd.copernicus.org
The increasing volume of scientific datasets requires the use of compression to reduce data
storage and transmission costs, especially for the oceanographic or meteorological datasets …

Bit Grooming: statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4. 4.8+)

CS Zender - Geoscientific Model Development, 2016 - gmd.copernicus.org
Geoscientific models and measurements generate false precision (scientifically
meaningless data bits) that wastes storage space. False precision can mislead (by implying …

Toward a multi-method approach: Lossy data compression for climate simulation data

AH Baker, H Xu, DM Hammerling, S Li… - … Computing: ISC High …, 2017 - Springer
Abstract Earth System Model (ESM) simulations are increasingly constrained by the amount
of data that they generate rather than by computational resources. The use of lossy data …

A collaborative effort to improve lossy compression methods for climate data

DM Hammerling, AH Baker, A Pinard… - 2019 IEEE/ACM 5th …, 2019 - ieeexplore.ieee.org
Climate model simulations produce large volumes of data, and reducing the storage burden
with data compression is increasingly of interest to climate scientists. A key concern to the …

[PDF][PDF] Error distributions of lossy floating-point compressors

P Lindstrom - 2017 - osti.gov
With ever increasing volumes of data being generated in scientific simulations, experiments,
and observations, storage and bandwidth concerns are mounting. As a means of data …

Toward decoupling the selection of compression algorithms from quality constraints

J Kunkel, A Novikova, E Betke, A Schaare - High Performance Computing …, 2017 - Springer
Data intense scientific domains use data compression to reduce the storage space needed.
Lossless data compression preserves the original information accurately but on the domain …

Implementation of CCSDS lossless data compression in HDF

PS Yeh, W Xia-Serafino, L Miles, B Kobler… - Earth Science …, 2002 - ntrs.nasa.gov
The Earth Science Data and Information System (ESDIS) handles over one terabyte (10 (exp
12) bytes) of data daily and is using the Hierarchical Data Format (EDF) for data archiving …

SPDP: An automatically synthesized lossless compression algorithm for floating-point data

S Claggett, S Azimi, M Burtscher - 2018 data compression …, 2018 - ieeexplore.ieee.org
Scientific computing produces, transfers, and stores massive amounts of single-and double-
precision floating-point data, making this a domain that can greatly benefit from data …

Using standard image compression algorithms to store data from computational fluid dynamics

J Schmalzl - Computers & geosciences, 2003 - Elsevier
Three-dimensional numerical modeling of fluid flows is an important research tool to
understand many fluid dynamical effects observed in nature. With the strong growth of …

Evaluating lossy compression on climate data

N Hübbe, A Wegener, JM Kunkel, Y Ling… - … Conference, ISC 2013 …, 2013 - Springer
While the amount of data used by today's high-performance computing (HPC) codes is
huge, HPC users have not broadly adopted data compression techniques, apparently …