High-fidelity scientific instruments and simulations are producing data at unprecedented volumes and rates, imposing substantial pressure on storage, transmission, and analysis. In order to alleviate the challenges brought to high-performance computing I/O and storage, compression techniques have been introduced in various scenarios, but understanding compression performance remains challenging due to the complex interactions among data characteristics and compressor-internal behaviors. In this paper, we analyze the internal behaviors of SZ, a representative prediction-based error-bounded lossy compressor, and identify representative compressor-related features for performance prediction. We then develop learning-based models to predict compression ratio and throughput, and further design a simplified prediction model using representative features. We compare the proposed method with existing sample-based and white-box prediction methods. The results show that compressor-internal features are important for performance prediction, but the best prediction method is scenario-dependent.
- Open Access
- Article
Compressor-Aware Feature Analysis and Learning-Based Performance Prediction for Scientific Data Compression
- Zhenlu Qin 1,
- Qirui Tian 2,
- Lei Wu 1,*,
- Jinzhen Wang 3,*
Author Information
Received: 24 May 2026 | Revised: 21 Aug 2026 | Accepted: 27 Aug 2026 | Published: 21 Sep 2026
Abstract
Keywords
high-performance computing | data compression | learning-based model | performance prediction
References
- 1.
Lindstrom, P. Fixed-Rate Compressed Floating-Point Arrays. IEEE Trans. Vis. Comput. Graph. 2014, 20, 2674–2683. https://doi.org/10.1109/tvcg.2014.2346458.
- 2.
Lu, B.; Li, Y.; Wang, J.; et al. ZFP-X: Efficient Embedded Coding for Accelerating Lossy Floating Point Compression. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS), St. Petersburg, FL, USA, 15–19 May 2023; pp. 1041–1050. https://doi.org/10.1109/ipdps54959.2023.00107.
- 3.
Di, S.; Cappello, F. Fast Error-Bounded Lossy HPC Data Compression with SZ. In Proceedings of the 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Chicago, IL, USA, 23–27 May 2016; pp. 730–739. https://doi.org/10.1109/ipdps.2016.11.
- 4.
Liang, X.; Di, S.; Tao, D.; et al. Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets. In Proceedings of the IEEE International Conference on Big Data, Seattle, WA, USA, 10–13 December 2018; pp. 438–447. https://doi.org/10.1109/bigdata.2018.8622520.
- 5.
Zhao, K.; Di, S.; Liang, X.; et al. Significantly Improving Lossy Compression for HPC Datasets with Second-Order Prediction and Parameter Optimization. In Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, Virtual, 23–26 June 2020; pp. 89–100.
- 6.
Zhao, K.; Di, S.; Dmitriev, M.; et al. Optimizing Error-Bounded Lossy Compression for Scientific Data by Dynamic Spline Interpolation. In Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece, 19–22 April 2021; pp. 1643–1654. https://doi.org/10.1109/icde51399.2021.00145.
- 7.
Liang, X.; Zhao, K.; Di, S.; et al. SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors. IEEE Trans. Big Data 2023, 9, 485–498. https://doi.org/10.1109/tbdata.2022.3201176.
- 8.
Ainsworth, M.; Tugluk, O.; Whitney, B.; et al. Multilevel Techniques for Compression and Reduction of Scientific Data—The Multivariate Case. SIAM J. Sci. Comput. 2019, 41, A1278–A1303. https://doi.org/10.1137/18m1166651.
- 9.
Liang, X.; Whitney, B.; Chen, J.; et al. MGARD+: Optimizing Multilevel Methods for Error-Bounded Scientific Data Reduction. IEEE Trans. Comput. 2022, 71, 1522–1536.
- 10.
Ainsworth, M.; Tugluk, O.; Whitney, B.; et al. Multilevel Techniques for Compression and Reduction of Scientific Data—Quantitative Control of Accuracy in Derived Quantities. SIAM J. Sci. Comput. 2019, 41, A2146–A2171. https://doi.org/10.1137/18m1208885.
- 11.
Di, S.; Liu, J.; Zhao, K.; et al. A Survey on Error-Bounded Lossy Compression for Scientific Datasets. ACM Comput. Surv. 2025, 57, 287.
- 12.
Claggett, S.; Azimi, S.; Burtscher, M. SPDP: An Automatically Synthesized Lossless Compression Algorithm for Floating-Point Data. In Proceedings of the 2018 Data Compression Conference, Snowbird, UT, USA, 27–30 March 2018; pp. 335–344. https://doi.org/10.1109/dcc.2018.00042.
- 13.
Tao, D.; Di, S.; Liang, X.; et al. Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP. IEEE Trans. Parallel Distrib. Syst. 2019, 30, 1857–1871. https://doi.org/10.1109/tpds.2019.2894404.
- 14.
Liang, X.; Di, S.; Li, S.; et al. Exploring Best Lossy Compression Strategy by Combining SZ with Spatiotemporal Decimation. In Proceedings of the 4th International Workshop on Data Reduction for Big Scientific Data, Dallas, TX, USA, 11 November 2018.
- 15.
Lu, T.; Liu, Q.; He, X. Understanding and Modeling Lossy Compression Schemes on HPC Scientific Data. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS), Vancouver, BC, Canada, 21–25 May 2018; pp. 348–357.
- 16.
Wang, J.; Liu, T.; Liu, Q.; et al. Compression Ratio Modeling and Estimation Across Error Bounds for Lossy Compression. IEEE Trans. Parallel Distrib. Syst. 2020, 31, 1621–1635. https://doi.org/10.1109/tpds.2019.2938503.
- 17.
Liu, T.; Wang, J.; Liu, Q. High-Ratio Lossy Compression: Exploring the Autoencoder to Compress Scientific Data. IEEE Trans. Big Data 2023, 9, 22–36.
- 18.
Liu, J.; Di, S.; Jin, S.; et al. SRN-SZ: Deep Learning-Based Scientific Error-Bounded Lossy Compression with Super-Resolution Neural Networks. arXiv 2023, arXiv:2309.04037.
- 19.
Liu, J.; Di, S.; Zhao, K.; et al. Exploring Autoencoder-Based Error-Bounded Compression for Scientific Data. In Proceedings of the 2021 IEEE International Conference on Cluster Computing (CLUSTER), Portland, OR, USA, 7–10 September 2021; pp. 294–306. https://doi.org/10.1109/cluster48925.2021.00034.
- 20.
Jia, W.; Hu, Z.; Liu, Y.; et al. NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression. In Proceedings of the 39th ACM International Conference on Supercomputing, Salt Lake City, UT, USA, 8–11 June 2025; pp. 26–42. https://doi.org/10.1145/3721145.3725763.
- 21.
Jin, S.; Di, S.; Liang, X.; et al. DeepSZ: A Novel Framework to Compress Deep Neural Networks by Using Error-Bounded Lossy Compression. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing, Phoenix, AZ, USA, 22–29 June 2019; pp. 159–170. https://doi.org/10.1145/3307681.3326608.
- 22.
Li, G.; Alhumaidi, M.; Skiadopoulos, S.; et al. Extreme Error-bounded Compression of Scientific Data via Temporal Graph Autoencoders. IEEE Trans. Knowl. Data Eng. 2026, 38, 3597–3610. https://doi.org/10.1109/tkde.2026.3681654.
- 23.
Li, X.; Lee, J.; Rangarajan, A.; et al. Foundation Model for Lossy Compression of Spatiotemporal Scientific Data. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 June 2025; pp. 368–380. https://doi.org/10.1007/978-981-96-8295-9_27.
- 24.
Collet, Y. Zstd GitHub Repository from Facebook. Available online: https://github.com/facebook/zstd (accessed on 28 April 2026).
- 25.
Alted, F. BLOSC GitHub Repository. 2009. Available online: https://github.com/Blosc/c-blosc (accessed on 28 April 2026).
- 26.
Deutsch, P. GZIP File Format Specification Version 4.3. 1996. Available online: https://www.rfc-editor.org/rfc/rfc1952 (accessed on 28 April 2026).
- 27.
Gailly, J.L. GZIP: The Data Compression Program. 2016. Available online: https://www.gnu.org/software/gzip/manual/gzip.pdf (accessed on 28 April 2026).
- 28.
Klus, L.; Klus, R.; Torres-Sospedra, J. EWOk: Towards Efficient Multidimensional Compression of Indoor Positioning Datasets. IEEE Trans. Mob. Comput. 2024, 23, 3589–3604. https://doi.org/10.1109/tmc.2023.3277333.
- 29.
Qin, Z.; Wang, J.; Liu, Q.; et al. Estimating Lossy Compressibility of Scientific Data Using Deep Neural Networks. IEEE Lett. Comput. Soc. 2020, 3, 5–8. https://doi.org/10.1109/locs.2020.2971940.
- 30.
Mumenin, K.M.; Dai, D.; Wang, J.; et al. QualityNet: Error-Bounded Lossy Compression Quality Prediction via Deep Surrogate. In Proceedings of the IEEE International Conference on Big Data (BigData), Washington, DC, USA, 15–18 December 2024; pp. 252–261.
- 31.
Zhang, Y.; Lin, H.; Sun, J.; et al. Learning to Predict Object-Wise Just Recognizable Distortion for Image and Video Compression. IEEE Trans. Multimed. 2024, 26, 5925–5938. https://doi.org/10.1109/tmm.2023.3340882.
- 32.
Li, C.; Yin, S.; Jia, C.; et al. Multirate Progressive Entropy Model for Learned Image Compression. IEEE Trans. Circuits Syst. Video Technol. 2024, 34, 7725–7741. https://doi.org/10.1109/tcsvt.2024.3376704.
- 33.
Zhang, J.; Chen, T.; Ding, D.; et al. Neural Compression System for Point Cloud Video Streaming. IEEE Trans. Image Process. 2025, 34, 8032–8045. https://doi.org/10.1109/tip.2025.3638126.
- 34.
Yang, Y.; Jiao, C.; Gao, X.; et al. Adaptive Volumetric Data Compression Based on Implicit Neural Representation. In Proceedings of the 17th International Symposium on Visual Information Communication and Interaction, Hsinchu, Taiwan, 11–13 December 2024; pp. 1–8. https://doi.org/10.1145/3678698.3678703.
- 35.
Han, J.; Zheng, H.; Bi, C. KD-INR: Time-Varying Volumetric Data Compression via Knowledge Distillation-Based Implicit Neural Representation. IEEE Trans. Vis. Comput. Graph. 2024, 30, 6826–6838. https://doi.org/10.1109/tvcg.2023.3345373.
- 36.
Bai, Y.; Zhao, Y.; Wang, K. Practical Lossless Volumetric Medical Image Compression via Tri-plane Context Tree Learning. IEEE Trans. Image Process. 2026, 35, 5938–5953.
- 37.
Li, Y.; Xia, M.; Liang, X.; et al. Preserving Discrete Morse–Smale Complexes in Error-Bounded Lossy Compression. IEEE Trans. Vis. Comput. Graph. 2026, 32, 6593–6609. https://doi.org/10.1109/tvcg.2026.3684385.
- 38.
Dai, W.; Fan, J.; Miao, Y.; et al. Deep Learning Model Compression with Rank Reduction in Tensor Decomposition. IEEE Trans. Neural Netw. Learn. Syst. 2025, 36, 1315–1328. https://doi.org/10.1109/tnnls.2023.3330542.
- 39.
Cappello, F.; Ainsworth, M. Scientific Data Reduction Benchmarks. 2018. Available online: https://sdrbench.github.io/ (accessed on 28 April 2026).
- 40.
Heaton, J. Introduction to Neural Networks with Java; Heaton Research, Inc.: St. Louis, MO, USA, 2008.
- 41.
Apley, D.W.; Zhu, J. Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. J. R. Stat. Soc. Ser. B Stat. Methodol. 2020, 82, 1059–1086. https://doi.org/10.1111/rssb.12377.
- 42.
Hinton, G.; Vinyals, O.; Dean, J. Distilling the Knowledge in a Neural Network. arXiv 2015, arXiv:1503.02531.
- 43.
Yang, J.; Martinez, B.; Bulat, A.; et al. Knowledge Distillation via Softmax Regression Representation Learning. In Proceedings of the International Conference on Learning Representations (ICLR), Virtual, 3–7 May 2021.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Suite 4002 Level 4, 447 Collins Street, Melbourne, Victoria 3000, Australia
General Inquiries: info@sciltp.com