A Compatible LZMA ORC-Based Optimization for High Performance Big Data Load

机译：基于LZMA ORC的兼容优化可实现高性能大数据加载

获取原文

页面导航

摘要
著录项
相似文献
相关主题

摘要

This paper presents several efficient ways to improve data loading and storage optimization in Hadoop cluster. We design a new method to leverage LZMA and ORC to gain performance edge, also improve ORC implementation in HDFS to have a higher compression ratio and better IO throughput. A complete optimization strategy for efficient big data loading, including byte array-oriented, record split, less serialization and shuffle, reducing middle data landing to earn great performance boost is presented. This paper provides preliminary results and analytics. Evaluation results indicate that our method achieves significant performance improvement for big data load.

机译：本文提出了几种有效的方法来改善Hadoop集群中的数据加载和存储优化。我们设计了一种利用LZMA和ORC来获得性能优势的新方法，还改进了HDFS中的ORC实施，以具有更高的压缩率和更好的IO吞吐量。提出了一种用于高效大数据加载的完整优化策略，包括面向字节数组，记录拆分，较少的序列化和混洗，减少中间数据着陆以提高性能。本文提供了初步结果和分析。评估结果表明，对于大数据负载，我们的方法可以显着提高性能。

著录项

来源
《IEEE International Congress on Big Data》|2014年|80-87|共8页
会议地点
作者
Zhang Liping; Chen Qi; Miao Kai;
展开▼
作者单位

展开▼
会议组织
原文格式 PDF
正文语种
中图分类
关键词
Arrays; Big data; Dictionaries; Loading; Optimization; Sorting; Throughput; HBase; Hadoop; I/O compression; LZMA; ORC; bulk load; performance optimization;

机译：数组;大数据;字典;加载;优化;排序;吞吐量; HBase; Hadoop; I / O压缩LZMA; ORC;散货性能优化;

相似文献

外文文献
中文文献
专利

1. Enhanced performances of ORC-based units for low grade waste heat recovery via evaporator layout optimization [J] . Vittorini Diego, Cipollone Roberto, Carapellucci Roberto Energy Conversion & Management . 2019,第Octa期

机译：通过优化蒸发器布局，提高基于ORC的单元的性能，可用于低品位废热回收
2. System performance optimization of ORC-based geo-plant with R245fa under different geothermal water inlet temperatures [J] . Liu Xinghua, Zhang Yufeng, Shen Jiang Geothermics . 2017,第mara期

机译：R245fa在不同地热水入口温度下基于ORC的地质植物的系统性能优化
3. Evaluation Performance of Hybrid Localized Multi Kernel SVR (LMKSVR) in Electrical Load Data Using 4 Different Optimizations [J] . Rezzy Eko Caraka, Sakhinah Abu Bakar Journal of Engineering & Applied Sciences . 2018,第17期

机译：使用4个不同优化的电负载数据中混合局部多核SVR（LMKSVR）的评估性能
4. A Compatible LZMA ORC-Based Optimization for High Performance Big Data Load [C] . Zhang Liping, Chen Qi, Miao Kai IEEE International Congress on Big Data . 2014

机译：基于LZMA的高性能大数据负载的兼容LZMA ORC的优化
5. SIMPLEX OPTIMIZATION IN GRADIENT ELUTION HIGH PERFORMANCE LIQUID CHROMATOGRAPHY. THE PREPARATION AND CHARACTERIZATION OF BIO-COMPATIBLE HIGH PERFORMANCE LIQUID CHROMATOGRAPHY PACKING MATERIALS [D] . WATSON, MARK WAINWRIGHT 1981

机译：梯度洗脱中的简单优化高效液相色谱法。生物相容性高性能液相色谱包装材料的制备与表征
6. Respiratory Effects of Thoracic Load Carriage Exercise and Inspiratory Muscle Training as a Strategy to Optimize Respiratory Muscle Performance with Load Carriage [O] . Ren-Jay Shei, Robert F. Chapman, Allison H. Gruber, -1

机译：胸负重运动的呼吸作用和吸气肌肉训练作为优化负重运动的呼吸肌性能的策略
7. The Study on Work Load Calibration by Using Step Test and Ergometer Test Was Carried Out in Order to Find the Best Way 10 Calibrate the Data Measurement and to Predict Human Work Load by Using Heart Rate Data. This Study Was Conducted on Four Male Subjects. But the Data Analysis Has Just Been Done to Three Subjects Due One of the Subject Was Insufficient Performance. the Sport Tester PE3000 Heart Rate Monitor is Used for Measuring the Heart Rate Data Which Will Be Useful for Predicting the Work [O] . Herodian, Sam, Kastaman, Roni 1998

机译：为了找到最佳方法10校准数据测量并使用心率数据预测人类工作负荷，进行了使用阶跃测试和测力计测试进行工作负荷校准的研究。这项研究是针对四个男性受试者进行的。但是，由于一个主题的性能不足，因此数据分析只针对三个主题。 Sport Tester PE3000心率监测器用于测量心率数据，这对预测工作很有用

A Compatible LZMA ORC-Based Optimization for High Performance Big Data Load

摘要

著录项

相似文献

相关主题

期刊订阅