• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文会议>IEEE International Parallel Distributed Processing Symposium >Model-Driven Data Layout Selection for Improving Read Performance
【24h】

Model-Driven Data Layout Selection for Improving Read Performance

机译：由模型驱动的数据布局选择，以提高读取性能

获取原文
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

Performance of reading scientific data from a parallel file system depends on the organization of data on physical storage devices. Data is often immutable after producers of data, such as large-scale simulations, experiments, and observations, write the data to the parallel file system. As a result, read performance of data analysis tasks is often slow when the read pattern does not conform with the original organization of the data. For example, reading small noncontiguous chunks of data from a large array is many times slower than reading the same size of contiguous chunks of data. Towards improving the data read performance during analysis phase, we are developing the Scientific Data Services (SDS) framework for automatically reorganizing previously written data to conform with the known read patterns. In this paper, we introduce a model-driven strategy for selecting the data layouts that benefit the performance of different read patterns. We have developed a parallel I/O model based on the striping parameters on Lustre file system and the block-level striping on RAID-based disks within an Object Storage Target (OST) of Lustre. We have applied the model to reorganize large 3D array datasets on a Cray XE6 platform and achieved 9X to 128X improvement in accessing the reorganized data compared to reading the data in its original layout.
机译：从并行文件系统读取科学数据的性能取决于物理存储设备上数据的组织。在大型模拟，实验和观测等数据生成者将数据写入并行文件系统之后，数据通常是不可变的。结果，当读取模式与数据的原始组织不一致时，数据分析任务的读取性能通常会很慢。例如，从大型数组中读取较小的非连续数据块比读取相同大小的连续数据块要慢许多倍。为了提高分析阶段的数据读取性能，我们正在开发科学数据服务（SDS）框架，该框架可自动重新组织先前写入的数据以符合已知的读取模式。在本文中，我们介绍了一种模型驱动的策略，用于选择有利于不同读取模式性能的数据布局。我们已经基于Lustre文件系统上的条带化参数以及Lustre的对象存储目标（OST）中基于RAID的磁盘上的块级条带化开发了并行I / O模型。我们已将该模型应用于在Cray XE6平台上重组大型3D数组数据集，与读取原始布局中的数据相比，在访问重组后的数据方面实现了9X到128X的改进。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号