• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>IEEE Transactions on Parallel and Distributed Systems >GraphD: Distributed Vertex-Centric Graph Processing Beyond the Memory Limit
【24h】

GraphD: Distributed Vertex-Centric Graph Processing Beyond the Memory Limit

机译：GraphD：超出内存限制的分布式顶点中心图处理

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

We propose GraphD, an out-of-core Pregel-like system targeting efficient big graph processing with a small cluster of commodity PCs connected by Gigabit Ethernet, an environment affordable to most users. This is in contrast to some recent efforts for out-of-core graph computation with specialized hardware. In our setting, a vertex-centric program is often data-intensive, since the CPU cost of calculating a message value is negligible compared with the network cost of transmitting that message. As a result, network bandwidth is usually the bottleneck, and out-of-core execution would not sacrifice performance if disk IO overhead can be hidden by message transmission, which is achieved by GraphD through the parallelism of computation and communication. GraphD streams edge and message data on local disks, and thus consumes negligible memory space. For a broad class of Pregel algorithms where message combiner is applicable, GraphD completely eliminates the need of any expensive external-memory join or group-by, which is required by existing systems such as Pregelix and Chaos. Extensive experiments show that GraphD beats existing out-of-core systems by orders of magnitude, and achieves comparable performance to in-memory systems running with adequate memory.
机译：我们提出了GraphD，这是一种类似于Pregel的核心系统，其目标是通过由千兆以太网连接的小型商用PC集群进行高效的大图形处理，这是大多数用户可以承受的环境。这与最近使用专用硬件进行核外图计算的一些努力形成对比。在我们的环境中，以顶点为中心的程序通常是数据密集型的，因为计算消息值的CPU成本与传输该消息的网络成本相比可以忽略不计。因此，网络带宽通常是瓶颈，如果可以通过消息传输隐藏磁盘IO开销，则核外执行不会牺牲性能，这是GraphD通过计算和通信的并行性实现的。 GraphD在本地磁盘上流送边缘和消息数据，因此消耗的内存空间可忽略不计。对于适用于消息组合器的广泛种类的Pregel算法，GraphD完全消除了诸如Pregelix和Chaos之类的现有系统所需的任何昂贵的外部内存联接或分组方式。大量的实验表明，GraphD击败了现有的核心外系统几个数量级，并获得了与具有足够内存的内存系统相媲美的性能。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号