...
  • 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>Journal of supercomputing >HFIM: a Spark-based hybrid frequent itemset mining algorithm for big data processing
【24h】

HFIM: a Spark-based hybrid frequent itemset mining algorithm for big data processing

机译：HFIM：一种基于Spark的混合频繁项集挖掘算法，用于大数据处理

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

Frequent itemset mining is one of the data mining techniques applied to discover frequent patterns, used in prediction, association rule mining, classification, etc. Apriori algorithm is an iterative algorithm, which is used to find frequent itemsets from transactional dataset. It scans complete dataset in each iteration to generate the large frequent itemsets of different cardinality, which seems better for small data but not feasible for big data. The MapReduce framework provides the distributed environment to run the Apriori on big transactional data. However, MapReduce is not suitable for iterative process and declines the performance. We introduce a novel algorithm named Hybrid Frequent Itemset Mining (HFIM), which utilizes the vertical layout of dataset to solve the problem of scanning the dataset in each iteration. Vertical dataset carries information to find support of each itemsets. Moreover, we also include some enhancements to reduce number of candidate itemsets. The proposed algorithm is implemented over Spark framework, which incorporates the concept of resilient distributed datasets and performs in-memory processing to optimize the execution time of operation. We compare the performance of HFIM with another Spark-based implementation of Apriori algorithm for various datasets. Experimental results show that the HFIM performs better in terms of execution time and space consumption.
机译：频繁项集挖掘是用于发现频繁模式的数据挖掘技术之一，用于预测，关联规则挖掘，分类等。Apriori算法是一种迭代算法，用于从事务数据集中查找频繁项集。它在每次迭代中扫描完整的数据集以生成具有不同基数的大型频繁项集，这对于小数据而言似乎更好，但对大数据而言则不可行。 MapReduce框架提供了在大事务数据上运行Apriori的分布式环境。但是，MapReduce不适合迭代过程，因此会降低性能。我们引入了一种称为混合频繁项集挖掘（HFIM）的新颖算法，该算法利用数据集的垂直布局来解决每次迭代中扫描数据集的问题。垂直数据集携带信息以找到每个项目集的支持。此外，我们还包括一些增强功能以​​减少候选项目集的数量。所提出的算法是在Spark框架上实现的，该框架结合了弹性分布式数据集的概念，并执行内存中处理以优化操作的执行时间。我们将HFIM的性能与针对各种数据集的另一种基于Spark的Apriori算法实现进行了比较。实验结果表明，HFIM在执行时间和空间消耗方面表现更好。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号