主题网络爬虫采用集中式体系结构，具有对单台服务器性能要求高、可扩展性差等缺点。提出了一种基于Hadoop的分布式主题网络爬虫架构，通过将主题网络爬虫部署在分布式集群中的不同机器，运用MapReduce编程模型对数据进行抓取分析，使不同机器共同完成对指定任务的抓取工作。实验证明，采用分布式架构，通过动态调节分布式集群中的节点个数，能够明显改善主题网络爬虫的抓取效果。%Topic Web crawler uses a centralized architecture for a single server have high performance requirements ,scal‐ability poor shortcomings ,this paper presents a distributed topic crawler Hadoop -based architecture .Topic by different machines in a distributed Web crawler deployment cluster ,using the MapReduce programming model for data analysis crawl ,crawl all the different machines together to complete work on a given task .Experiments show that the use of a dis‐tributed architecture ,distributed by dynamically adjusting the number of nodes in the cluster ,can significantly improve the topic craw ler to craw l effect .
展开▼
