Development of a Large-scale Web Crawler and Search Engine Infrastructure

机译：大型Web爬网程序和搜索引擎基础结构的开发

获取原文

获取原文并翻译 | 示例

页面导航

摘要
著录项
相似文献
相关主题

摘要

This paper reports the ongoing development of a large-scale Web crawler and search engine infrastructure at National Institute of Information and Communications Technology. This infrastructure has the following characteristics: (1) It collects one billion Japanese Web pages while keeping them up-to-date. (2) It selects 100 million pages from among the collected pages and converts them into a standard data format to store the results of morphological analysis, dependency parsing, and synonym augmentation. (3) The selected set of pages is searchable and accessible to the users. (4) The scalability of the system is achieved by using a large-scale cluster machine for distributed data processing.

机译：本文报告了国家信息和通信技术研究所正在开发的大型Web搜寻器和搜索引擎基础结构。该基础结构具有以下特征：（1）收集十亿个日语网页，同时保持其最新状态。（2）从收集的页面中选择1亿个页面，并将其转换为标准数据格式，以存储形态分析，依存关系分析和同义词增强的结果。（3）所选页面集可供用户搜索和访问。（4）系统的可伸缩性是通过使用大型集群计算机进行分布式数据处理来实现的。

著录项

来源
《3rd international universal communications symposium 2009》|2009年|P.126-131|共6页
会议地点 Tokyo(JP);Tokyo(JP)
作者
Susumu Akamine; rnYoshikiyo Kato; rnDaisuke Kawahara; rnKeiji Shinzato; rnKentaro Inui; rnSadao Kurohashi; rnYutaka Kidawara;
展开▼
作者单位

National Institute of Information and Communications Technology 3-5 Hikaridai Seika-cho, Soraku-gun, Kyoto 619-0289, Japan;

rnNational Institute of Information and Communications Technology 3-5 Hikaridai Seika-cho, Soraku-gun, Kyoto 619-0289, Japan;

rnNational Institute of Information and Communications Technology 3-5 Hikaridai Seika-cho, Soraku-gun, Kyoto 619-0289, Japan;

rnGraduate School of Informatics, Kyoto University Yoshida Honmachi, Kyoto 606-8501, Japan;

rnNational Institute of Information and Communications Technology 3-5 Hikaridai Seika-cho, Soraku-gun, Kyoto 619-0289, Japan Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, NARA 630-0192 JAPAN;

rnNational Institute of Information and Communications Technology 3-5 Hikaridai Seika-cho, Soraku-gun, Kyoto 619-0289, Japan G;

展开▼
会议组织
原文格式 PDF
正文语种 eng
中图分类通信;
关键词
web information analysis; search engine; crawler;

机译：网站信息分析；搜索引擎;履带式;

相似文献

外文文献
中文文献
专利

1. Using Exclusive Web Crawlers to Store Better Results in Search Engines'Database [J] . Ali Tourani, Amir Seyed Danesh International Journal of Web & Semantic Technology . 2013,第2期

机译：使用独家网络抓取工具将更好的结果存储在搜索引擎的数据库中
2. An XML based Web Crawler with Page Revisit Policy and Updation in Local Repository of Search Engine [J] . Jyoti Mor, Dr Dinesh Rai, Dr Naresh Kumar International Journal of Engineering & Technology . 2018,第3期

机译：基于XML的Web爬网程序，在搜索引擎的本地存储库中具有页面重新访问策略和更新
3. Design of a Least Cost (LC) Vertical Search Engine based on Domain Specific Hidden Web Crawler [J] . Sudhakar Ranjan, Komal Kumar Bhatia International journal of information retrieval research . 2017,第2期

机译：基于特定于域的隐藏Web爬虫的最低成本（LC）垂直搜索引擎的设计
4. Development of a large-scale web crawler and search engine infrastructure [C] . Susumu Akamine, Yoshikiyo Kato, Daisuke Kawahara, Proceedings of the 3rd International Universal Communication Symposium . 2009

机译：开发大型网络爬虫和搜索引擎基础架构
5. Providing content by Web -based delivery methods: Using digital video, instructor -selected Websites, and search engines, to deliver information about the principles of behaviorism. [D] . Quinn, Andrew Stewart. 2004

机译：通过基于Web的传递方法提供内容：使用数字视频，讲师选择的网站和搜索引擎来传递有关行为主义原理的信息。
6. Quantitative evaluation of recall and precision of CAT Crawler a search engine specialized on retrieval of Critically Appraised Topics [O] . Peng Dong, Ling Ling Wong, Sarah Ng, 2004

机译：CAT Crawler的召回率和准确性的定量评估CAT Crawler是专门检索关键评估主题的搜索引擎
7. Design and Implementation of Scalable, Fully Distributed Web Crawler for a Web Search Engine [O] . M. Sunil Kumar 2011

机译：Web搜索引擎的可扩展，完全分布式Web爬网程序的设计和实现

Development of a Large-scale Web Crawler and Search Engine Infrastructure

摘要

著录项

相似文献

相关主题

期刊订阅