首页>
外国专利>
The crawling method of the distributed reptile system of the task and task for system of creeping
The crawling method of the distributed reptile system of the task and task for system of creeping
展开▼
机译：任务的分布式爬行动物系统的爬行方法和爬行系统的任务
展开▼
页面导航
摘要
著录项
相似文献
摘要
A task-crawling system for a distributed crawler system includes a controlling end, a crawling end, and a task monitoring module. The crawling end acquires a corresponding task, and sends data of the task to the controlling end. The controlling end works for assigning a number to the task, defining a timeout period for the task, generating a task-distribution event, and storing timestamp data of distribution of the task. The controlling end distributes the task distribution to the task monitoring module and the crawling end. The crawling end performs corresponding crawling logic to the crawl task, and sends information about completion of the task to the controlling end. In case of abnormality that prevents the crawl task from being performed properly, the task monitoring module re-pushes the task to the controlling end, thereby avoiding failure of the task otherwise caused by web-related problems.
展开▼
