This paper researches the four key web crawler program modules on the basis of the campus BBS. They are climbing up the pages,parsing the pages,tasking schedulings and eliminating duplicate datas.It discusses the use of regular expressions and HTMLParser method when parsing the pages.In dealing with crawl data,the task scheduling algorithm has been improved and the speed of crawl has been raised.As well the campus BBS pages through keywords has been crawled.%在校园 BBS 的基础上研究了网络爬虫程序的4个关键模块：页面抓取、页面解析、任务调度和去重模块。探讨了 HTMLParser 和正则表达式在解析页面时的运用方法。在处理爬取数据时，改进了任务调度算法，提高了爬取速度，并针对校园论坛 BBS 经关键词搜索的页面进行了爬取。
展开▼
