基于CheerIO的MEAN Stack气象数据网络爬虫研究
CSTR:
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:

国家自然科学基金项目(41371001)和北京林业大学青年教师科学研究中长期项目(2015ZCQ—LX—01)


Web Crawler of Atmosphere and Weather Data Based on MEAN Stack with CheerIO
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    为全面、即时搜集分散于互联网上游离的气象数据,满足各行业、各领域、各学科科研部门的数据需求,提出使用Google MEAN Stack 全栈技术开发基于CheerIO的高效定向爬虫,充分利用Node.js 高性能I/O的特性,实现气象信息的快速搜集。同时将技术栈与地理信息系统技术、数据可视化技术以及云计算技术相结合,通过地理信息系统的数据存储、查询、自动制图、统计分析等功能对信息进行分析和处理,在阿里云平台上构建了一个能抓取并存储海量数据、提供实时气象数据的应用系统,提供便捷的检索、查询功能,有较强的实用性。本文结合气象数据爬虫的解决方案,对MEAN Stack 数据爬虫的开发框架、项目架构以及爬虫核心技术(抓取目标策略、网页分析算法、多线程并发运算等)进行了深入分析研究与实现。

    Abstract:

    To collect the meteorological data dispersed in various industries, fields and disciplines in a comprehensive and real-time way, and meet the needs of scientific research departments for data, an efficient directional crawler was developed based on Google’s full-Stack technology called MEAN (MongoDB + Express +AngularJS + Node.js) Stack and an fast flex Javascript Document Object Model module called CheerIO, the functions such as fast-gathering weather information, information analysis and processing by data storage, query, automatic mapping, statistical analysis, forecasting of GIS were realized. An application system deployed on Alicloud server which can real-timely update and forecast meteorological data was created, and it can also provide practical functions of massive data storage, convenient search and query. An efficient and practical web application system was built, which not only provided effective solutions for scattered online data collection but show people date intuitively by using HTML5 data visualizing technology. In actual project, it offered a great number of data support and example to the weather-related fields, such as forestry and preventive medicine. GIS data visualization is a constantly evolving concept, whose borders are expanding fast. At the age of the internet, especially in the globalization of information, the long-term value of data has been gained more and more recognition and affirmation from small companies to national political decision-making. It should be recognized what really it is and how it can help us.

    参考文献
    相似文献
    引证文献
引用本文

胡戎,冯仲科,蒋君志伟.基于CheerIO的MEAN Stack气象数据网络爬虫研究[J].农业机械学报,2016,47(6):275-282. Hu Rong, Feng Zhongke, Jiang Junzhiwei. Web Crawler of Atmosphere and Weather Data Based on MEAN Stack with CheerIO[J]. Transactions of the Chinese Society for Agricultural Machinery,2016,47(6):275-282.

复制
相关视频

分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2016-02-29
  • 最后修改日期:
  • 录用日期:
  • 在线发布日期: 2016-06-10
  • 出版日期: 2016-06-10
文章二维码