李涛,冯仲科,孙素芬,程文生.基于Hadoop的气象大数据分析GIS平台设计与试验[J].农业机械学报,2019,50(1):180-188.
LI Tao,FENG Zhongke,SUN Sufen,CHENG Wensheng.Design and Test of GIS Platform for Meteorological Data Analysis Based on Hadoop[J].Transactions of the Chinese Society for Agricultural Machinery,2019,50(1):180-188.
摘要点击次数: 1807
全文下载次数: 1187
基于Hadoop的气象大数据分析GIS平台设计与试验   [下载全文]
Design and Test of GIS Platform for Meteorological Data Analysis Based on Hadoop   [Download Pdf][in English]
投稿时间:2018-07-31  
DOI:10.6041/j.issn.1000-1298.2019.01.019
中文关键词:  气象数据  分布式  Hadoop  MapReduce
基金项目:国家自然科学基金项目(U1710123)、北京市自然科学基金项目(6161001)和北京林业大学青年教师科学研究中长期项目(2015ZCQ-LX-01)
作者单位
李涛 北京林业大学 
冯仲科 北京林业大学 
孙素芬 北京市农林科学院农业科技信息研究所 
程文生 北京林业大学 
中文摘要:针对海量气象数据在传统WebGIS平台下存储和分析计算受到限制的问题,提出基于Hadoop的分布式计算和存储框架,使用了Hadoop生态体系中的HDFS分布式文件存储框架来存储管理分析海量气象数据。在海量数据的并行计算分析方面,使用MapReduce作为分布式计算编程模型,该模型通过分析海量气候数据可对农业生产进行决策。最后,利用地理信息系统空间可视化技术,在前端页面以三维形式对分析结果进行展示,并分析比较数据量和集群中节点数对计算耗时的影响。试验结果表明,使用分布式多节点集群可以有效提高海量气象数据的存储和计算效率,解决了传统WebGIS平台数据存储与计算的局限性问题。
LI Tao  FENG Zhongke  SUN Sufen  CHENG Wensheng
Beijing Forestry University,Beijing Forestry University,Beijing Forestry University;Institute of Agricultural Science and Technology Information, Beijing Academy of Agricultural and Forestry Sciences and Beijing Forestry University
Key Words:meteorological data  distributed  Hadoop  MapReduce
Abstract:Massive meteorological data is limited in storage and analysis on the traditional WebGIS platform. A distributed computing and storage framework based on Hadoop to manage and analyze a large number of meteorological data was proposed. The HDFS distributed file storage framework was used in Hadoop ecosystem to store and manage massive meteorological data. In the aspect of parallel computing and analysis of massive data, MapReduce was used as the basis of distributed computing programming model. This model can make decision for agricultural production by analyzing massive climatic data. The application of regional large data decision analysis suitable for crop growth and the analysis of large data for meteorological disaster assessment were tried out. It had great application value for the research of climate change information extraction and analysis in agricultural production decision making and other fields. Finally, the front end pages displayed the analysis results in three dimensional form by using the geographic information system spatial visualization technology, which made the analysis results more intuitive, and easier to analyze and decision making, and then the impact of size of data and the number of nodes in the cluster on computing time consuming was analyzed and compared, and the configuration was tuned the most efficient. Experiment results showed that using distributed multi node cluster can effectively improve the storage and calculation efficiency of massive meteorological data, and solve the limitations of traditional WebGIS platform.

Transactions of the Chinese Society for Agriculture Machinery (CSAM), in charged of China Association for Science and Technology (CAST), sponsored by CSAM and Chinese Academy of Agricultural Mechanization Science(CAAMS), started publication in 1957. It is the earliest interdisciplinary journal in Chinese which combines agricultural and engineering. It always closely grasps the development direction of agriculture engineering disciplines and the published papers represent the highest academic level of agriculture engineering in China. Currently, nearly 8,000 papers have been already published. There are around 3,000 papers contributed to the journal each year, but only around 600 of them will be accepted. Transactions of CSAM focuses on a wide range of agricultural machinery, irrigation, electronics, robotics, agro-products engineering, biological energy, agricultural structures and environment and more. Subjects in Transactions of the CSAM have been embodied by many internationally well-known index systems, such as: EI Compendex, CA, CSA, etc.

   下载PDF阅读器