基于Solr的农田数据索引方法与大数据平台构建
作者:
作者单位:

作者简介:

通讯作者:

中图分类号:

基金项目:

国家重点研发计划项目(2017YFD0700601)


Index Method of Farmland Data Based on Solr and Construction of Big Data Platform
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    针对农田数据在高吞吐量、高并发、多条件处理过程中易产生运算负载大、响应速度慢等难题,研究了负载均衡大规模集群数据处理技术,优化了多条件检索时Hbase农田数据库,提出了基于Solr的二级非主键索引方法,搭建了基于Hadoop的农田大数据平台,采用农机深松、植保、保护性耕作等8种作业生成的100TB数据对平台进行了检索实验和压力测试实验。实验结果表明,多条件检索时,优化后的技术模型在数据规模达到5×107条时,系统的响应时间小于1s,优化的性能与原生Hbase相比提高了3倍;在模拟用户达到5×105次时,系统的QPS及TPS提高了1倍左右、RT提高了2.5倍,系统的平均响应时间为183ms。本研究解决了高吞吐量、高并发导致农田数据检索效率低的问题,提高了海量农田数据实时处理的计算能力。

    Abstract:

    Aiming at the problems of high throughput, high concurrency and slow response of farmland data in the process of multi-condition processing, such as high computational load and slow response speed. The data processing technology of load balancing large-scale cluster was studied, the Hbase farmland database in multi-condition retrieval was optimized, a two-level non-primary key index method was put forward based on Solr, and a large farmland data platform was buildt based on Hadoop. The 100TB data was generated by eight operations, such as subsoiling, plant protection and conservation tillage, and those data was retrieved and tested on the platform based on Hadoop. The experimental results showed that the response time of the optimized technical model was less than 1 s when the concurrent volume of farm data was 50 million, and the performance of the optimized model was improved by about four times compared with the original Hbase. When the concurrent volume of simulated users were 500000, the query per second (QPS)and transaction per second (TPS)of the system were increased by about one time, the response time (RT) of the system was increased by 2.5 times, and the average response time was 183ms. To a certain extent, this system solved the problem of low efficiency of farmland data retrieval caused by high throughput and concurrency, and improved the computing ability of real-time processing of massive farmland data.

    参考文献
    相似文献
    引证文献
引用本文

苑严伟,冀福华,赵博,姜含露,王猛,樊学谦.基于Solr的农田数据索引方法与大数据平台构建[J].农业机械学报,2019,50(11):186-192. YUAN Yanwei, JI Fuhua, ZHAO Bo, JIANG Hanlu, WANG Meng, FAN Xueqian. Index Method of Farmland Data Based on Solr and Construction of Big Data Platform[J]. Transactions of the Chinese Society for Agricultural Machinery,2019,50(11):186-192.

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2019-05-07
  • 最后修改日期:
  • 录用日期:
  • 在线发布日期: 2019-11-10
  • 出版日期: