林家豪,张元泽,梁千月,陈耀晖,朱明,李善军.基于视触觉与深度学习的猕猴桃无损硬度检测方法[J].农业机械学报,2023,54(10):390-398.
LIN Jiahao,ZHANG Yuanze,LIANG Qianyue,CHEN Yaohui,ZHU Ming,LI Shanjun.Non-destructive Firmness Testing of Kiwifruit Based on Visioned-based Tactile Sensor and Fusion Learning[J].Transactions of the Chinese Society for Agricultural Machinery,2023,54(10):390-398.
摘要点击次数: 201
全文下载次数: 87
基于视触觉与深度学习的猕猴桃无损硬度检测方法   [下载全文]
Non-destructive Firmness Testing of Kiwifruit Based on Visioned-based Tactile Sensor and Fusion Learning   [Download Pdf][in English]
投稿时间:2023-03-27  
DOI:10.6041/j.issn.1000-1298.2023.10.039
中文关键词:  猕猴桃  硬度  视触觉  无损检测  深度学习  卷积神经网络-长短时记忆单元
基金项目:国家柑橘产业技术体系项目(CARS-Citrus)、国家重点研发计划项目(2021YFD1400802-4、2020YFD1000101)、国家数字种植业(果园)创新分中心项目(农规发[2022]10号)和柑橘全程机械化科研基地建设项目(农计发[2017]19号)
作者单位
林家豪 华中农业大学 
张元泽 华中农业大学 
梁千月 华中农业大学 
陈耀晖 华中农业大学 
朱明 华中农业大学 
李善军 华中农业大学 
中文摘要:硬度是确定猕猴桃成熟度的重要指标之一,对其贮藏周期与销售节点均具有重要指导意义。针对现阶段缺乏使用简易、成本低且精度高的猕猴桃无损硬度检测方法的问题,提出了一种基于视触觉与深度学习的猕猴桃硬度检测方法,通过分析柔性触觉传感层与猕猴桃接触时的形变,获取猕猴桃的动态触觉信息,并据此推断其硬度。以树莓派开发板为机电控制平台,制作了猕猴桃视触觉序列图像采集装置,并对装置按压猕猴桃间隔3h后接触面果肉与非接触面果肉的CIELAB颜色分量平均数进行差异显著性检验,随后采集了猕猴桃视触觉序列图像数据集600组,分别搭建了CNN网络、CNN-LSTM迁移学习网络、CNN-LSTM联合学习网络对视触觉序列图像进行分析及硬度预测。研究结果表明,接触面果肉与非接触面果肉颜色L*、a*、b*三通道分量下平均值无显著差异;深度学习模型LSTM引入长时和短时信息可以动态关联CNN提取的单帧图像特征,从而有效推断猕猴桃硬度,其中CNN-LSTM联合学习模型预测效果最优,其均方根误差(RMSE)、平均绝对误差(MAE)、决定系数R2分别为 1.611N、1.360N、0.856,优于现阶段光谱技术检测猕猴桃硬度的结果,随后将模型嵌入树莓派中制作了猕猴桃硬度自动检测装置,可实现短时间内猕猴桃硬度的较为准确检测。因此,结合视触觉传感方法与联合学习模型可以实现对单个猕猴桃硬度的准确无损测量。
LIN Jiahao  ZHANG Yuanze  LIANG Qianyue  CHEN Yaohui  ZHU Ming  LI Shanjun
Huazhong Agricultural University
Key Words:kiwifruit  firmness  vision-based tactile sensor  non-destructive testing  deep learning  CNN-LSTM
Abstract:Firmness is one of the vital indicators to confirm the maturity of kiwifruits, which is of most significance to its storage cycle and sales node. In view of lacking non-destructive testing methods with high precision, low cost and easy use for kiwifruits at the present stage, a non-destructive testing method for kiwifruits firmness was proposed based on vision-based tactile sensor and deep learning technology. The dynamic tactile information of kiwifruit were obtained by analyzing the deformation of the flexible tactile sensing layer when it contacted with the kiwifruit, which could infer its firmness accordingly. By using the Raspberry Pi development board as an electromechanical control platform, a non-destructive firmness testing device for kiwifruit was developed and significant difference tests were conducted on the average CIELAB color components of the contact and non-contact surfaces after pressing the kiwifruit for an interval of 3 h. Subsequently, totally 600 sets of visual tactile sequence image datasets of kiwifruits were collected. At the same time, by setting the CNN network, the CNN-LSTM migration learning network and the CNN-LSTM joint learning network respectively, the firmness of visual tactile sequence images was analyzed and predicted. The research results showed that there was no significant difference between the average values of contact and non contact surfaces under the three colors’ components L*,a*, and b*. By introducing long-term and short-term information, the deep learning model LSTM can dynamically correlate the features of a single frame image extracted by CNN, thereby effectively inferring the firmness of kiwifruit. Among them, the CNN-LSTM fusion learning model had the best prediction effect, with the root mean square error (RMSE), average absolute error (MAE), and determination coefficient (R2)values of 1.611N, 1.360N, and 0.856, respectively, which was superior to the results of current spectral technology in detecting the firmness of kiwifruit. Subsequently, the model was embedded into the Raspberry Pi to create an automatic kiwifruit firmness detection device, which can achieve testing kiwifruit firmness in a short time. Combining visual and tactile sensing methods with CNN-LSTM fusion learning model can achieve accurate and non-destructive measurement of the firmness of a single kiwifruit. As well, the research result can also provide technical reference for non-destructive testing of kiwifruit firmness.

Transactions of the Chinese Society for Agriculture Machinery (CSAM), in charged of China Association for Science and Technology (CAST), sponsored by CSAM and Chinese Academy of Agricultural Mechanization Science(CAAMS), started publication in 1957. It is the earliest interdisciplinary journal in Chinese which combines agricultural and engineering. It always closely grasps the development direction of agriculture engineering disciplines and the published papers represent the highest academic level of agriculture engineering in China. Currently, nearly 8,000 papers have been already published. There are around 3,000 papers contributed to the journal each year, but only around 600 of them will be accepted. Transactions of CSAM focuses on a wide range of agricultural machinery, irrigation, electronics, robotics, agro-products engineering, biological energy, agricultural structures and environment and more. Subjects in Transactions of the CSAM have been embodied by many internationally well-known index systems, such as: EI Compendex, CA, CSA, etc.

   下载PDF阅读器