Firmness is a key indicator for evaluating the ripeness of kiwifruit, which is essential for fruit maturity evaluation and commercial grading. It’s aimed at the challenges of poor 'ready-to-eat' quality and the lack of low-cost grading technology for domestic kiwifruit. Focusing on the 'Cuixiang' kiwifruit, a novel method combining vision-based tactile sensing and deep learning techniques for kiwifruit firmness detection was proposed. An integrated desktop automation device, which combined feeding, conveying, detecting, and grading functions, was developed to enable non-destructive firmness evaluation and 'ready-to-eat' maturity grading for small to medium batches of kiwifruit. The fabrication method, detection principles, and hardware system construction of the vision-based tactile sensor, as well as the software control process were firstly outlined. A deep learning model, ResNet18_CBAM-LSTM with the CBAM attention mechanism, was constructed and trained. A 'ready-to-eat' grading method was proposed, and its correlation with firmness was analyzed and used as the basis for the device's grading process. Subsequently, the prototype device was evaluated, and non-destructive testing was assessed by using Mann-Whitney U significance tests based on kiwifruit respiration intensity. Experimental results showed that the proposed vision-based tactile sensing method was repeatable and stable, with a contact force variation coefficient of 0.54%. The neural network's predicted firmness and the actual firmness of the kiwifruit yielded an R2 value of 0.81 and an RMSE of 1.39 N, demonstrating accurate firmness assessment. The optimal 'ready-to-eat' firmness range for the test kiwifruit was found to be between 8.72 N and 14.28 N. Using this reference, the prototype's fruit feeding success rate was 89.92%, with a grading accuracy of 90.16% and sorting efficiency of 4.12 s per fruit. Under Mann-Whitney U testing, the P-value for the difference between the test and control groups after grading by the device was greater than 0.05. Overall, the prototype demonstrated stable operation and effective detection, achieving non-destructive grading. The research result can provide a reference for the firmness detection and 'ready-to-eat' grading of domestic kiwifruit.