Firmness is one of the vital indicators to confirm the maturity of kiwifruits, which is of most significance to its storage cycle and sales node. In view of lacking non-destructive testing methods with high precision, low cost and easy use for kiwifruits at the present stage, a non-destructive testing method for kiwifruits firmness was proposed based on vision-based tactile sensor and deep learning technology. The dynamic tactile information of kiwifruit were obtained by analyzing the deformation of the flexible tactile sensing layer when it contacted with the kiwifruit, which could infer its firmness accordingly. By using the Raspberry Pi development board as an electromechanical control platform, a non-destructive firmness testing device for kiwifruit was developed and significant difference tests were conducted on the average CIELAB color components of the contact and non-contact surfaces after pressing the kiwifruit for an interval of 3 h. Subsequently, totally 600 sets of visual tactile sequence image datasets of kiwifruits were collected. At the same time, by setting the CNN network, the CNN-LSTM migration learning network and the CNN-LSTM joint learning network respectively, the firmness of visual tactile sequence images was analyzed and predicted. The research results showed that there was no significant difference between the average values of contact and noncontact surfaces under the three colors’ components L*,a*, and b*. By introducing long-term and short-term information, the deep learning model LSTM can dynamically correlate the features of a single frame image extracted by CNN, thereby effectively inferring the firmness of kiwifruit. Among them, the CNN-LSTM fusion learning model had the best prediction effect, with the root mean square error （RMSE）, average absolute error (MAE), and determination coefficient （R2）values of 1.611N, 1.360N, and 0.856, respectively, which was superior to the results of current spectral technology in detecting the firmness of kiwifruit. Subsequently, the model was embedded into the Raspberry Pi to create an automatic kiwifruit firmness detection device, which can achieve testing kiwifruit firmness in a short time. Combining visual and tactile sensing methods with CNN-LSTM fusion learning model can achieve accurate and non-destructive measurement of the firmness of a single kiwifruit. As well, the research result can also provide technical reference for non-destructive testing of kiwifruit firmness.