In order to meet the real-time detection requirements under the limited resources of embedded devices, a passion fruit detection model based on improved YOLO v5 lightweight network (MbECA-v5) was proposed. Firstly, MobileNetV3 was used to replace the feature extraction network, the depth separable convolution was used to replace the traditional convolution to reduce the number of model parameters. Secondly, the effective channel attention network (ECANet) was embedded to focus on the whole passion fruit. Point-by-point convolution connection feature extraction network and feature fusion network were introduced to improve the feature extraction ability and fitting ability of the network for passion fruit images. Finally, the transfer learning strategy combined with cross-domain and within-domain multi-training was used to improve the network detection accuracy. Experimental results showed that the accuracy and recall of the improved model were 95.3% and 88.1%, respectively. The mAP value of 88.3%,compared with the model before the improvement, it was increased by 0.2 percentage points. And the number of calculations was 6.6 GFLOPs. The model volume was only 6.41MB, which was about half of the improved model. The real-time detection speed in embedded device was 10.92f/s, the detection speed in embedded device was about 14 times,39 times and 1.7 times of SSD, Faster RCNN and YOLO v5s. Therefore, the lightweight model based on improved YOLO v5 greatly reduced the amount of calculation and model volume, and it can detect passion fruit in complex orchard environment efficiently on embedded devices, which was of great significance to improve the intelligent level of orchard production.