The ear of winter wheat, as an important agronomic component, is not only closely associated with yield, but also plays an important role in phenotypic analysis. It was reported that the number of winter wheat ears per unit area was one of the commonly used indicators to indicate the winter wheat yield. However, the traditional manual counting method is timeconsuming and laborintensive, as well as subjective, lacking a unified winter wheat ear counting standard. In order to increase the accuracy of winter wheat ear recognition and detection in field condition, a winter wheat ear detection system was constructed based on image processing and deep learning. Firstly, a winter wheat ear recognition model was proposed, which was based on manual image segmentation and convolutional neural network classification. A 27layer network with five convolutional layers, four pooling layers and two fully connected layers was constructed. The gradient descending method (SGD) was used to train and validate the model by setting the maximum number of epochs at 200. The network was trained with an initial learning rate of 0001. In the winter wheat ear detection and counting stage, a nonmaximal suppression (NMS) method was used to overcome the effect of overlapping results by using a confidence score. The confidence score p was set to be 0.95, and the I threshold was set to be 0.1. The results showed that the system achieved an overall recognition accuracy of 99.6%, 99.9% for winter wheat ear, 99.7% for shadow and 99.3% for leaf, which indicated that the winter wheat ear detection system was capable of recognizing winter wheat ears. The linear regression was used to test the accuracy of the counting results. Normalized root mean squared error (NRMSE) and coefficient of determination (R2) were used as the criterion for evaluation. The comparison between the counting results by the system of the selected 100 photos and the manual counting results showed that R2 was 0.62 and NRMSE was 11.73%. It was revealed that the accuracy of winter wheat ears could be achieved by the system, which can provide support to yield estimation and field management of winter wheat.