Aiming to solve the problems that traditional grain weight classification depends on manual sorting, such as heavy workload, high error rate and lax classification standard, an improved two-stream convolutional neural network model was proposed based on ECA to classify rice by single grain weight. Firstly, images of each group of rice (a group consists seven single rice grains) were taken from two different perspectives: front view and top view. For five traditional supervised models (naive Bayes, decision tree, random forest, K-nearest neighbor, support vector machine), voting mechanism optimization based on genetic algorithm (GA)(GA-SVM) and integrated model (RF+GA-SVM), single grain images were separated through image preprocessing and contour detection. Color moment, local binary pattern (LBP) and Canny operator were used to extract grain color, texture and edge features. And then through principal component analysis (PCA), the principal features were extracted to train each model. For the constructed single-stream convolutional neural network model, two-stream convolutional neural network model (FV-CNN) and the improved two-stream convolutional neural network model were proposed based on ECA (ECA-FV-CNN), the pre-processed images were divided into training set, verification set and test set according to the ratio of 6∶2∶2, and data enhancement were carried out for each data set, and then the models were trained. By comparing and analyzing the above models, the traditional machine learning model, RF+GA-SVM, had the best effect, but its highest accuracy was only 72% when the single grain weight was set for three-graded. Experimental verification showed that the ECA-FV-CNN model proposed had the best performance, and its accuracy for the single grain weight classification of three-graded, four-graded and five-graded reached 94.0%, 92.3% and 71.0%, respectively. However, the accuracies of single-stream convolutional neural network model and FV-CNN model for single grain weight grading were 92.7%, 91.1%, 61.1% and 93.0%, 91.6%, 65.6%, respectively. The grading effect of FV-CNN model was better than that of single-stream convolutional neural network model in three experiments, which showed that the two-branch network training was better than that of single-branch rice single grain weight grading. The accuracy of ECA-FV-CNN model in three grading experiments was 16.2% higher than that of single-stream convolutional neural network model and 8.2% higher than that of FV-CNN model. The results showed that the introduction of ECA module was effective for rice single grain classification, and the improved two-stream convolutional neural network model based on ECA can improve the classification accuracy of rice single grain weight, and the classification of rice single grain weight can be achieved by using computer vision technology, making up for the shortcomings of traditional methods, and improving the classification standard of grain screening.