Fish activity intensity is one of the characteristic indicators of fish health and welfare farming. The fine-grained classification of fish activity intensity is beneficial to describe fish health status and assess fish welfare levels. The fine-grained classification of Atlantic salmon activity intensity where a smallscaled underwater video dataset was collected in the industrial recirculating aquaculture system was carried out. Firstly, the features of video frames were extracted through a small convolutional neural network with residual connections. Then the inter-frame features were obtained by performing differential and square operations between adjacent frames. Finally, the inter-frame features were inputted into the classification network IFDNet based on the external attention mechanism to obtain the video category. The experimental results showed that the classification accuracy of the CNN-IFDNet model proposed reached 97.72%, and the F1 score reached 97.42%. With low computational complexity, the three classification of the fish activity intensity video was realized. Compared with the laboratory environment, the algorithm research based on the real farming environment for fish activity intensity was more practical. The research result can provide a reference for elaborately describing the activity intensity of fish school and realizing intelligent monitoring of fish health status, which can help aquaculture workers discover abnormal conditions and investigate factors causing abnormal fish activity intensity, such as water quality environment and diseases.