In order to realize the detection of fish passage in fishway under complex water, an improved model based on YOLOv5s was proposed, and it was deployed and applied to the fishway site of one hydropower station with TensorRT. Firstly, in view of the problems of underwater image blur and target detection difficulty, the Swin Transformer (STR) was proposed as the detection layer, which improved the detection ability of the model for targets. Secondly, in view of the problem of dense fish and little image information, the efficient channel attention (ECA) attention mechanism was used as the Bottleneck of the backbone feature extraction network C3 structure, which reduced the calculation parameters and improved the detection accuracy. Then aiming at the problem of detection target positioning error and non convergence of regression, taking focal and efficient IOU loss (FIOU) as the loss function of the model to optimize the overall performance of the model. Finally, the model was deployed in TensorRT framework for optimization, and the processing speed was greatly improved. Based on the actual collection of complex water body data set, the experiment results showed that the algorithm mAP was 91.9%, and the processing time of a single image was 10.4ms. Under the same conditions, the precision was 4.8 percentage points higher than that of YOLOv5s, and the processing time was 0.4ms. After the model was deployed with TensorRT, the reasoning speed reached 2.3ms/img, a 4.5 times improvement in reasoning speed without affecting detection accuracy. In conclusion, the algorithm model had good effectiveness and superiority, and it was more suitable for fish passage detection in complex water bodies.