In order to solve the existence of some problems such as the low weighing efficiency of dairy cows in the current pasture, and being easy to cause the stress of dairy cows by manual participation, an end-to-end method of dairy cow weight estimation (Cow-DETR) based on improved detection transformer (DETR) network was proposed. The non-contact estimation on the dairy cow weight was carried out by using the depth image of dairy cow back. Firstly, a data acquisition device was designed and built, with which the cows back depth image and weight data were collected by using the Intel RealSense D435 depth camera and the weight scale. Then, deep image data was filled by using the edge flat filter and hole filling filter to reduce the impact of deep data loss on weight estimation. Finally, by adding the weight prediction unit with an alternate fully connection layer (AFC) to the prediction module of DETR to establish a cow weight estimation model. AFC was added to improve the ability of dairy cow weight-related feature extracting. It implemented the end-to-end dairy cow back positioning while performing a non-contact estimation of dairy cow weight by Cow-DETR model. The data of 139 cows were used to evaluate the model, and the results through 5-fold cross validation showed that the weight estimation method proposed can achieve a high accuracy in dairy cow weight estimation. The average absolute error of weight estimation was below 17.21kg, the average relative error was less than 3.71%. The average recognition time was 0.026s per image. Compared with the existing weight estimation methods, the results showed that Cow-DETR got lower average absolute error and average relative error than the other six methods in more dairy cow data. In the meantime, the method proposed had less requirements on the posture of dairy cow, which can more comply with the actual production demand of ranch and provide a solution for the weight estimation of dairy cow.