The red turpentine beetle (RTB) is a major forestry invasive insect that damages the coniferous species of pine trees in northern China. Therefore, the monitoring of RTB plays an important role in forestry pest controlling. However, traditional trap-based monitoring depends on human experts to manually recognize and count pests, which prohibits the modern RTB monitoring. To automatically recognize and count RTB captured by pheromone traps, a RGB camera was integrated in traditional cup trap to capture in-trap images and build the bark beetles dataset. The default boxes of Faster R-CNN object detection model based on deep learning were optimized by the K-means clustering algorithm. The optimized Faster R-CNN models were trained end to end by the GPU server, which enabled the in-trap detection of RTB with unconstrained postures. The models were evaluated by two metrics: the object oriented quantitative metric and the trap oriented qualitative metric. The experiments demonstrated that the optimized models outperformed the original Faster R-CNN model in terms of both metrics. The area under the curve (AUC) of precision-recall plot for object and trap on difficult test sets were increased by 4.33% and 3.28%, respectively. The AUC for object and trap on all test sets reached 0.9350 and 0.9722, respectively. The detection speed of the model was 1.6s per image. The optimized models outperformed the SSD, Faster R-CNN object detection models in terms of accuracy, which was robust to pose variance, bark interference, alcohol volatilization, etc. The proposed method distinguished and counted RTBs from the six species of scolytidae insects attracted by the pheromone lure, which could reduce the human cost of pest monitoring and forecasting.