Due to the varying degree of various pests’ damage, people tend to make some counter measures to protect the vegetables. Up to now, the most common method is to spray pesticides on vegetable pests. Farmers often lead to the excessive use of pesticides for lack of information about the number of pests. Traditionally, manual counting methods are carried out on the number of pests. It needs large labor costs, heavy workload, with subjective and other shortcomings, and using machine vision to monitor vegetable pests is a popular method recently. But the vast majority of current visual methods are to be carried out under the condition of ideal laboratory, which cannot be directly applied to pest monitoring in the field. Using visual perception technology to identify pests has become a hotspot in the field of agricultural engineering in recent years. Because of the shortcomings of the pests identification under the current field conditions, a new algorithm for counting the southern vegetable pests was studied by using yellow sticky trap. Based on the classical image processing algorithm, some new algorithms, including pest image segmentation sub-algorithm based on the structure of random forest, feature extraction sub-algorithm of irregular structure, background removal sub-algorithm, interference target removal sub-algorithm and detection model counting sub-algorithm were proposed. Those sub-algorithms were integrated to create a vegetable pest count algorithm based on visual perception (VPCA-VP). The images taken in the field environment were used for experimentation and analysis, and 9351 thrips, 202 whiteflies and 23 fruit flies were recognized. Compared with the artificial count, the accuracy rate of the vegetable pest counting algorithm based on visual perception was 94.89%. Among them, the accuracy rate of the thrip was 93.19%, the accuracy rate of the whitefly was 91% and the exact rate of the fruit fly was 100%. The algorithm had good performance and achieved the rapid counting demand, which had wide application prospect in farmland monitoring.