Camellia oleifera is a unique woody oil tree species in China, which plays an important role in grain and oil security. However, there are problems with harvesting of Camellia oleifera fruits in hilly and mountainous areas such as low efficiency in manual harvesting, difficulties in flower and fruit synchronization, and difficulties in large-scale mechanical, limiting the development of mechanized harvesting of Camellia oleifera fruits. A handheld impacting combtype Camellia oleifera fruit harvester was designed. The impact component of the device was provided with a buffer sleeve and an angle adjustment mechanism. The harvesting of Camellia oleifera fruit was completed by using the collision and brushing action of the impact finger with the Camellia oleifera fruit. The goal was to minimize the quality of harvester. The topology optimization section of Ansys Workbench 19.2 software was used to lightweight design the harvester. After optimization the weight reduction of the rack was nearly 30.59%. Meanwhile, the “impact finger-Camellia oleifera” collision model and “main branch-secondary twig-fruit” three pendulum dynamic model were established. By analyzing, it can be seen that the main factors affecting the abscission of Camellia oleifera fruit were the mass and pressing deformation of Camellia oleifera fruit, the quality and rotation speed of the impact finger, and the length and quality of the branches after collision. The rotational speed of the impact finger, the number of comb strokes, and the angle between the impact fingers were taken as experimental factors. The rate of harvesting efficiency and net harvesting rate of camellia fruits as well as the rate of flower bud abscission were calculated under different experimental conditions. The experiment results were analyzed by response surface analysis. The results indicated that the performance of the picking device was best when the number of combs was 4.5, the rotating speed of the impact finger was 409.8r/min, and the impact finger included angle was 4.1°. In this case, the picking rate and net picking rate of Camellia oleifera fruit were 43.67kg/h and 86.42%, and the falling rate of bud was 8.89%, which can meet the working requirements of high net picking rate and low damage rate of flower buds.