Phenotyping aims to measure traits of interest and a key part of this requires the accurate identification of defined parts of the organism. Instance segmentation of organs, such as leaves, is a crucial prerequisite for plant phenotyping. Firstly, whether deep learning methods (such as Mask-RCNN) had generality for leaf and stem segmentation was evaluated. Training was conducted using four datasets about three plants, a public Arabidopsis dataset (CVPPP2014), and three developmental multi-view datasets from Arabidopsis, maize, and physalis. Multi-view images of given plants were collected at different developmental periods. The Arabidopsis datasets contained only leaf, and the maize and physalis datasets were different from the Arabidopsis datasets, having clearly distinct leaf, stems, and petioles. The results showed that the mean accuracy precision (mAP0.5) of the Mask-RCNN model for Arabidopsis in the public datasets which was in the same growth period reached 85.3% and the mean intersection over union (mIOU) was 73.4%. The mean accuracy precision was more than 70.0% across different growth periods of Arabidopsis, maize, and physalis. The mean intersection over union was more than 60.0% across different growth periods of Arabidopsis, which indicated that Mask-RCNN displayed satisfying versatility for plant phenotyping and had high value for plant phenotyping. The results showed that the model had competitive advantage compared with previous plant segmentation algorithms. Furthermore, taking advantage of multi-view images, a leaf tracking method was presented to solve the problem of plant occlusions. It was helpful for the leaf counting and leaf area calculation of plants. The results showed that the proposed methods had a superior performance compared with other existing plant segmentation algorithms, and was promising to build a dynamic modeling for various plants during their entire growth cycles.