As one of the important fungi, mushrooms have a wide variety. There are about 100000 species of fungi that have been found so far, and the phenotypes of most fungi have little difference. The identification and classification for the variety of fungi is a challenging task, which needs professional fungus expert knowledge to complete. As an edible mushroom, the study of its classification is of great importance. In order to be able to perform fine-grained phenotype recognition of mushrooms, a fine-grained mushroom recognition method was proposed based on transfer learning and bilinear convolutional neural network of Inception-ResNet-v2. For extracting the fine-grained features of mushroom image data, the Inception-ResNet-v2 network combined with bilinear convergence operation was employed. In addition, for improving the training performance, the pre-trained model parameters based on the ImageNet dataset were transferred for the fine-grained mushroom phenotype dataset using transfer learning skills. In order to evaluate the performance of the approach, extensive experiments were conducted, and the experimental results showed that the identification accuracy was 87.15% and 93.94% on the open source data set and the private data, respectively. Finally, a Flask-based online mushroom phenotype identification system was developed to facilitate the online identification and analysis of fine-grained mushroom phenotypes as well.