A multispectral and panchromatic image fusion algorithm based on mean-shift segmentation and the non-subsampled shearlet transform (NSST) was presented. Mean-shift segmentation was performed on the panchromatic image, and the variance of each region was used to distinguish the multispectral image into regions that need to be spatially enhanced or not. Then, the multi-scale NSST was performed on the panchromatic image and the intensity component of the multispectral image. The low frequency components were fused based on the fourth-order correlation coefficient, and the band-pass frequency components were fused based on the variances of the segmented regions. At last, the fused intensity component was obtained by reconstruction of those fused components, and the inverse IHS transform was performed to obtain the fused high resolution multispectral image. Experimental results indicated that the proposed image fusion method could keep a balance between spectral preservation and spatial enhancement.