With the development of information technology and sensor technology, people have paid more attention to data-driven fault diagnosis. As one of the commonly used methods in fault diagnosis, deep learning has achieved significant results. However, in engineering practice, the insufficient number of labeled samples for fault diagnosis and the poor targeting of extracted features lead to a limited structural depth of deep learning models and inadequate model training, limiting the diagnostic accuracy of fault diagnosis. A novel fault diagnosis method is proposed in this paper by implementing model-based transfer learning in the Inception-ResNet-v2 model. Firstly, the process applies a signal-to-image transformation method in the feature extraction stage to merge the frequency weighted energy operator (FWEO), kurtosis, and raw vibration signals into RGB images as the input dataset for diagnosing the type of rolling bearing faults. Secondly, a new combined transfer learning and Inception-ResNet-v2 CNN model (TL-IRCNN) is proposed for rolling bearing fault diagnosis under minor sample conditions. Finally, The performance of the proposed method was validated using the motor bearing dataset from Case Western Reserve University (CWRU) and the rolling bearing dataset from a local laboratory. The results show that the proposed TL-IRCNN method achieves high fault classification accuracy under minor sample conditions in bearing diagnosis.