Monitoring the state of polarization (SOP) is crucial for tracking vibrations or disturbances in the vicinity of optical fibers, such as precursors to fiber cuts. While SOP data are valuable for machine learning (ML) models in identifying vibrations, acquiring a sufficient amount of data presents a significant challenge. To overcome this hurdle, we introduce an innovative transfer learning framework designed for the identification of vibrations (events) when confronted with limited SOP data. Our methodology leverages the pre-trained convolutional neural network MobileNet as a feature extractor, incorporating the encoding of time series SOP measurements into images for MobileNet input. We explore different time series encoding techniques, including the Gramian Angular Difference Field (GADF) and the Gramian Angular Summation Field (GASF). Different architectures for building our transfer learning framework based on MobileNet are investigated. Validation of our proposed approaches is conducted using experimental data that simulates movements indicative of fiber break precursors. The experimental results clearly demonstrate the superior performance of our approaches compared to other ML algorithms, especially in scenarios with limited data. Furthermore, our framework surpasses pre-trained CNN models in terms of predictive power, affirming its effectiveness in enhancing the accuracy of vibration identification in the presence of constrained SOP data.