Self-supervised learning (SSL) has improved the performance of various downstream tasks such phoneme recognition and speech recognition. However, training SSL models require a large amount of pre-training data. In this thesis, we are exploring ways to pre-train the SSL models with low resource training conditions which is an unexplored research direction for low resource languages.