Gender Stereotypes refer to the widely held beliefs and assumptions about the typical traits, behaviours, and roles associated with a collective group of individuals of a particular gender in society. These typical beliefs about how people of a particular gender are described in text can cause harmful effects to individuals leading to unfair treatment and gender bias. In this research, the aim is to identify words and language constructs that can influence a text to be considered a gender stereotype potentially leading to gender bias. To do so, we train a transformer model  and fine-tune it for gender stereotype detection. Thereafter, using different/ensemble eXplainable AI (XAI) techniques, words/language constructs used for the model’s decision are studied. From the evaluations conducted, the outcome of this research is an algorithm that effectively signal to authors, implicit gender stereotype markers of language that could lead to gender bias.