Human-Robot Interaction (HRI) and Human-Avatar Interaction (HAI) focus on enhancing interactions between humans and robots/agents to make these interactions more intelligent and effective. User confusion is a complex mental state related to cognitive load and other mental states. This PhD research aims to systematically investigate what it means to be confused in HRI and HAI. Three main experimental studies were conducted with a total of 134 participants. Data were collected and analyzed from approximately 129 out of 134 participants, using machine learning and statistical analysis to demonstrate that user confusion can be modeled and detected in HRI and HAI.