The key to the phenomenological success of inflation models with axion and SU(2) gauge fields is the isotropic background of the SU(2) field. Previous studies showed that this isotropic background is an attractor solution during inflation starting from anisotropic (Bianchi Type I) spacetime; however, not all possible initial anisotropic parameter space was explored... In this paper, we explore more generic initial conditions without assuming the initial slow-roll dynamics. We find some initial anisotropic parameter space which does not lead to the isotropic background, but to violation of slow-roll conditions, terminating inflation prematurely. The basin of attraction increases when we introduce another scalar field acting as inflaton and make the axion-SU(2) system a spectator sector. Therefore, the spectator axion-SU(2) model is phenomenologically more attractive. (read more)