I've always imagined that the broadsword members of the Yiga clan are some sort of sheikah gerudo mix and that's why they're so big compared to the sheikah. Sooga included in that. They did set up their main base in the gerudo highlands after all, probably because they know Gannon was a gerudo and wanted to be close to where he was born. I also would like to think Yiga clan members dye their hair black and are tattooed on the face to show dedication to the clan