That, and after recently replaying OoT, there's a NPC in Castle Town who says the Gerudo sometimes come to town to look for boyfriends, and they have the carpenters they took hostage, and besides, they're a theif clan, they certianly aren't stealing from each other. Hell Nabooru's whole reason for rejecting ganondorf was he was attacking women and children.

Edit: Ninja'd