Considering the fact that society didnt freak out to the fact that gems are aliens is an indicator. Opposite to that point, locations like the tower in the middle of the ocean not  discovered by humans may indicate the resistance took special care to avoid fighting near humans and living with humans a long time ago.