Okay, okay. I'll rewrite it to say that the High Elves were the FIRST to craft the portals. The idea is that they are significant, and that most races will revere them to some degree. 

Unless you're all just blood thirsty barbarians who want to rape and pillage all day long. 

They are there to give quests and rewards; you don't have to do what they tell you, but if you do want to benefit, you might want to IC some kind of connection to them - even if it's just to grab more power for yourselves by pretending to loyally obey them. 

You will be able to destroy them, but i wont advise it early on.