I'm not sure what mixed messages you are receiving nor do I understand what you mean by "more structure means less freedom".
What it sounds like to me is you want everyone to pitch in and write the world with you, which is a completely
different kind of RP entirely and one in which requires immense dedication. Since you are the GM, the responsibility falls on you to create a world that feels alive. Currently, it feels empty mainly because a lot of things are left up in the wind. Not talking about every minute detail needs to be accounted for, but generalities like kingdoms, culture, influence of magic and aristocracy, beastiary, lay of the land, temperature, etc. etc.
These things create a world worth exploring.