I've done this before. It's sometimes called a collaborative, or if there is a regular swap of the GM role, a round-robin. What you need to make it realistic is some essential theme or framework that helps tie people's disparate ideas together. If you just go full-on free-for-all then everyone will wind up shoehorning in the idea they personally want rather than thinking of how it makes sense in the world. The result is always a tumescent pile of clashing and self contradictory concepts that feel nothing like a coherent and believable setting. 

tl;dr A city on the coast with high fantasy elements is not enough for player creativity to build on and have the results appear to belong in the same setting together. A good world has that special trademark that sets it apart and manifests in many, if not all, aspects of the world. 

If you give me some coherent theme then I'm in. If not, I'm afraid it's just going to be a random salad of every player's favorite trope and [url=http://gunshowcomic.com/comics/20111118-pissworld.png]magical realm[/url].