The hard part about world building in my experience is adaptability. How easy is it for players to understand the mechanics of the world? What about new potential? Loopholes, inconsistencies, that sort of thing. When you do something like a high fantasy, everyone comes in with a basic understanding of how things work: Elves are ageless and generally more perspective than most other races, dwarves are short but hardy folk, humans are humans, goblins are small and mischievous, magic is why you can throw fire balls, etc. 

Adding new, different things challenges previously established understandings. For example, what is a Half-Elf? What about a Half-Dwarf? Half-Human? How does magic respond to mechanical engineering? Are the two totally incompatible or can they work just fine? And that's just with regular fantasy stuff. It's a different can of worms if you're using different genres, like Modern or Futuristic. How do super powers work? What stops The Hulk from using an Iron Man suit? Where to super power come from anyways?

When things start breaking down into a science, that's what either makes or make a world. Because then if falls into who would actually be interested in trying to do something innovated, or who just wants to stick with what's already established and won't try to challenge the standards. Your mileage may vary of course.