Think about your race. Creatures with longer lifespans usually breed less in fantasy, so larger population means lower life expectancy.

For a real life example, some estimates put Englands population in the Mid-Middle Ages between 3 and 5 million and Europe was a little more than 100 million. That might help.