I actually decreased my population by reducing the number of people living in colonies.
Now my total population is somewhere around 160 million.