I always assumed ~12 or so percent of the population in total. Maybe more. Since there is no evidence given by the show, I am thinking of the amount of black people in 1860s America, which was only 14%.

Even in the height of the civil rights movement African Americans only made up ~10% of the U.S. Population.

Edit: in other news, my university Internet is shitting the bed, so please excuse the previous and likely continuing poor posts, seeing as my iPhone is not charitable to my attempts to write intelligible things.