in any classic medieval society, dyes would be fairly expensive, so it makes sense to go with browns, grays and maybe hints of reds. (red/brown/brown-red dye can be iron-based, which is easily harvested from certain types of marshland). Whites, greens and blues were rarer. Yellow might also have been available.
mind you, this is mainly for the poorer segment. Nobility would have access to far more.

The poorest probably wouldn't have any dye at all.

But that's the nice thing about fantasy... We can twist these limits around. Invent plants that supply dyes/medicine/etc. at need.