Yes, you can use any number of colors, but since you'll have more magic to master every additional color of magic chosen will slow your growth. It's a matter of versatility versus magnitude.