With no separate ability scale then magic people can be as physically able as people who want to specialize in physical abilities. No more point in physical based characters and they already don't get a lot of love right now to begin with.

This would be a game breaker to me.