[@Lucidnonsense]
[@JohnSolaris] and I actually discussed this, though not to the extent I'll go into it, and some of the undiscussed info may be reworked later. Essentially, the obligatory [url=http://tvtropes.org/pmwiki/pmwiki.php/Main/BackgroundMagicField]Background Magic Field[/url] "flows" in a way that's something like a river, with any given individual's magical power being akin to an offshoot stream - the bigger the stream, the more power flowing through them that they can channel into effects. The logical conclusion is that a fair number of people simply aren't connected to the river at all, and thus cannot use magic, and it's not always clear-cut as to which children receive the ability to use magic, though it is more likely for a mage to have mage offspring than not. For most of those who are mages, the metaphorical stream is relatively small and unlikely to widen all too much even with concentrated practice through their lifetime, as mentioned in the OP, and in any case doesn't "want" to be used for any other purpose than destroying things ("directing large amounts of energy in a way that tends toward damaging the target", if we want to give a more physical explanation) in a manner that's vaguely allegorical to how water doesn't "want" to flow uphill, thus magic tending toward being more low fantasy than high fantasy.

In other words, mages have innate access to magic basically because they got lucky, and there isn't any particularly convenient way for muggles to gain access to it automatically. "Wizardry Works Weirdly", is a saying that mages sometimes use.