The way I see it, magic is hard to learn, harder to control, and not everyone has a knack for it. Sure, you could go learn to fly a helicopter and get around much faster; but most people tend to go by car. Why? Because 1) helicopters are super expensive, 2) helicopters are very complicated to fly, and 3) it's not very appropriate to land in a parking lot with a helicopter.
In my opinion, it's the same with magic. Let's say you want to light a torch in the middle of town, because it's dark. You have fire magic, but using it to simply light a torch would take a lot of finesse. Still, you try it. You end up incinerating the whole torch; wasting a lot of your "mana" while you're at it, and the town guards come to arrest you for putting the lives of those around you in danger by recklessly using magic.
And what happens if you aren't trying to kill someone? You just want a quick sparring match. with swords, it's easy. Just get a dull-edged sword and boom, no lethality. It'll hurt, but you don't have to fear for your life. With magic, it's trickier. You could accidentally vaporize your best friend, or transform him into a slug with no knowledge of how to transform him back.
Magic is scary and unpredictable. That's why I think there's effort in learning it.