Agility means a lot of different things. You have actual speed, burst speed, top speed within that. You have finesse/dexterity/control, (No one should argue that Saber Alter is super slow, after all.) and a few other things.

At the very least it isn't like you hurt yourself more trying.