It has to do with a part of an area Petyr's father ruled, called "The Fingers"

And also combined with the fact he's short.