Not sure how feasible this is from a coding perspective, but I was thinking something of "Quick Friend" feature where instead of going to someone's page to friend or unfriend them, you can just click a special button under their avi or sig in a post.

You could also have a small red X next to someone's name in the Friend's List to instantly unfriend them.